New Relic OTLP Troubleshooting

New Relic has supported native OTLP ingest for several years. In the process of working through support cases that come up from time to time, we've learned about common issues users face. For some, the problem is easy to identify and fix. Others are deviously tricky, given that the internet is unreliable and there are many components (software, networking, hardware, etc.) involved under the control of various parties, such as customers, New Relic, and public networking infrastructure outside the control of either. With so much complexity, configuration, and failure points, it can be difficult to determine which is at fault and how to best address.

Working through a support case can be time consuming and at times frustrating for customers (and for New Relic!). Therefore, we've put together this troubleshooting guide to help establish a shared understanding, and provide tools to self-diagnose and fix issues when possible.

First, please review New Relic OTLP configuration requirements and recommendations. It contains essential advice and context that anyone looking to use OTLP with New Relic should be aware of.

Next, see the sections below:

• General troubleshooting: Troubleshooting for general OTLP issues
• Issues catalog: Common customer issues and mitigation steps

General troubleshooting

When you encounter an issue with the New Relic OTLP endpoint, first follow these basic troubleshooting steps. If you end up opening a support ticket, these are the first things we ask:

1. Enable diagnostic logs. Ensure that your OTLP client has logging enabled such that you can see details about any errors which may be occruing. The mechanism for enabling logs varies by implementation. Please consult relevant docs.
2. Test that New Relic's OTLP endpoint is reachable. A simple shell command curl http://otlp.nr-data.net from the machine in question can help determine if some local network configuration issue (i.e. firewall restriction) is preventing connectivity to New Relic.
3. Check for NrIntegrationError events. New Relic OTLP ingest performs minimal validation synchronously before returning a success status code. If you don't see indications of export errors in your application logs, but don't see data in New Relic, try querying for NrIntegrationError. There may be issues with your data which were detected during asynchronous validation.
4. Determine if the problem is localized. Often errors are localized to a specific application or environment. In these cases, it's useful to evaluate the differences between the areas which are problematic and properly functioning.
5. Look for signs of invalid API key. The New Relic OTLP endoint requires setting an api-key header. Invalid or missing API keys are a common issue which present with HTTP 403 or 401 status codes, or gRPC Unauthenticated or PermissionDenied status codes. If you see these, check that your API key is valid and is being properly set.
6. Check if the export succeeds after retry. We recommend that retry is enabled, and expect export attempts to occasionally initially fail with transient errors but succeed after retrying. However, we do have an SLA. If you suspect that transient errors are frequent enough that they exceed our SLA, please open a support case.
7. Check for signs that transient errors are not being retried. Despite our best efforts, there may be corner cases where the New Relic OTLP endpoint returns non-retriable status codes for transient errors. If you believe you've encountered this scenario, please open a support case.

Issue catalog

The table below catalogs issues we've seen customers encounter with the New Relic OTLP endpoint. Many are straight forward to resolve with proper configuration. The Fingerprint column shows a typical log when an application encounters the particular class of issue. See the Known resolution and Notes columns for mitigation steps.