Troubleshooting OpenTelemetry with New Relic may just be a matter of making sure you are following best practices, but sometimes you may need to take additional steps to diagnose your issues. Here are some examples of specific problems you might encounter, along with steps and tools to resolve them.
You sent OpenTelemetry metrics, logs, or traces using OTLP and are unable to view the data. Before digging deeper, make sure you've checked the following:
- The OTLP endpoint configured matches one of our documented endpoints, is properly formatted, and includes the official default port, 4317. Sending OTLP data via port 443 is not supported at this time. Please note the specific endpoint for FedRAMP compliance, if applicable.
- The outbound traffic is not restricted by a firewall. Our Networks document explains domains and network blocks that you may need to explicitly allow.
- The client is configured to use TLS 1.2 or higher and the request includes the
api-keyheader with a valid New Relic account (ingest) license key.
- Requests include valid protobuf payloads and use gRPC and HTTP/2 transport, preferably with gzip compression enabled. Sending protobuf or JSON-encoded payloads over HTTP/1.1 is not supported at this time.
- Client output and logs do not indicate
5xxresponse codes are being returned.
There are number of tools you can use to validate the successful delivery of telemetry data to our platform. A good first step is to check the data management hub to facet data ingest and determine how much data is arriving from various sources. You can also use the data explorer or query builder to look for data faceted by
FROM Log, Metric, Span SELECT datapointcount() WHERE instrumentation.provider = 'opentelemetry' FACET instrumentation.provider, newrelic.source
This query should tell you whether data is arriving via OTLP. If the data you expect is not present, try this alternate query:
FROM Log, Metric, Span SELECT count(*) where newrelic.source LIKE 'api.%.otlp'
You can also check for integration errors by querying NrIntegrationError events. This can help you determine whether you have configuration or format issues or if you've run into our platform limits.
The ingest limits for metrics, logs, and traces via OTLP are the same as our other data ingest API limits.
Various parts of the New Relic UI rely on the presence of specific attributes to function properly. You can use the NRQL console feature in many places to check the
FACET clauses of the query for required attributes. You can also edit those clauses and re-run the query to determine whether there is data present with those attributes missing. Examples of required attributes include
service.instance.id. For a more complete list of examples, see resources.
OpenTelemetry entities will be synthesized based on the public rules described for the
EXT-SERVICE entity type. The standard rule to match relies on the presence of the
service.name dimension which follows the OpenTelemetry semantic conventions.
To set the
service.name with the OpenTelemetry Java SDK, include it in your resource:
var resource = Resource.getDefault().merge(Resource.builder().put(SERVICE_NAME, serviceName).build());
Depending on the SDK, you may also set the
service.name by declaring it in the
OTEL_SERVICE_NAME environment variables.
For Logs, you can use a structured log template to inject the
service.name. Here are some log examples:
For more OpenTelemetry examples with New Relic, visit the newrelic-opentelemetry-examples repository on GitHub.