OpenTelemetry traces: Best practices

Familiarize yourself with these OpenTelemetry trace topics to ensure your traces and spans appear in New Relic.

Required fields

The startTimeUnixNano and endTimeUnixNano fields on spans are required according to the OpenTelemetry protocol for trace data. When startTimeUnixNano is not present, the span is dropped and a NrIntegrationError is created. When endTimeUnixNano is not present, the duration of your span is large and negative.

The timeUnixNano field on span events is required. When timeUnixNano is not present, the span event is dropped and a NrIntegrationError is created.

The traceId and spanId fields on spans are required according to the OpenTelemetry protocol for trace data. When traceId or spanId are not present, the span is dropped and a NrIntegrationError is created.

Deprecated and removed fields

Span status codes were deprecated in OTLP v0.6.0 and have been removed as of v0.12.0. Clients setting and sending these messages and fields will see the value set to UNRECOGNIZED. Please contact support of this causes issues.

Sampling

Trace data is the most mature OpenTelemetry data type. Because of this, New Relic's OpenTelemetry user experience is largely based on trace data and is therefore influenced by your sampling strategy.

You can configure sampling in a number of places:

Service: Use the OpenTelemetry SDK for your language.
Collector: If you're running your own instance of the OpenTelemetry collector, you can configure it to do more sophisticated forms of sampling, such as tail-based sampling (see below).

Check out this documentation about how to configure different types of sampling:

Infinite Tracing is New Relic's tail-based sampling option. You can use this in conjunction with your OpenTelemetry instrumented services. In setting up Infinite Tracing, you need to configure applications (or the collector) to export trace data to the New Relic trace observer using OTLP gRPC:

Follow the steps in Set up the trace observer to get the value for YOUR_TRACE_OBSERVER_URL.
As you complete the steps in the quick start guide, use the value of YOUR_TRACE_OBSERVER_URL to configure your integration. YOUR_TRACE_OBSERVER_URL follows the form https://{trace-observer}:443/trace/v1. When setting the OTLP gRPC endpoint, strip off the /trace/v1 suffix, resulting in a URL of the form https://{trace-observer}:443.
Since you want New Relic to analyze all your traces, make sure to verify that all applications involved in the trace have configured the OpenTelemetry SDK with a sampler which allows tail-based sampling. The default parentbased_always_on as well as the always_on samplers are good choices.
Note that only trace data can be sent to trace observer endpoints. Your application (or collector) will need to separately configure export strategies for OpenTelemetry metrics and logs.

Required fields

Deprecated and removed fields

Sampling

OpenTelemetry built-in samplers

OpenTelemetry tail-based samplers

New Relic tail-based sampling with Infinite Tracing