OpenTelemetry traces: Best practices

Familiarize yourself with these OpenTelemetry trace topics to ensure your traces and spans appear in New Relic.

Required fields

The startTimeUnixNano and endTimeUnixNano fields on spans are required according to the OpenTelemetry protocol for trace data. When startTimeUnixNano is not present, the span is dropped and a NrIntegrationError is created. When endTimeUnixNano is not present, the duration of your span is large and negative.

The timeUnixNano field on span events is required. When timeUnixNano is not present, the span event is dropped and a NrIntegrationError is created.

The traceId and spanId fields on spans are required according to the OpenTelemetry protocol for trace data. When traceId or spanId are not present, the span is dropped and a NrIntegrationError is created.

Deprecated and removed fields

Span status codes were deprecated in OTLP v0.6.0 and have been removed as of v0.12.0. Clients setting and sending these messages and fields will see the value set to UNRECOGNIZED. Please contact support of this causes issues.

Sampling

Trace data is the most mature OpenTelemetry data type. Because of this, New Relic's OpenTelemetry user experience is largely based on trace data and is therefore influenced by your sampling strategy.

You can configure sampling in a number of places:

Service: Use the OpenTelemetry SDK for your language.
Collector: If you're running your own instance of the OpenTelemetry Collector, you can configure it to do more sophisticated forms of sampling, such as tail-based sampling (see below).

Check out this documentation about how to configure different types of sampling:

Infinite Tracing is New Relic's tail-based sampling option. You can use this in conjunction with your OpenTelemetry instrumented services. In setting up Infinite Tracing, you need to configure applications (or the collector) to export trace data to the New Relic trace observer using OTLP gRPC:

Follow the steps in Set up the trace observer to get the value for YOUR_TRACE_OBSERVER_URL.
As you complete the steps in the quick start guide, use the value of YOUR_TRACE_OBSERVER_URL to configure your integration. YOUR_TRACE_OBSERVER_URL follows the form https://{trace-observer}:443/trace/v1. When setting the OTLP gRPC endpoint, strip off the /trace/v1 suffix, resulting in a URL of the form https://{trace-observer}:443.
Since you want New Relic to analyze all your traces, make sure to verify that all applications involved in the trace have configured the OpenTelemetry SDK with a sampler which allows tail-based sampling. The default parentbased_always_on as well as the always_on samplers are good choices.
Note that only trace data can be sent to trace observer endpoints. Your application (or collector) will need to separately configure export strategies for OpenTelemetry metrics and logs.

Resource attributes and stored bytes

OpenTelemetry resource attributes are defined to be an immutable representation of the entity producing telemetry and stored as attributes. It's important to understand how these attributes are applied to tracing data within New Relic and how they can impact stored bytes. We can observe the impact of the resource attributes by using an example OTLP payload and inspecting the resulting NRQL query output.

Example OTLP Trace Payload

{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "newrelic-otlp-service"
            }
          },
          {
            "key": "process.command_line",
            "value": {
              "stringValue": "/opt/java/openjdk/bin/java -javaagent:agent/opentelemetry-agent.jar"
            }
          }
        ]
      },
      "scopeSpans": [
        {
          "scope": {
            "name": "newrelic-instrumentation-library",
            "version": "1.0.0"
          },
          "spans": [
            {
              "attributes": [
                {
                  "key": "message_id",
                  "value": {
                    "stringValue": "000000-aaaaaa-111111-bbbbbb"
                  }
                }
              ],
              "startTimeUnixNano": "1677182057000000000",
              "endTimeUnixNano": "1677182059000000000",
              "kind": "SPAN_KIND_INTERNAL",
              "name": "example-span",
              "spanId": "c469d81892057f5f",
              "traceId": "aa04993b9acefbedea802f8d96e4bc58"
            }
          ]
        }
      ]
    }
  ]
}

NRQL query representation

{
  "duration.ms": 2000.0,
  "entity.guid": "OBFUSCATED",
  "entity.name": "newrelic-otlp-service",
  "entity.type": "SERVICE",
  "entityGuid": "OBFUSCATED",
  "guid": "c469d81892057f5f",
  "id": "c469d81892057f5f",
  "instrumentation.provider": "opentelemetry",
  "message_id": "000000-aaaaaa-111111-bbbbbb",
  "name": "example-span",
  "newRelic.ingestPoint": "api.traces",
  "newrelic.source": "api.traces.otlp",
  "nr.isPrimaryEntityData": true,
  "otel.library.name": "newrelic-instrumentation-library",
  "otel.library.version": "1.0.0",
  "process.command_line": "/opt/java/openjdk/bin/java -javaagent:agent/opentelemetry-agent.jar",
  "service.name": "newrelic-otlp-service",
  "span.kind": "internal",
  "timestamp": 1677182057000,
  "trace.id": "aa04993b9acefbedea802f8d96e4bc58",
  "traceId": "aa04993b9acefbedea802f8d96e4bc58"
}

Tip

This example is a simplified comparison meant to illustrate the underlying concepts. Real-world versions will look a little different because they have more complexity.

The most important thing to notice with resource attributes is the potential difference in the size of the payload being sent compared to what is stored in NRDB. All resource attribute values will be applied to every span in the OTLP payload. The example above only shows a single span being sent but if the payload contained 100 spans, each one of them would have process.command_line and service.name applied to them.

For some Java based applications, the default process.command_line attribute can be thousands of characters long which may result in a significant and unexpected increase in billable bytes. If these resource attributes do not provide value they can be disabled by following the OpenTelemetry and attribute lengths: Best Practices

Required fields

Deprecated and removed fields

Sampling

OpenTelemetry built-in samplers

OpenTelemetry tail-based samplers

New Relic tail-based sampling with Infinite Tracing

Resource attributes and stored bytes

Tip