New Relic OTLP endpoint

OpenTelemetry Protocol, or OTLP for short, is a general purpose telemetry data delivery protocol designed for the OpenTelemetry project. Each OpenTelemetry language SDK provides OTLP exporters, and the OpenTelemetry collector has OTLP receivers and exporters. Additionally, various tools outside the OpenTelemetry project have added support for OTLP export.

New Relic supports native OTLP ingest, and recommends it as the preferred method for sending OpenTelemetry data to the New Relic platform. This document delves into New Relic's OTLP support, including configuration requirements and recommendations.

Before you begin

• If you haven't already done so, sign up for a free New Relic account.
• Get the license key for the New Relic account to which you want to report data. This license key will be used to configure the api-key header.
• Review your OTLP version: New Relic uses OTLP release v0.18.0. Later versions are supported but new features are not yet implemented. Experimental features which were are no longer available in 0.18.0 are not supported.

Config: OTLP endpoint, port, and protocol

Requirement level: Required

In order to configure send OTLP data to New Relic, you must configure your OTLP exporter to use the relevant endpoint and port from the table below based on your environment.

The mechanism to configure the endpoint will vary, but OpenTelemetry language SDKs generally support setting the OTEL_EXPORTER_OTLP_ENDPOINT=<INSERT_ENDPOINT> environment variable (see OpenTelemetry docs for more info).

Additionally, you should configure your OTLP exporter to use the OTLP/HTTP binary protobuf version of the protocol if available. While New Relic supports all versions of OTLP, OTLP/HTTP binary protobuf has proved to be more robust than gRPC without any apparent reduction in performance.

The mechanism to configure the endpoint will vary, but OpenTelemetry language SDKs generally support setting the OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf environment variable (see OpenTelemetry docs for more info).

If you're using a collector, we recommend using the otlphttpexporter.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otlp.nr-data.net:4318/v1/traces

Config: TLS encryption

Requirement level: Required

In order to send OTLP data to New Relic, you must configure your OTLP exporter to use TLS 1.2 (see TLS encryption for more information). Generally, SDK and collector exporters meet this requirement by default.

While many OTLP exporters infer TLS settings from the https endpoint scheme, some gRPC exporters may require you to explicitly enable TLS. The mechanism to configure gRPC TLS will vary, but OpenTelemetry language SDKs generally support setting the OTEL_EXPORTER_OTLP_INSECURE=false environment variable (see OpenTelemetry docs for more info).

Config: Setting the API key

Requirement level: Required

In order to send OTLP data to New Relic, you must configure your OTLP exporter to include a header named api-key with the value set to your license key. Failure to do so will result in authentication errors.

The mechanism to configure headers will vary, but OpenTelemetry language SDKs generally support setting the OTEL_EXPORTER_OTLP_HEADERS=api-key=<INSERT_LICENSE_KEY> environment variable (see OpenTelemetry docs for more info).

Config: Attribute limits

Requirement level: Required

In order to send OTLP data to New Relic, you must configure your telemetry source to conform to New Relic attribute limits. Failure to do so may result in NrIntegrationError events during asynchronous data validation.

Attribute limits are as follows:

• Maximum length of attribute name: 255 characters
• Maximum length of attribute value: 4095 characters
• Maximum size of attribute array value: 64 entries

See metric attribute limits and event attribute limits for other limits.

The mechanism to configure attribute limits will vary, but OpenTelemetry language SDKs generally support setting the OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT=4095 and OTEL_ATTRIBUTE_COUNT_LIMIT=64 environment variables (see OpenTelemetry docs for more info).

If using the collector, the transform processor can be configured to truncate attributes to New Relic limits.

Notes:

• Resource attributes are subject to attribute limits, but there are no standard environment variables to limit them. If a resource attribute is over the allowed limit, consider truncating using the collector transform processor, or overwriting the resource attribute to another value.
• There is no standard mechanism to limit attribute names. However, instrumentation generally does not produce attribute names which exceed New Relic limits. If you encounter name length limits, remove the attribute using the collector.

Config: Payload batching, compression and rate limits

Requirement level: Required

In order to send OTLP data to New Relic, your payloads must be smaller than the 1MB (10^6 bytes) maximum payload size. Larger payloads will be rejected with an error status code. Larger payloads may also fail to export with a timeout before an error status code is returned.

Additionally, New Relic imposes rate limits. When the rate limit is exceeded, requests will be rejected with an error status code.

To avoid payload size limits and rate limits, you must configure your OTLP exporter to use an appropriate batch size which causes data to be exported on an appropriate interval.

The mechanism for configuring batching will vary. OpenTelemetry SDKs generally support setting the following environment variables (see OpenTelemetry docs for more info):

• OTEL_BSP_* for spans
• OTEL_METRIC_EXPORT_* for metrics
• OTEL_BLRP_* for logs

If using the collector, the batch processor controls batch size.

Additionally, you should enable compression to reduce payload size and limit the likelihood of encountering payload size limits. New Relic supports gzip and zstd compression. zstd compression is higher performance and recommended if your exporter supports it. See compression comparison for more details on benchmark information.

The mechanism to configure the endpoint will vary, but OpenTelemetry language SDKs generally support setting the OTEL_EXPORTER_OTLP_COMPRESSION=gzip environment variable (see OpenTelemetry docs for more info).

If using the collector, gzip is the default compression, but zstd can be optionally configured.

Config: Retry

Requirement level: Recommended

In order to send OTLP data to New Relic, you should configure your OTLP exporter to retry when transient errors occur. The internet is unreliable, and failure to retry increases the likelihood of data loss.

The mechanism to configure retry will vary. Some OpenTelemetry SDKs may have language specific environment variables (for example java supports setting OTEL_EXPERIMENTAL_EXPORTER_OTLP_RETRY_ENABLED=true), but there is no general mechanism. Programmatic configuration may be required.

If using the collector, the otlphttpexporter and otlpexporter retry by default. See exporterhelper for more details.

Config: Metric aggregation temporality

Requirement level: Recommended

In order to send OTLP metric data to New Relic, you should configure your OTLP metrics exporter to prefer delta aggregation temporality. While New Relic supports cumulative aggregation temporality, the New Relic metrics architecture is generally a delta metrics system. Using the default cumulative setting will generally incur more memory usage from SDKs and result in high data ingest.

The mechanism to configure the endpoint will vary, but OpenTelemetry language SDKs generally support setting the OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta environment variable (see OpenTelemetry docs for more info). If manually setting temporality, configure by instrument kind as follows:

• Counter, Asynchronous Counter, Histogram: Delta
• UpDownCounter, Asynchronous UpDownCounter, Gauge, Asynchronous Gauge: Cumulative

Cumulative temporality is used for instruments which map to New Relic gauge types, and which are generally analyzed using the cumulative value.

Config: Metric mistogram aggregation

Requirement level: Recommended

In order to send OTLP metric data to New Relic, you should configure your OTLP metrics exporter to aggregate measurements from histogram instruments to exponential histograms. In contrast to the static buckets used with the default explicit bucket histograms, exponential histograms automatically adjust their buckets to reflect the range of measurements recorded. Additionally, they use a highly compressed representation to send over the wire. Exponential histograms provide more useful distribution data in the New Relic platform.

The mechanism to configure the endpoint will vary, but OpenTelemetry language SDKs generally support setting the OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION=base2_exponential_bucket_histogram environment variable (see OpenTelemetry docs for more info).

OTLP response payloads

Please note the following details regarding New Relic OTLP endpoint response payloads:

• Successful responses from New Relic have no response body, instead of a Protobuf-encoded response based on the data type.
• New Relic responds after validation of authentication, payload size, and rate limiting. Validation of payload contents occurs asynchronously. Therefore, New Relic may return success status codes despite data ingestion ultimately failing and resulting in NrIntegrationError events.
• Failure responses from New Relic do not include Status.message or Status.details.