OpenTelemetry Protocol (OTLP) is a general-purpose telemetry data delivery protocol designed for the OpenTelemetry project. Each OpenTelemetry language SDK provides OTLP exporters, and the OpenTelemetry Collector has OTLP receivers and exporters. Additionally, various tools outside the OpenTelemetry project have added support for OTLP export.
New Relic supports native OTLP ingest and recommends it as the preferred method for sending OpenTelemetry data to the New Relic platform. This page covers New Relic's OTLP support including configuration, requirements, and recommendations.
Before you begin
If you haven't already done so, sign up for a free New Relic account.
Get your for the New Relic account to which you want to report data. This license key will be used when configuring the
api-keyheader.
Config: OTLP endpoint, port, and protocol
Requirement level: Required
In order to configure send OTLP data to New Relic, you must configure your OTLP exporter to use the relevant endpoint and port from the table below based on your environment.
The mechanism to configure the endpoint will vary, but OpenTelemetry language SDKs generally support setting the OTEL_EXPORTER_OTLP_ENDPOINT=<INSERT_ENDPOINT> environment variable (see OpenTelemetry docs for more info).
Additionally, you should configure your OTLP exporter to use the OTLP/HTTP binary protobuf version of the protocol if available. While New Relic supports all versions of OTLP, OTLP/HTTP binary protobuf has proved to be more robust than gRPC without any apparent reduction in performance.
The mechanism to configure the endpoint will vary, but OpenTelemetry language SDKs generally support setting the OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf environment variable (see OpenTelemetry docs for more info).
If you're using a Collector, we recommend using the otlphttpexporter.
Environment | gRPC | HTTP | Endpoint | Supported ports |
|---|---|---|---|---|
US OTLP | ✅ | ✅ |
|
|
EU OTLP | ✅ | ✅ |
|
|
US FedRAMP OTLP | ✅ | ✅ |
|
|
Infinite tracing | ✅ | ✅ |
|
|
Config: TLS encryption
Requirement level: Required
In order to send OTLP data to New Relic, you must configure your OTLP exporter to use TLS 1.2 (see TLS encryption for more information). Generally, SDK and Collector exporters meet this requirement by default.
While many OTLP exporters infer TLS settings from the https endpoint scheme, some gRPC exporters may require you to explicitly enable TLS. The mechanism to configure gRPC TLS will vary, but OpenTelemetry language SDKs generally support setting the OTEL_EXPORTER_OTLP_INSECURE=false environment variable (see OpenTelemetry docs for more info).
Config: Setting the API key
Requirement level: Required
In order to send OTLP data to New Relic, you must configure your OTLP exporter to include a header named api-key with the value set to your license key. Failure to do so will result in authentication errors.
The mechanism to configure headers will vary, but OpenTelemetry language SDKs generally support setting the OTEL_EXPORTER_OTLP_HEADERS=api-key=<INSERT_LICENSE_KEY> environment variable (see OpenTelemetry docs for more info).
Config: Attribute limits
Requirement level: Recommended
New Relic imposes a variety of limits on attributes in OTLP payloads. In general, we have a lenient posture on validation and will prune, truncate, or otherwise modify attributes rather than dropping the data all together. Whenever an attribute is modified, we create an NrIntegrationError to help track down and fix the problem at the source.
Case | Description | Relevant config |
String value length limit | Attribute values of type string have a max length of 4095 characters. Values exceeding this limit are truncated. | Set |
Key value length limit | Attribute keys have a max length of 255 characters. Keys exceeding this limit are truncated. | |
Attribute count limit | Each attribute source is subject to a count limit. When exceeded, excess attributes are pruned, prioritizing attributes defined in the OpenTelemetry semantic conventions. Limits are as follows:
| Partial mitigation by setting |
Value types | Attribute values are limited to types: string, integer, double, boolean, and byte array. Attribute values with complex types are pruned. Log attributes are excluded from this limit. [1] | |
Unset values | Attribute values must set (i.e. | |
Array length limit | Arrays have a max length of 64 entries. Attributes exceeding this limit are pruned. | |
Array homogeneity | Arrays entries must all be of the same type. Attributes with heterogeneous array entries are pruned. Log attributes are excluded from this limit. [1] | |
Unset array entries | Array entries must not be unset (i.e. | |
Array entry value types | Array entries are limited to types: string, integer, double, and boolean. Array attribute values with complex types are pruned. Log attributes are excluded from this limit. [1] | |
Byte array length limit | Byte arrays have a max length of 128,000. Attributes exceeding this limit are pruned. | |
Attribute value types | Attribute values are limited to types: string, integer, double, boolean, and byte array. Attribute values with complex types are pruned. Log attributes are excluded from this limit. [1] |
[1]: See OTLP attribute types.
See metric attribute limits and event attribute limits for other limits.
Config: Payload batching, timeout, compression and rate limits
Requirement level: Required
In order to send OTLP data to New Relic, your payloads must be smaller than the 1MB (10^6 bytes) maximum payload size. Larger payloads will be rejected with an error status code. Larger payloads may also fail to export with a timeout before an error status code is returned.
Additionally, New Relic imposes rate limits. When the rate limit is exceeded, requests will be rejected with an error status code.
To avoid payload size limits and rate limits, you must configure your OTLP exporter to use an appropriate batch size which causes data to be exported on an appropriate interval.
The mechanism for configuring batching will vary. OpenTelemetry SDKs generally support setting the following environment variables (see OpenTelemetry docs for more info):
OTEL_BSP_*for spansOTEL_METRIC_EXPORT_*for metricsOTEL_BLRP_*for logs
If using the Collector, the batch processor controls batch size.
Additionally, you should pay attention to exporter timeout settings. Generally, export requests take longer when payloads are larger and when networks are slower (higher latency, lower bandwidth). If your application produces large payloads because telemetry volume is high or the export interval is high, you may need to increase the default timeout settings to avoid export errors.
The mechanism to configure timeout will vary, but OpenTelemetry language SDKs generally support setting the OTEL_EXPORTER_OTLP_TIMEOUT environment variable (see OpenTelemetry docs for more info).
Additionally, you should enable compression to reduce payload size and limit the likelihood of encountering payload size limits. New Relic supports gzip and zstd compression. zstd compression is higher performance and recommended if your exporter supports it. See compression comparison for more details on benchmark information.
The mechanism to configure the endpoint will vary, but OpenTelemetry language SDKs generally support setting the OTEL_EXPORTER_OTLP_COMPRESSION=gzip environment variable (see OpenTelemetry docs for more info).
If using the Collector, gzip is the default compression, but you can optionally configure zstd.
Config: Retry
Requirement level: Recommended
In order to send OTLP data to New Relic, you should configure your OTLP exporter to retry when transient errors occur. The internet is unreliable, and failure to retry increases the likelihood of data loss.
The mechanism to configure retry will vary. Some OpenTelemetry SDKs may have language specific environment variables (for example java supports setting OTEL_EXPERIMENTAL_EXPORTER_OTLP_RETRY_ENABLED=true), but there is no general mechanism. Programmatic configuration may be required.
If using the Collector, the otlphttpexporter and otlpexporter retry by default. See exporterhelper for more details.
Config: Metric aggregation temporality
Requirement level: Recommended
In order to send OTLP metric data to New Relic, you should configure your OTLP metrics exporter to prefer delta aggregation temporality. While New Relic supports cumulative aggregation temporality, the New Relic metrics architecture is generally a delta metrics system. Using the default cumulative setting will generally incur more memory usage from SDKs and result in high data ingest.
The mechanism to configure the endpoint will vary, but OpenTelemetry language SDKs generally support setting the OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta environment variable (see OpenTelemetry docs for more info). If manually setting temporality, configure by instrument kind as follows:
- Counter, Asynchronous Counter, Histogram: Delta
- UpDownCounter, Asynchronous UpDownCounter, Gauge, Asynchronous Gauge: Cumulative
Cumulative temporality is used for instruments which map to New Relic gauge types, and which are generally analyzed using the cumulative value.
Config: Metric histogram aggregation
Requirement level: Recommended
In order to send OTLP metric data to New Relic, you should configure your OTLP metrics exporter to aggregate measurements from histogram instruments to exponential histograms. In contrast to the static buckets used with the default explicit bucket histograms, exponential histograms automatically adjust their buckets to reflect the range of measurements recorded. Additionally, they use a highly compressed representation to send over the wire. Exponential histograms provide more useful distribution data in the New Relic platform.
The mechanism to configure the endpoint will vary, but OpenTelemetry language SDKs generally support setting the OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION=base2_exponential_bucket_histogram environment variable (see OpenTelemetry docs for more info).
OTLP protocol version
New Relic uses OTLP release v1.4.0. Later and earlier versions are supported but new features are not yet implemented. Experimental features which were removed after 0.18.0 are not supported.
See Traces, Metrics, and Logs for specific details on how data is mapped and which features are implemented.
OTLP attribute types
Attributes are a recurring concept in OpenTelemetry and OTLP. OpenTelemetry has a standard attribute definition, stating that attribute values are primitives (string, boolean, double floating point, 64-bit integer) or homogeneous arrays of primitives. However, at the OTLP protocol level, attributes are represented using a more expansive AnyValue definition. Because of this, it's possible for OTLP clients to send attributes which do not conform to OpenTelemetry standard definition.
The New Relic OTLP endpoint supports the standard attribute definition. Complex types like maps of maps, object arrays, and heterogeneous arrays are not supported. The OpenTelemetry SDKs should only produce data which conforms to the standard attribute definition.
Important
While the standard attribute definition is generally used, log record attributes are exceptional and support complex values (for example, log record attributes type is map<string, any>). Despite this, New Relic currently only supports log record attributes which conform to the standard attribute definition.
OTLP response payloads
Please note the following details regarding New Relic OTLP endpoint response payloads:
- Successful responses from New Relic have no response body, instead of a Protobuf-encoded response based on the data type.
- New Relic responds after validation of authentication, payload size, and rate limiting. Validation of payload contents occurs asynchronously. Therefore, New Relic may return success status codes despite data ingestion ultimately failing and resulting in
NrIntegrationErrorevents. - Failure responses from New Relic do not include
Status.messageorStatus.details.