Trace API general requirements and limits

This document contains details about New Relic's Trace API data requirements, including:

  • Data specifications and max limits
  • Required metadata (headers, query parameters)
  • Response validation details

This document applies to the entire Trace API. For rules regarding specific data formats, see:

Feature requirements

Requirements to use this feature:

  • New Relic APM Pro subscription: All APM Pro customers are entitled to send up to 5,000 spans/minute, per account family.
  • New Relic Traces subscription: Customers with a New Relic Traces contract receive a default span per minute rate limit that is 3x the monthly purchased span amount and is calculated as (number of monthly purchased spans / 750 hours / 60 minutes) * 3.

Endpoint details and maximum limits

All trace data is sent via a POST to:

https://trace-api.newrelic.com/trace/v1

If your account hosts data in the EU data center, ensure you're using the proper API endpoints for EU region accounts.

Currently, the Trace API accepts two types of data formats:

The following limits apply for a paid Trace API subscription, for all trace data formats:

Condition Limit with subscription
Max age of span timestamp values

20 minutes. timestamp must be within 20 minutes of current time at ingest, or within 20 minutes from the time the last span with the same trace.id was received by New Relic.

Max payload size 1 MB (gzip compression supported)
Max requests per minute 100K
Default spans per minute per account family*

The default span per minute rate limit is 3x the monthly purchased span amount and is calculated as (number of monthly purchased spans / 750 hours / 60 minutes) * 3.

To request a rate limit increase, send an email to rate-limit-request@newrelic.com with the account ID(s) and the desired rate limit.

Max spans per minute per account family

Traces subscription: Dependent on agreement. Max limit: 2M.

APM Pro without Traces subscription: 5K/minute.

Max spans per trace 50K
Max attributes per span 200
Max span attribute value length 4000 characters
Allowed HTTP protocols HTTPS only
Cross-account visibility of span details Potential data obfuscation

*Limits apply to an account family, defined as a set of related accounts under a single customer contract. This limit is set at the account family level because a distributed trace can traverse services across many different accounts.

To see an example of how span limits are enforced, see Exceeding limits.

Restricted attributes

The attributes in the table below are restricted in the newrelic-format JSON (in the attributes block) and in the zipkin-format JSON (in the tags block). Any values with these keys will be omitted:

Restricted attribute description

entityGuid
string

Unique identifier for the entity that created this span. Generated from service.name, if available.

guid
string

Used for backwards compatibility with data from New Relic APM agents.

Request metadata (headers and query parameters)

The following table shows the required request metadata for all trace data formats. This metadata can be sent as HTTP headers on an ingest request or, in some cases, provided as query parameters, which may be required for tracing frameworks that don't allow header modification.

Security note: We suggest using headers because query parameters are present in the URL and may be logged before being encrypted and received by New Relic. All data sent as query parameters must be URL-safe.

Header Query param? Details
Content-Type No Required. Must be application/json.
Content-Length No Required. The length of the request body in octets (8-bit bytes) unless sent with chunked encoding. This header is generally set by default by the underlying HTTP client sending the data and in most cases should not require any additional effort by the end user.
Api-Key Yes (case-sensitive) Required. The Trace API requires the Event API insert key. If this is provided as both a header and a query parameter, the values must match.
Content-Encoding No Required if compressed payload. The value must be gzip.
Data-Format Yes

Required for zipkin. Optional for newrelic.

If present, Data-Format-Version must also be present.

Data-Format-Version Yes

Required for zipkin.

If present, Data-Format must also be present.

There are only two possible pairings for these values:

  • If Data-Format is zipkin, Data-Format-Version must be 2.
  • If Data-Format is newrelic, Data-Format-Version must be 1.

Response validation

A response for successfully sending trace data will include a requestId. For example:

{"requestId":"c1bb62fc-001a-b000-0000-016bb152e1bb"}

There are two ways success/errors are signaled:

  • HTTP status code (synchronous). Authentication and request errors will be signaled via HTTP status code.

    See HTTP status codes
    Code Meaning
    202 Data accepted. This means that you've passed preliminary checks, but is not a guarantee that the data has been successfully parsed and indexed as part of a distributed trace.
    400 The structure of the request was invalid. Errors with query parameters, etc.
    403 Authentication error. May occur with an invalid license key or if you lack necessary entitlement to use the Trace API.
    404 The request path is incorrect.
    405 For any request method other than POST.
    408 The request took too long to reach the endpoint.
    411 The Content-Length header wasn’t included.
    413 The payload was too big.
    414 The request URI was too long.
    415 The Content-Type or Content-Encoding was invalid.
    429 The request rate quota has been exceeded.
    431 The request headers are too long.
    5xx There was a server error (please retry).
  • NrIntegrationError events (asynchronous). Errors with the JSON payload or other semantic errors are asynchronously signaled via NrIntegrationError events that are stored in the account whose license key is associated with the request. For all errors of this type, the attribute newRelicFeature will be Distributed Tracing and requestId will be the requestId from the endpoint response.

If you receive a 202 response and don't see an NrIntegrationError event, your data should be visible in New Relic One's global distributed tracing UI in about a minute. You should be able to find the trace using a standard trace search like:

traceId = TRACE_ID_SENT

Exceeding span limits

When you exceed your span rate limit, an NrIntegrationError event is generated. You can query rate limit messages with this NRQL:

SELECT * FROM NrIntegrationError WHERE newRelicFeature = 'Distributed Tracing' AND category = 'RateLimit' AND rateLimitType = 'SpansPerMinute'

To get a notification when you exceed the limit, you can set up a NRQL alert.

We calculate a rolling 10-minute average based on your span rate limit. This allows for temporary rate bursts, and lets us prioritize keeping and dropping complete traces instead of indiscriminately dropping spans on a per minute limit basis.

In the example below of exceeding the rate, the rate limit is the default 100,000 spans per minute. New Relic allows a burst above 100K for a couple of minutes without downsampling, because the remaining minutes in the 10-minute window averaged under 100K spans/minute. For the previous 10 minutes (8:50 - 9:00) the service received 60,000 spans/minute.

Minute Spans sent to API Total for past 10 minutes
8:59 60,000 600,000
9:00 40,000 580,000
9:01 50,000 570,000
9:02 250,000 760,000
9:03 220,000 920,000
9:04 125,000 985,000
9:05 70,000 995,000
9:06 50,000 985,000
9:07 40,000 965,000
9:08 40,000 945,000
9:09 40,000 925,000

For more help

Recommendations for learning more: