• /
  • Log in

Trace API general requirements and limits

Information about Trace API data requirements, including:

  • Data specifications and max limits
  • Required metadata (headers, query parameters)
  • Response validation details

This document applies to the Trace API overall. For rules regarding specific data formats, see:

Endpoints

All trace data is sent via HTTPS POST to a Trace API endpoint. We have a few endpoints, depending on your setup:

Data formats

Currently, the Trace API accepts two types of data formats:

Restricted attributes

The attributes in the table below are restricted in the newrelic-format JSON (in the attributes block) and in the zipkin-format JSON (in the tags block). Any values with these keys will be omitted:

Restricted attribute

Description

entityGuid

string

Unique identifier for the entity that created this span. Generated from service.name, if available.

guid

string

Used for backwards compatibility with data from APM agents.

The attributes in the table below are used internally to identify entities. Any values submitted with these keys in the attributes section of a metric data point may cause undefined behavior such as missing entities in the UI or telemetry not associating with the expected entities. For more information please refer to Entity synthesis:

Restricted attribute

description

entity.guid

string

Unique identifier for the entity associated with this span.

entity.name

string

Human-readable name of an entity, often used to identify an entity in the UI.

entity.type

string

Used to differentiate between different types of entities, like hosts, applications, etc.

Request metadata (headers and query parameters)

The following table shows the required request metadata for all trace data formats. This metadata can be sent as HTTP headers on an ingest request or, in some cases, provided as query parameters, which may be required for tracing frameworks that don't allow header modification.

Important

Security note: We suggest using headers because query parameters are present in the URL and may be logged before being encrypted and received by New Relic. All data sent as query parameters must be URL-safe.

Header

Query param?

Details

Content-Type

No

Required. Must be application/json.

Content-Length

No

Required. The length of the request body in octets (8-bit bytes) unless sent with chunked encoding. This header is generally set by default by the underlying HTTP client sending the data and in most cases should not require any additional effort by the end user.

Api-Key

Yes (case-sensitive)

Required. The Trace API requires a license key. If this is provided as both a header and a query parameter, the values must match.

Content-Encoding

No

Required if compressed payload. The value must be gzip.

Data-Format

Yes

Required for zipkin. Optional for newrelic.

If present, Data-Format-Version must also be present.

Data-Format-Version

Yes

Required for zipkin.

If present, Data-Format must also be present.

There are only two possible pairings for these values:

  • If Data-Format is zipkin, Data-Format-Version must be 2.
  • If Data-Format is newrelic, Data-Format-Version must be 1.

x-request-id

No

Optional - Reserved for future use. The value must be a valid UUID4. The value is expected to be unique for each request.

Response validation

A response for successfully sending trace data will include a requestId. For example:

{"requestId":"c1bb62fc-001a-b000-0000-016bb152e1bb"}

There are two ways success/errors are signaled:

  • HTTP status code (synchronous). Authentication and request errors will be signaled via HTTP status code.

  • NrIntegrationError events (asynchronous). Errors with the JSON payload or other semantic errors are asynchronously signaled via NrIntegrationError events that are stored in the account whose license key is associated with the request. For all errors of this type, the attribute newRelicFeature will be Distributed Tracing and requestId will be the requestId from the endpoint response.

If you receive a 202 response and don't see an NrIntegrationError event, your data should be visible in New Relic One's global distributed tracing UI in about a minute. You should be able to find the trace using a standard trace search like:

traceId = TRACE_ID_SENT

Data limits

Distributed tracing rate limits are set per account and data type. For details about data limits, see New Relic data usage limits and policies.

When you exceed your span rate limit, an NrIntegrationError event is generated. You can query rate limit messages with this NRQL:

SELECT * FROM NrIntegrationError WHERE newRelicFeature = 'Distributed Tracing' AND category = 'RateLimit' AND rateLimitType = 'SpansPerMinute'

To get a notification when you exceed the limit, you can set up a NRQL alert.

We calculate a rolling 10-minute average based on your span rate limit. This allows for temporary rate bursts, and lets us prioritize keeping and dropping complete traces instead of indiscriminately dropping spans on a per minute limit basis.

In the example below of exceeding the rate, the rate limit is the default 100,000 spans per minute. New Relic allows a burst above 100K for a couple of minutes without downsampling, because the remaining minutes in the 10-minute window averaged under 100K spans/minute. For the previous 10 minutes (8:50 - 9:00) the service received 60,000 spans/minute.

Minute

Spans sent to API

Total for past 10 minutes

8:59

60,000

600,000

9:00

40,000

580,000

9:01

50,000

570,000

9:02

250,000

760,000

9:03

220,000

920,000

9:04

125,000

985,000

9:05

70,000

995,000

9:06

50,000

985,000

9:07

40,000

965,000

9:08

40,000

945,000

9:09

40,000

925,000

For more help

If you need more help, check out these support and learning resources:

Create issueEdit page
Copyright © 2021 New Relic Inc.