Report traces via the Trace API (New Relic format)

You can send traces in the New Relic format to our Trace API without using a New Relic agent or integration. This is useful if you want to integrate with sources that emit tracing data and require a backend for trace storage.

Since distributed systems can generate a lot of trace data, telemetry tools rely on data sampling (filtering) to find the most useful subset of that data. When you send us your trace data, you have two options related to sampling:

  • Use your tool’s native sampling: This approach sends the telemetry data directly to the Trace API and assumes you are relying on your telemetry tool to sample the data before the integration passes the data to New Relic. Even though your tool may be sampling data before the New Relic integration sends it, our Trace API may randomly sample out traces that exceed your rate limit.

  • Use Infinite Tracing: This approach assumes you set sampling to 100% in your telemetry tool so you send all your data to a New Relic trace observer in AWS. The trace observer selects the most important and actionable traces using tail-based sampling and then sends your traces to our Trace API.

To send Zipkin-formatted data to the Trace API, see our Zipkin instructions.

We'll look at two examples for sending your telemetry data to New Relic, as well as how to structure your data:

Example: Use your tool’s native sampling

If you are using a telemetry tool that is sampling your trace data, you can send the data directly to New Relic. The following procedure explains how to send a test payload to the Trace API using the newrelic format.

For details about how to structure your data, see JSON requirements and recommendations.

  1. Get an Insert API key: Go to: one.newrelic.com > account dropdown > Account settings > API keys, and select Insights API keys. If you don't have a key, create a new one by selecting Insert keys +.

  2. Insert your Insert API key into the following JSON and then send the JSON to our endpoint.

    curl -i -H "Content-Type: application/json" \
        -H "Api-Key: YOUR_INSERT_API_KEY" \
        -H 'Data-Format: newrelic' \
        -H 'Data-Format-Version: 1' \
        -X POST \
        -d '[
            {
                "common": {
                "attributes": {
                    "service.name": "Test Service A",
                    "host": "host123.test.com"
                }
                },
                "spans": [
                {
                    "trace.id": "123456",
                    "id": "ABC",
                    "attributes": {
                    "duration.ms": 12.53,
                    "name": "/home"
                    }
                },
                {
                    "trace.id": "123456",
                    "id": "DEF",
                    "attributes": {
                    "service.name": "Test Service A",
                    "host": "host456.test.com",
                    "duration.ms": 2.97,
                    "name": "/auth",
                    "parent.id": "ABC"
                    }
                }
                ]
            }
            ]' 'https://trace-api.newrelic.com/trace/v1'
    

    If you're sending more than one POST, change the trace.id to a unique value. Sending the same payload or span id multiple times for the same trace.id may result in fragmented traces in the UI.

  3. If your test returned HTTP/1.1 202 Accepted, go to our UI to see a query of your test data using the span attribute service.name = Test Service A.

    Traces may take up to one minute to be processed by both the trace observer and the Trace API.

Example: Use Infinite Tracing

Infinite Tracing requires you to provision a trace observer that receives your spans. The trace observer examines the spans and keeps the actionable ones. You provision a trace observer in a cluster of services in AWS called New Relic Edge.

If you send your data to a trace observer in one of our EU provider regions, you'll still need a US-based New Relic account because the tail-based sampling data is reported to data centers in the United States. If you have questions, contact your account representative.

Use our Edge app to set up the trace observer. This app also includes some optional configurations: trace observer monitoring and span attribute trace filter. If you're not sure whether you want to configure these, you can set them up later.

Complete the following:

Step A. Set up the trace observer endpoint
  1. Go to one.newrelic.com.
  2. In the menu bar, click Apps, expand Your apps, and then click New Relic Edge.
  3. Select an account in the upper-left dropdown.

    If you have access to multiple account hierarchies, make sure you're in the hierarchy where you want a trace observer.

  4. If no trace observers are listed, click New trace observer to add one, insert a descriptive name, select a region, and then click Create.
  5. For the trace observer you want to use, go to the Endpoints dropdown and select For other integrations.
  6. Click the adjacent clipboard icon to copy and hold the endpoint to use later as YOUR_TRACE_OBSERVER_URL in Step B. Send test data to the trace observer.
  7. (Optional) If you want to start gathering trace observer metrics, click the toggle for Trace observer monitoring. For details, see trace observer monitoring.
  8. (Optional) If you want to customize the span attribute trace filter, click the gear icon below Trace filters to see the current rules. For details, see span attribute filter.
Step B. Send test data to the trace observer

This test includes a sample payload with one trace and two spans from the same service: Test Service A. Follow these steps to send a test request:

  1. Get or generate your Insert API key so you can use it later in the test.
  2. Copy the following curl request into a text editor:
    curl request
    curl -i -H "Content-Type: application/json" \
        -H "Api-Key: YOUR_INSERT_API_KEY" \
        -H 'Data-Format: newrelic' \
        -H 'Data-Format-Version: 1' \
        -X POST \
        -d '[
            {
                "common": {
                "attributes": {
                    "environment": "staging"
                }
                },
                "spans": [
                {
                    "trace.id": "123456",
                    "id": "ABC",
                    "attributes": {
                        "duration.ms": 12.53,
                        "host": "host123.test.com",
                        "name": "/home",
                        "service.name": "Test Service A"
                    }
                },
                {
                    "trace.id": "123456",
                    "id": "DEF",
                    "attributes": {
                        "duration.ms": 2.97,
                        "host": "host456.test.com",
                        "error.message": "Invalid credentials",
                        "name": "/auth",
                        "parent.id": "ABC",
                        "service.name": "Test Service B"
                    }
                }
                ]
            }
            ]' \
    'YOUR_TRACE_OBSERVER_URL'
    
  3. Insert your own values into the curl request:
    Value Description
    YOUR_INSERT_API_KEY Replace this with your Insert API key (not the same as your personal API key for NerdGraph API explorer)
    YOUR_TRACE_OBSERVER_URL Replace this with the value you copied in Step A. Set up the trace observer endpoint.
  4. Copy the content of the text editor into a terminal, and then execute the request.
  5. If the test does not return HTTP/1.1 202 Accepted indicating success, check the following and try again:
    • Confirm that you used the Edge app value For other integrations as YOUR_TRACE_OBSERVER_URL.
    • Confirm that you only have single quotes around the value you inserted for YOUR_TRACE_OBSERVER_URL.
    • Check that you are using the Insert API Key (not a license).
  6. If your test returned HTTP/1.1 202 Accepted, go to New Relic One to see a query of your test data using the span attribute service.name = Test Service A. Because the sample payload contains an error attribute, the error sampler will mark it for keeping. If you modify the payload to remove the error attributes, the random sampler may not choose to keep this particular trace.

    Traces may take up to one minute to be processed by both the trace observer and the Trace API.

JSON requirements and recommendations

Requirements and guidelines for trace JSON using the newrelic format:

  • Each JSON payload is an array of objects.
  • Each object should contain a required spans key.
  • Each object may contain an optional common key. Use this if you want share information across multiple spans in a object.
  • Any keys on a span have precedence over the same key in the common block.
  • The value for spans key is a list of span objects.
  • Certain attributes are required, and must be included either in the optional common block, or in each span.
  • Recommended and custom attributes may be optionally included in a list of key/value pairs under a key named attributes, in the optional common block and/or in each span.

In the following example POST, there are two spans, both of which will have the trace.id 12345 and the custom attribute host: host123.test.com. The first span has no parent.id, so that is the root of the trace; the second span's parent.id points to the ID of the first.

[
  {
    "common": {
      "attributes": {
        "host": "host123.test.com"
      }
    },
    "spans": [
      {
        "trace.id": "12345",
        "id": "abc",
        "attributes": {
          "user.email": "bob@newr.com",
          "service.name": "my-service",
          "duration.ms": 750,
          "name": "my-span"
        }
      },
      {
        "trace.id": "12345",
        "id": "def",
        "attributes": {
          "parent.id": "abc",
          "service.name": "second-service",
          "duration.ms": 750,
          "name": "second-span"
        }
      }
    ]
  }
]

Attributes

Review the sections below describing the attributes for the attributes block of a newrelic formatted JSON POST.

Required attributes

Requests without these attributes will be rejected, and an NrIntegrationError will be generated. These are the only attributes that don't belong under attributes.

attribute description
id
string
Unique identifier for this span.
trace.id
string
Unique identifier shared by all spans within a single trace.

While not required, these attributes should be included for the best experience with your data. They are included in the attributes section.

attribute default description
duration.ms
float
none Duration of this span in milliseconds.
name
string
none The name of this span.
parent.id
string
none The id of the caller of this span. Value is null if this is the root span. Traces without a root span will not displayed.
service.name
string
none The name of the entity that created this span.
timestamp
long
Defaults to the current time in UTC timezone if not provided. The span's start time in Epoch milliseconds.

Reserved attributes

These attributes are currently reserved for internal New Relic usage. While they are not explicitly blocked, we recommend not using them.

attribute default description
entity.name
string
service.name This is derived from the service.name attribute.
entity.type
string
service The entity type is assumed to be a service.
entity.guid
string
None The entity.guid is a derived value that uniquely identifies the entity in New Relic's backend.

Other attributes

You can add any arbitrary attributes you want, with the exception of the restricted attributes. For example, you might want to add attributes like customer.id or user.id to help you analyze your trace data.

To learn how to control how spans appear in New Relic (for example, adding errors or setting a span as a datastore span), see Decorate spans.

Explore more about distributed tracing:

For more help

If you need more help, check out these support and learning resources: