Report traces via the Trace API (New Relic format)

This document explains how to send traces in the New Relic format to our Trace API without using a New Relic agent or integration. This is useful if you want to integrate with sources that emit tracing data and require a backend for trace storage.

Since distributed systems can generate a lot of trace data, telemetry tools rely on data sampling (filtering) to find the most useful subset of that data. When you send us your trace data, you have two options related to sampling:

  • Use your tool’s native sampling: This approach sends the telemetry data directly to the New Relic Trace API and assumes you are relying on your telemetry tool to sample the data before the integration passes the data to New Relic. Even though your tool may be sampling data before the New Relic integration sends it, our Trace API may randomly sample out traces that exceed your rate limit.

  • Use New Relic Infinite Tracing (limited release): This approach assumes you turn off sampling in your telemetry tool and then send all your data to a New Relic trace observer in AWS. The trace observer selects the most important and actionable traces using tail-based sampling and then sends your traces to the New Relic UI.

To send Zipkin-formatted data to the New Relic Trace API, see our Zipkin instructions.

We'll look at two examples for sending your telemetry data to New Relic, as well as how to structure your data:

Example: Use your tool’s native sampling

If you are using a telemetry tool that is sampling your trace data, you can send the data directly to New Relic. The following procedure explains how to send a test payload to the Trace API using the newrelic format.

For details about how to structure your data, see JSON requirements and recommendations.

  1. Get an Insert API key: Go to: one.newrelic.com > account dropdown > Account settings > API keys, and select Insights API keys. If you don't have a key, create a new one by selecting Insert keys +.

  2. Insert your Insert API key into the following JSON and then send the JSON to our endpoint.

    curl -i -H "Content-Type: application/json" \
        -H "Api-Key: YOUR_INSERT_API_KEY" \
        -H 'Data-Format: newrelic' \
        -H 'Data-Format-Version: 1' \
        -X POST \
        -d '[
            {
                "common": {
                "attributes": {
                    "service.name": "Test Service A",
                    "host": "host123.test.com"
                }
                },
                "spans": [
                {
                    "trace.id": "123456",
                    "id": "ABC",
                    "attributes": {
                    "duration.ms": 12.53,
                    "name": "/home"
                    }
                },
                {
                    "trace.id": "123456",
                    "id": "DEF",
                    "attributes": {
                    "service.name": "Test Service A",
                    "host": "host456.test.com",
                    "duration.ms": 2.97,
                    "name": "/auth",
                    "parent.id": "ABC"
                    }
                }
                ]
            }
            ]' 'https://trace-api.newrelic.com/trace/v1'
    

    If you're sending more than one POST, change the trace.id to a unique value. Sending the same payload or span id multiple times for the same trace.id may result in fragmented traces in the UI.

  3. If your test returned HTTP/1.1 202 Accepted, go to New Relic One to see a query of your test data using the span attribute service.name = Test Service A.

    Traces may take up to one minute to be processed by both the trace observer and the Trace API.

Example: Use New Relic Infinite Tracing

Infinite Tracing can examine all of your data and find the most useful traces. To get the biggest benefit from Infinite Tracing, you should turn off sampling in your telemetry tool and send all your trace data to the trace observer.

To start using Infinite Tracing, please go to our sign-up page.

The following steps show you how to find a trace observer endpoint using GraphQL and send a test payload using the newrelic format. For details about how to structure your data, see JSON requirements and recommendations.

Step A. Open NerdGraph API explorer

Prepare to execute some GraphQL:

  1. Open NerdGraph API explorer.
  2. In the API key dropdown next to Tools, select your personal API key for your account under one of these options:
    • Select an existing API Key
    • Create a new API Key
    Screen capture showing the dropdown options in the NerdGraph API explorer.
Step B. Execute a query to find an existing trace observer endpoint

The following steps show you how to execute a GraphQL query to find out if you can use an existing trace observer in your AWS region. If one isn't available, you can go to Step C. Create a new one.

  1. Copy the following query into the middle pane of the NerdGraph API explorer and replace YOUR_ACCOUNT_ID with your account ID (the number next to your account name in NerdGraph API explorer):
    {
      actor {
        account(id: YOUR_ACCOUNT_ID) {
          edge {
            tracing {
              traceObservers {
                errors {
                  message
                  type
                }
                traceObservers {
                  endpoints {
                    agent {
                      host
                      port
                    }
                    endpointType
                    https {
                      host
                      port
                      url
                    }
                    status
                  }
                  id
                  name
                  providerRegion
                }
              }
            }
          }
        }
      }
    }     
            
  2. Click the triangle button to execute the query or press Ctrl+Enter.
  3. Check the right pane showing the results:
Step C. If you can't find an existing trace observer endpoint, create a new one

If you didn't find a trace observer after running the query in the previous section, you need to create one. To do this, you execute a GraphQL mutation that passes your configuration details.

  1. Copy the following into the middle pane of the NerdGraph API explorer.
    mutation {
      edgeCreateTraceObserver(accountId: YOUR_ACCOUNT_ID, traceObserverConfigs: {name: "YOUR_DESCRIPTIVE_NAME", providerRegion: YOUR_PROVIDER_REGION}) {
        responses {
          errors {
            message
            type
          }
          traceObserver {
            endpoints {
              agent {
                host
                port
              }
              endpointType
              https {
                host
                port
                url
              }
              status
            }
            id
            name
            providerRegion
          }
        }
      }
    }
                
  2. Insert your own values into the mutation:
    Value Description
    YOUR_ACCOUNT_ID Replace this with your account ID (the number next to your account name in NerdGraph API explorer).
    YOUR_DESCRIPTIVE_NAME Replace this with a name that describes the services that report to this trace observer (for example, production or query services).
    YOUR_PROVIDER_REGION Replace this with desired provider region enum value (currently only AWS_US_EAST_1 is available).
  3. Click the triangle button to execute the mutation or press Ctrl+Enter.
  4. In the right pane showing the results, copy the value of https: url to a text file so you can use it later as YOUR_TRACE_OBSERVER_URL in Send test data to the trace observer:
    Screen capture showing where to find the url value in the results of a GraphQL mutation.
Step D. Send data to the trace observer

Here's a test that includes a sample payload with one trace and two spans from the same service: Test Service A. Follow these steps to send a test request:

  1. Get or generate your Insert API key so you can use it later in the test.
  2. Copy the following curl request into a text editor:
    curl request
    curl -i -H "Content-Type: application/json" \
        -H "Api-Key: YOUR_INSERT_API_KEY" \
        -H 'Data-Format: newrelic' \
        -H 'Data-Format-Version: 1' \
        -X POST \
        -d '[
            {
                "common": {
                "attributes": {
                    "environment": "staging"
                }
                },
                "spans": [
                {
                    "trace.id": "123456",
                    "id": "ABC",
                    "attributes": {
                        "duration.ms": 12.53,
                        "host": "host123.test.com",
                        "name": "/home",
                        "service.name": "Test Service A"
                    }
                },
                {
                    "trace.id": "123456",
                    "id": "DEF",
                    "attributes": {
                        "duration.ms": 2.97,
                        "host": "host456.test.com",
                        "error.message": "Invalid credentials",
                        "name": "/auth",
                        "parent.id": "ABC",
                        "service.name": "Test Service B"
                    }
                }
                ]
            }
            ]' \
    'YOUR_TRACE_OBSERVER_URL'
    
  3. Insert your own values into the curl request:
    Value Description
    YOUR_INSERT_API_KEY Replace this with your Insert API key (not the same as your personal API key for NerdGraph API explorer)
    YOUR_TRACE_OBSERVER_URL Replace this with the value under https: url from above.
  4. Copy the content of the text editor into a terminal, and then execute the request.
  5. If the test does not return HTTP/1.1 202 Accepted indicating success, check the following and try again:
    • Confirm that you substituted the url (not host) value for YOUR_TRACE_OBSERVER_URL.
    • Confirm that you only have single quotes around the value you inserted for YOUR_TRACE_OBSERVER_URL.
    • Check that you are using the Insert API Key (not a license).
  6. If your test returned HTTP/1.1 202 Accepted, go to New Relic One to see a query of your test data using the span attribute service.name = Test Service A. Because the sample payload contains an error attribute, the error sampler will mark it for keeping. If you modify the payload to remove the error attributes, the random sampler may not choose to keep this particular trace.

    Traces may take up to one minute to be processed by both the trace observer and the Trace API.

JSON requirements and recommendations

Requirements and guidelines for trace JSON using the newrelic format:

  • Each JSON payload is an array of objects.
  • Each object should contain a required spans key.
  • Each object may contain an optional common key. Use this if you want share information across multiple spans in a object.
  • Any keys on a span have precedence over the same key in the common block.
  • The value for spans key is a list of span objects.
  • Certain attributes are required, and must be included either in the optional common block, or in each span.
  • Recommended and custom attributes may be optionally included in a list of key/value pairs under a key named attributes, in the optional common block and/or in each span.

In the following example POST, there are two spans, both of which will have the trace.id 12345 and the custom attribute host: host123.test.com. The first span has no parent.id, so that is the root of the trace; the second span's parent.id points to the ID of the first.

[
  {
    "common": {
      "attributes": {
        "host": "host123.test.com"
      }
    },
    "spans": [
      {
        "trace.id": "12345",
        "id": "abc",
        "attributes": {
          "user.email": "bob@newr.com",
          "service.name": "my-service",
          "duration.ms": 750,
          "name": "my-span"
        }
      },
      {
        "trace.id": "12345",
        "id": "def",
        "attributes": {
          "parent.id": "abc",
          "service.name": "second-service",
          "duration.ms": 750,
          "name": "second-span"
        }
      }
    ]
  }
]

Attributes

Review the sections below describing the attributes for the attributes block of a newrelic formatted JSON POST.

Required attributes

Requests without these attributes will be rejected, and an NrIntegrationError will be generated. These are the only attributes that don't belong under attributes.

attribute description
id
string
Unique identifier for this span.
trace.id
string
Unique identifier shared by all spans within a single trace.
timestamp
long
The span's start time in Epoch milliseconds. Defaults to the current time in UTC timezone.

While not required, these attributes should be included for the best experience with your data. They are included in the attributes section.

attribute default description
duration.ms
float
none Duration of this span in milliseconds.
name
string
none The name of this span.
parent.id
string
none The id of the caller of this span. Value is null if this is the root span. Traces without a root span will not displayed.
service.name
string
none The name of the entity that created this span.

Reserved attributes

These attributes are currently reserved for internal New Relic usage. While they are not explicitly blocked, we recommend not using them.

attribute default description
entity.name
string
service.name This is derived from the service.name attribute.
entity.type
string
service The entity type is assumed to be a service.
entity.guid
string
None The entity.guid is a derived value that uniquely identifies the entity in New Relic's backend.

Other attributes

You can add any arbitrary attributes you want, with the exception of the restricted attributes. For example, you might want to add attributes like customer.id or user.id to help you analyze your trace data.

To learn how to control how spans appear in New Relic (for example, adding errors or setting a span as a datastore span), see Decorate spans.

Explore more about distributed tracing:

For more help

If you need more help, check out these support and learning resources: