Set up the trace observer

If you're following the Enable distributed tracing procedures and want to set up Infinite Tracing, you'll need to set up a trace observer.

A trace observer is a cloud-based New Relic tool that decides what trace data to keep and send to New Relic. The trace observer lives in New Relic Edge, which is an AWS-based service that provides you with a low-latency and low-cost way to send your telemetry data to New Relic. The trace observer sends data via our Trace API, which is the entry point for all distributed trace data we ingest.

This documentation is for our Infinite Tracing feature. To learn about all our distributed tracing options, see Intro to distributed tracing.

Set up the trace observer

Before setting up a trace observer, understand these points:

  • With the exception of the Trace API, these instructions are not standalone; they're part of larger enable procedures. If you're still figuring out what you need, see Enable distributed tracing.
  • To avoid configuration conflict issues, you should ideally enable Infinite Tracing for all associated services. If some services in a trace have our standard distributed tracing enabled, you should upgrade those to Infinite Tracing.

To set up a trace observer:

  1. Go to one.newrelic.com, and click Apps. Under Your apps, click New Relic Edge.
  2. Select an account in the upper-left dropdown. If you have access to multiple accounts, make sure you're in the account where you want Infinite Tracing enabled.
  3. If no trace observers are already present, click New trace observer to add one, fill out the information, and click Create.

    Note: If you select a trace observer in an EU region, you’ll still need a US-based New Relic account because data is reported to US data centers.

  4. Under the Endpoints dropdown:
    • Copy the For other integrations endpoint value and have it ready: this will be referred to in later instructions as YOUR_TRACE_OBSERVER_URL.
    • If you're enabling a language agent, also copy the For language agents value and have it ready: this will be referred to as YOUR_TRACE_OBSERVER_HOST.
  5. (Optional but recommended) To verify things are working, we recommend sending a sample trace payload. If you're using our Trace API: this step is especially recommended to learn how the API works.
    Send sample payload

    If you're using Zipkin-format data, see Send Zipkin payload.

    This test sends a sample trace payload with one trace and two spans from a service named Test Service A.

    To send this sample request:

    1. Get or generate your Insights insert key and have it ready.
    2. Copy the following curl request into a text editor:
      curl -i -H "Content-Type: application/json" \
        -H "Api-Key: $YOUR_INSERT_KEY" \
        -H 'Data-Format: newrelic' \
        -H 'Data-Format-Version: 1' \
        -X POST \
        -d '[
            {
                "common": {
                "attributes": {
                    "environment": "staging"
                }
                },
                "spans": [
                {
                    "trace.id": "123456",
                    "id": "ABC",
                    "attributes": {
                        "duration.ms": 12.53,
                        "host": "host123.test.com",
                        "name": "/home",
                        "service.name": "Test Service A"
                    }
                },
                {
                    "trace.id": "123456",
                    "id": "DEF",
                    "attributes": {
                        "duration.ms": 2.97,
                        "host": "host456.test.com",
                        "error.message": "Invalid credentials",
                        "name": "/auth",
                        "parent.id": "ABC",
                        "service.name": "Test Service B"
                    }
                }
                ]
            }
            ]' \
      '$YOUR_TRACE_OBSERVER_URL'
      
    3. Insert your own values into the curl request:
      Value Description
      $YOUR_INSERT_KEY Replace this with your Insights insert API key.
      $YOUR_TRACE_OBSERVER_URL Replace this with the For other integrations endpoint value you copied in a previous step.
    4. Copy the curl request into a terminal and execute it.
    5. The test should return HTTP/1.1 202 Accepted, indicating success. If it does not, check the following common issues:
      • Confirm that you used the For other integrations endpoint value.
      • Confirm you're using single quotes around YOUR_TRACE_OBSERVER_URL.
      • Check that you're using the correct API key.
    6. If your test returned HTTP/1.1 202 Accepted, go to the New Relic UI to see a query of the sample payload data using the span attribute service.name = Test Service A (here's a link for that query). Because the sample payload contains an error attribute, the error sampler will mark it for keeping. If you modify the payload to remove the error attributes, the random sampler may not choose to keep this particular trace.

      Traces may take up to one minute to show up in the UI.

  6. (Optional) There are several ways to configure Infinite Tracing. This configuration can wait until after you've completed the enable procedures.
  7. This procedure is complete. Next, return to finish any remaining instructions for the tracing tool you started enabling:

Trace observer endpoints

In the trace observer UI, there's an Endpoints dropdown. When setting up the trace observer, we have you copy these values for use at various points of our tracing tool setup instructions. There are two values:

For more help

If you need more help, check out these support and learning resources: