Set up the trace observer

If you're following the steps to set up distributed tracing and are continuing with the steps for the advanced option called Infinite Tracing, you need to set up a trace observer. The trace observer is an AWS-based service for Infinite Tracing that provides you with a low-latency and low-cost way to send your telemetry data to New Relic. As it receives your tracing data, the trace observer uses tail-based sampling to decide what to keep and send to New Relic.

Complete the trace observer setup up that fits your types of data sources (services, apps, or entities). You may need to do steps in more than one section if you have a variety of types:

• Services instrumented with APM agents and third-party telemetry tools
• Services instrumented with New Relic browser, mobile, and Lambda
• (Optional) Additional configuration for all service types

Services instrumented with APM agents and third-party telemetry tools

To create a new trace observer if you're using New Relic APM agents or third-party integrations:

1. Go to one.newrelic.com > All capabilities > Infinite Tracing settings.

2. Select an account in the upper-left dropdown. If you have access to multiple accounts, make sure you're in the account where you want Infinite Tracing enabled. If you can't add an observer, it's likely because we allow only one observer per region, per account family.

3. If no trace observers are already present, click New trace observer to add one, fill out the information, and click Create. Note that we offer the following regions:

   • us-east-1 (US)

   • us-west-2 (US)

   • eu-central-1 (EU): Select this region if you are collecting data from EU requests and you have an EU-based New Relic account that reports in EU data centers.
   • eu-west-1 (US): Select this region if you are collecting data from EU requests and you have a US-based New Relic account that reports in US data centers.

   • ap-southeast-1 (US)

   • ap-southeast-2 (US)

   • us-east-2 (US, FedRAMP)

4. Under the Endpoints dropdown, copy the endpoint value under For other integrations and have it ready. This value is referenced in our code examples as YOUR_TRACE_OBSERVER_URL. This is used to test the trace observer, as well as to configure our telemetry integrations and for sending data via the Trace API.

5. If you are setting up an APM agent, you should also copy the endpoint value under For language agents and have it ready. This value is referenced in our code examples as YOUR_TRACE_OBSERVER_HOST. This is used for configuring our language agents to send data to the trace observer.

6. (Optional) To verify the trace observer is working, we recommend sending a sample trace payload. If you're using our Trace API directly, this will help you learn how the API works.

   Send sample payload

   Important

   If you're using Zipkin-format data, see Send Zipkin payload.

   This test sends a sample trace payload with one trace and two spans from a service named Test Service A.

   To send this sample request:

   1. Get the license key for the account you want to report data to and have it ready.

   2. Copy the following curl request into a text editor:

      bash

      Copy

      $
      curl -i -H 'Content-Type: application/json' \
      >
          -H 'Api-Key: YOUR_LICENSE_KEY' \
      >
          -H 'Data-Format: newrelic' \
      >
          -H 'Data-Format-Version: 1' \
      >
          -X POST \
      >
          -d '[
      $
              {
      $
                  "common": {
      $
                      "attributes": {
      $
                          "environment": "staging"
      $
                      }
      $
                  },
      $
                  "spans": [
      $
                      {
      $
                          "trace.id": "123456",
      $
                          "id": "ABC",
      $
                          "attributes": {
      $
                              "duration.ms": 12.53,
      $
                              "host": "host123.example.com",
      $
                              "name": "/home",
      $
                              "service.name": "Test Service A"
      $
                          }
      $
                      },
      $
                      {
      $
                          "trace.id": "123456",
      $
                          "id": "DEF",
      $
                          "attributes": {
      $
                              "error.message": "Invalid credentials",
      $
                              "service.name": "Test Service B",
      $
                              "host": "host456.example.com",
      $
                              "duration.ms": 2.97,
      $
                              "name": "/auth",
      $
                              "parent.id": "ABC"
      $
                          }
      $
                      }
      $
                  ]
      $
              }
      $
          ]' 'YOUR_TRACE_OBSERVER_URL'

   3. Insert your own values into the curl request:

      Value	Description
      YOUR_LICENSE_KEY	Replace this with your ingest license key.
      YOUR_TRACE_OBSERVER_URL	Replace this with the For other integrations endpoint value you copied in a previous step.

   4. Copy the curl request into a terminal and execute it.

   5. The test should return HTTP/1.1 202 Accepted, indicating success. If it does not, check the following common issues:

      • Confirm that you used the For other integrations endpoint value.
      • Confirm you're using single quotes around YOUR_TRACE_OBSERVER_URL.
      • Check that you're using the correct API key.

   6. If your test returned HTTP/1.1 202 Accepted, go to the New Relic UI to see a query of the sample payload data using the span attribute service.name = Test Service A (here's a link for that query). Because the sample payload contains an error attribute, the error sampler will mark it for keeping. If you modify the payload to remove the error attributes, the random sampler may not choose to keep this particular trace.

      Tip

      Traces may take up to one minute to show up in the UI.

7. Bookmark the optional configurations below so you can return to them after you experiment with Infinite Tracing.

8. Return to your original distributed tracing setup instructions for your agent or third-party telemetry tool and insert your new endpoint value in the configuration for each service.

9. If you also have services instrumented with New Relic browser, mobile, and Lambda, follow the steps in that section to select your data sources (services, apps, or entities) in the trace observer.

Services instrumented with New Relic browser, mobile, and Lambda

To create a new trace observer if you're using New Relic browser, mobile, or Lambda, you follow similar steps to those for APM agents and third-party integrations. Where this setup differs is that you don't need to insert a new trace observer endpoint in each service's configuration. Instead, you complete a separate configuration in the trace observer that specifies which data sources (services, apps, or entities) should have their data sent to the Infinite Tracing trace observer.

If you already created a trace observer while setting up another type of service (for example, an APM agent), you can skip the steps about creating the trace observer and go right to Initial trace observer configuration for browser, mobile, and Lambda to select your data sources.

Create a trace observer

If you haven't already set up a trace observer, complete the following:

1. Go to one.newrelic.com > All capabilities > Infinite Tracing settings.

2. Select an account in the upper-left dropdown. If you have access to multiple accounts, make sure you're in the account where you want Infinite Tracing enabled. If you can't add an observer, it's likely because we allow only one observer per region, per account family.

3. If no trace observers are already present, click New trace observer to add one, fill out the information, and click Create. Note that we offer the following regions:

   • us-east-1 (US)

   • us-west-2 (US)

   • eu-central-1 (EU): Select this region if you are collecting data from EU requests and you have an EU-based New Relic account that reports in EU data centers.
   • eu-west-1 (US): Select this region if you are collecting data from EU requests and you have a US-based New Relic account that reports in US data centers.

   • ap-southeast-1 (US)

   • ap-southeast-2 (US)

   • us-east-2 (US, FedRAMP)

4. Under the Endpoints dropdown, copy the endpoint value under For other integrations and have it ready. This value is referenced in our code examples as YOUR_TRACE_OBSERVER_URL. This is used to test the trace observer.

5. (Optional) To verify the trace observer is working, we recommend sending a sample trace payload.

   Send sample payload

   Important

   If you're using Zipkin-format data, see Send Zipkin payload.

   This test sends a sample trace payload with one trace and two spans from a service named Test Service A.

   To send this sample request:

   1. Get the license key for the account you want to report data to and have it ready.

   2. Copy the following curl request into a text editor:

      bash

      Copy

      $
      curl -i -H 'Content-Type: application/json' \
      >
          -H 'Api-Key: YOUR_LICENSE_KEY' \
      >
          -H 'Data-Format: newrelic' \
      >
          -H 'Data-Format-Version: 1' \
      >
          -X POST \
      >
          -d '[
      $
              {
      $
                  "common": {
      $
                      "attributes": {
      $
                          "environment": "staging"
      $
                      }
      $
                  },
      $
                  "spans": [
      $
                      {
      $
                          "trace.id": "123456",
      $
                          "id": "ABC",
      $
                          "attributes": {
      $
                              "duration.ms": 12.53,
      $
                              "host": "host123.example.com",
      $
                              "name": "/home",
      $
                              "service.name": "Test Service A"
      $
                          }
      $
                      },
      $
                      {
      $
                          "trace.id": "123456",
      $
                          "id": "DEF",
      $
                          "attributes": {
      $
                              "error.message": "Invalid credentials",
      $
                              "service.name": "Test Service B",
      $
                              "host": "host456.example.com",
      $
                              "duration.ms": 2.97,
      $
                              "name": "/auth",
      $
                              "parent.id": "ABC"
      $
                          }
      $
                      }
      $
                  ]
      $
              }
      $
          ]' 'YOUR_TRACE_OBSERVER_URL'

   3. Insert your own values into the curl request:

      Value	Description
      YOUR_LICENSE_KEY	Replace this with your license key.
      YOUR_TRACE_OBSERVER_URL	Replace this with the For other integrations endpoint value you copied in a previous step.

   4. Copy the curl request into a terminal and execute it.

   5. The test should return HTTP/1.1 202 Accepted, indicating success. If it does not, check the following common issues:

      • Confirm that you used the For other integrations endpoint value.
      • Confirm you're using single quotes around YOUR_TRACE_OBSERVER_URL.
      • Check that you're using the correct API key.

   6. If your test returned HTTP/1.1 202 Accepted, go to the New Relic UI to see a query of the sample payload data using the span attribute service.name = Test Service A (here's a link for that query). Because the sample payload contains an error attribute, the error sampler will mark it for keeping. If you modify the payload to remove the error attributes, the random sampler may not choose to keep this particular trace.

      Tip

      Traces may take up to one minute to show up in the UI.

Initial trace observer configuration for browser, mobile, and Lambda

Once you have a trace observer, you need to specify which data sources (services, apps, or entities) should have their data sent to the Infinite Tracing trace observer. Before you select your data sources, familiarize yourself with the following:

The procedure below is for using the New Relic UI, but if you prefer a programmatic approach, you can specify data sources with GraphQL. If you use GraphQL, return to this procedure and continue with step 4 (optional configurations) below.

To finish the configuration for browser, mobile, and Lambda in the UI:

1. Return to the trace observer app (one.newrelic.com > All capabilities > Infinite Tracing settings).

2. For your trace observer, go to the end of the row, click the ellipsis icon ..., and then click Edit mobile, browser, lambda sources.

   

3. Select the data sources you want to include, keeping the following in mind:

   • You can use the filter to narrow your options.
   • If your browser app is connected to an APM agent that has multiple names, you can can only choose one name.
   • If an entry is gray, it is already being used by another trace observer.

4. Bookmark the optional configurations below so you can return to them after you experiment with Infinite Tracing.

5. View your Infinite Tracing data in the New Relic distributed tracing UI.

6. If you also have services instrumented with APM agents or third-party telemetry tools follow the steps in that section to ensure their data is reported to New Relic.

(Optional) Additional configuration for all service types

You can control the behavior of the trace observer with various configurations. Here are some of the features you can configure:

• Configure trace observer monitoring: Allows you to see the amount of traces seen and retained.
• Configure span attribute trace filter: Make custom filter rules for spans.
• Infinite Tracing: Configure random trace filter: Adjust the percentage used by the random trace filter.