Integrations: Enable distributed tracing

If you have telemetry data from Istio, Kamon, OpenCensus, and OpenTelemetry, you can view it in the New Relic UI by sending it through one of our telemetry integrations. Since distributed systems can generate a lot of trace data, telemetry tools rely on data sampling (filtering) to find useful data.

Our telemetry integrations offer you two installation options related to sampling:

  • Use your tool’s native sampling (default): This default installation assumes you are relying on your telemetry tool to sample (filter) the data before the integration passes the data to our Trace API. Even though your tool may be sampling data, our Trace API may randomly sample out traces that exceed your rate limit.

  • Use Infinite Tracing (limited release): This approach assumes you turn off sampling in your telemetry tool and then send all your data to a New Relic trace observer in AWS. The trace observer selects the most important and actionable traces using tail-based sampling and then sends your traces to our Trace API. To use this approach, you complete the default installation but then configure a new endpoint for a trace observer.

To start using Infinite Tracing, please go to our sign-up page.

Here's an overview of the steps to enable distributed tracing:

  1. Meet the prerequisites
  2. (Infinite Tracing) Provision a trace observer on New Relic Edge
  3. (Infinite Tracing) Configure your open source integration
  4. View your traces in New Relic One

Step 1. Prerequisites

If you haven't already installed one of these integrations in your services, follow the link for your integration and complete the installation. After the installation, if you are not enabling Infinite Tracing, you can go directly to Step 4. View traces in New Relic One.

Step 2. (Infinite Tracing) Find or create a trace observer endpoint

Infinite Tracing collects your span data in a trace observer, which runs in a cluster of services in AWS called New Relic Edge. Integrations send all your spans to the trace observer so they can be assembled into traces for you to view in New Relic.

Since Infinite Tracing uses a trace observer, you use GraphQL to find an existing trace observer endpoint, or if you can't find one, you need to create a new endpoint.

Complete the following trace observer steps:

2a. Open NerdGraph API explorer

Prepare to execute some GraphQL:

  1. Open NerdGraph API explorer.
  2. In the API key dropdown next to Tools, select your personal API key for your account under one of these options:
    • Select an existing API Key
    • Create a new API Key
    Screen capture showing the dropdown options in the NerdGraph API explorer.
2b. Execute a query to find an existing trace observer endpoint

The following steps show you how to execute a GraphQL query to find out if you can use an existing trace observer in your AWS region. If one isn't available, you can go to Step 2c. Create a new one.

  1. Copy the following query into the middle pane of the NerdGraph API explorer and replace YOUR_ACCOUNT_ID with your account ID (the number next to your account name in NerdGraph API explorer):
    {
      actor {
        account(id: YOUR_ACCOUNT_ID) {
          edge {
            tracing {
              traceObservers {
                errors {
                  message
                  type
                }
                traceObservers {
                  endpoints {
                    agent {
                      host
                      port
                    }
                    endpointType
                    https {
                      host
                      port
                      url
                    }
                    status
                  }
                  id
                  name
                  providerRegion
                }
              }
            }
          }
        }
      }
    }     
            
  2. Click the triangle button to execute the query or press Ctrl+Enter.
  3. Check the right pane showing the results:
2c. If you can't find an existing trace observer endpoint, create a new one

If you didn't find a trace observer after running the query in the previous section, you need to create one. To do this, you execute a GraphQL mutation that passes your configuration details.

  1. Copy the following into the middle pane of the NerdGraph API explorer.
    mutation {
      edgeCreateTraceObserver(accountId: YOUR_ACCOUNT_ID, traceObserverConfigs: {name: "YOUR_DESCRIPTIVE_NAME", providerRegion: YOUR_PROVIDER_REGION}) {
        responses {
          errors {
            message
            type
          }
          traceObserver {
            endpoints {
              agent {
                host
                port
              }
              endpointType
              https {
                host
                port
                url
              }
              status
            }
            id
            name
            providerRegion
          }
        }
      }
    }
                
  2. Insert your own values into the mutation:
    Value Description
    YOUR_ACCOUNT_ID Replace this with your account ID (the number next to your account name in NerdGraph API explorer).
    YOUR_DESCRIPTIVE_NAME Replace this with a name that describes the services that report to this trace observer (for example, production or query services).
    YOUR_PROVIDER_REGION Replace this with desired provider region enum value (currently only AWS_US_EAST_1 is available).
  3. Click the triangle button to execute the mutation or press Ctrl+Enter.
  4. In the right pane showing the results, copy the following endpoint values to a text file so you can use them later:
    Screen capture showing integration mutation results.
2d. (Optional) Send test data to the trace observer

This test includes a sample payload with one trace and two spans from the same service: Test Service A. Follow these steps to send a test request:

  1. Get or generate your Insert API key so you can use it later in the test.
  2. Copy the following curl request into a text editor:
    curl request
    curl -i -H "Content-Type: application/json" \
        -H "Api-Key: YOUR_INSERT_API_KEY" \
        -H 'Data-Format: newrelic' \
        -H 'Data-Format-Version: 1' \
        -X POST \
        -d '[
            {
                "common": {
                "attributes": {
                    "environment": "staging"
                }
                },
                "spans": [
                {
                    "trace.id": "123456",
                    "id": "ABC",
                    "attributes": {
                        "duration.ms": 12.53,
                        "host": "host123.test.com",
                        "name": "/home",
                        "service.name": "Test Service A"
                    }
                },
                {
                    "trace.id": "123456",
                    "id": "DEF",
                    "attributes": {
                        "duration.ms": 2.97,
                        "host": "host456.test.com",
                        "error.message": "Invalid credentials",
                        "name": "/auth",
                        "parent.id": "ABC",
                        "service.name": "Test Service B"
                    }
                }
                ]
            }
            ]' \
    'YOUR_TRACE_OBSERVER_URL'
    
  3. Insert your own values into the curl request:
    Value Description
    YOUR_INSERT_API_KEY Replace this with your Insert API key (not the same as your personal API key for NerdGraph API explorer)
    YOUR_TRACE_OBSERVER_URL Replace this with the value under https: url from above.
  4. Copy the content of the text editor into a terminal, and then execute the request.
  5. If the test does not return HTTP/1.1 202 Accepted indicating success, check the following and try again:
    • Confirm that you substituted the url (not host) value for YOUR_TRACE_OBSERVER_URL.
    • Confirm that you only have single quotes around the value you inserted for YOUR_TRACE_OBSERVER_URL.
    • Check that you are using the Insert API Key (not a license).
  6. If your test returned HTTP/1.1 202 Accepted, go to New Relic One to see a query of your test data using the span attribute service.name = Test Service A. Because the sample payload contains an error attribute, the error sampler will mark it for keeping. If you modify the payload to remove the error attributes, the random sampler may not choose to keep this particular trace.

    Traces may take up to one minute to be processed by both the trace observer and the Trace API.

Step 3. (Infinite Tracing) Configure the integration

To enable Infinite Tracing, configure your integration to send the telemetry data to the trace observer by using the results from Step 2. Find or create a trace observer endpoint.

Find the section below describing these steps for your integration:

Istio adapter

Set the spansHost value with YOUR_TRACE_OBSERVER_URL when deploying the Helm chart.

Kamon reporter

Set the span-ingest-uri value with YOUR_TRACE_OBSERVER_URL in your kamon.newrelic configuration block.

OpenCensus (Go language exporter)

Set the SpansURLOverride field on the Config object with YOUR_TRACE_OBSERVER_URL when creating the Exporter.

OpenCensus (Python language exporter)

Pass the host and port parameters to the Trace Exporter using YOUR_TRACE_OBSERVER_HOST and YOUR_TRACE_OBSERVER_PORT.

OpenTelemetry (Go language exporter)

Set the SpansURLOverride field on the Config object with YOUR_TRACE_OBSERVER_URL when creating the Exporter.

OpenTelemetry (Java language exporter)

Complete the following:

  1. Create a java.net.URI with YOUR_TRACE_OBSERVER_URL.
  2. Pass the URI to com.newrelic.telemetry.opentelemetry.export.NewRelicSpanExporter builder’s uriOverride method.

See an example where a NewRelicSpanExporter is created.

OpenTelemetry (.NET language exporter)

Configure the TraceUrlOverride parameter with YOUR_TRACE_OBSERVER_URL.

Step 4. View traces in New Relic One

After you configure your integration to send data to New Relic, you are ready to view traces. On the New Relic One home page, there are two launchers that let you view distributed traces:

Entity explorer

From one.newrelic.com, select the Entity explorer to be able to navigate to a specific service and view traces that include that service.

  1. Go to New Relic One.
  2. Click the Entity explorer launcher.
  3. Filter to the service you enabled for Infinite Tracing by typing the service name, and then press Enter.
  4. In the left navigation, click Distributed tracing under the Troubleshoot section.
Distributed tracing

From one.newrelic.com, select the Distributed tracing launcher to search all traces across all New Relic accounts in your organization that you have access to.

  1. Go to New Relic One.
  2. Click the Distributed tracing launcher.
  3. In the Find traces… search, type a search clause to find the service. For example, you can query service.name or trace.id:
    service.name = YOUR_SERVICE_NAME
    
    trace.id = YOUR_TRACE_ID
    

For more help

Recommendations for learning more: