Integrations: Enable distributed tracing

If you have telemetry data from Istio, Kamon, OpenCensus, and OpenTelemetry, you can view it in the New Relic UI by sending it through one of our telemetry integrations.

To integrate your AWS X-Ray telemetry data, use the infrastructure integration.

Since distributed systems can generate a lot of trace data, telemetry tools rely on data sampling (filtering) to find useful data. Our telemetry integrations offer you two sampling options:

  • Use your tool’s native sampling (default): The default installation assumes you are relying on your telemetry tool to sample the data before the integration passes the data to our Trace API. Even though your tool may be sampling data, our Trace API may randomly sample out traces that exceed your rate limit.

  • Use Infinite Tracing: This approach assumes you set sampling to 100% in your telemetry tool so you send all your data to a New Relic trace observer in AWS. The trace observer selects the most important and actionable traces using tail-based sampling and then sends your traces to our Trace API. To use this approach, complete the default installation but then configure a new endpoint for a trace observer.

Here's an overview of the steps to enable distributed tracing:

  1. Install the integration
  2. (Infinite Tracing) Provision a trace observer on New Relic Edge
  3. (Infinite Tracing) Configure your open source integration
  4. View your traces in New Relic One

Step 1. Install the integration

If you haven't already installed one of these integrations in your services, follow the link for your integration and complete the installation. After the installation, if you are not enabling Infinite Tracing, skip to Step 4. View traces in New Relic One.

Step 2. (Infinite Tracing) Set up a trace observer

This section only applies to Infinite Tracing. If you're enabling standard distributed tracing, skip to Step 4. View traces in New Relic One.

Infinite Tracing requires you to provision a trace observer that receives your spans. The trace observer examines the spans and keeps the actionable ones. You provision a trace observer in a cluster of services in AWS called New Relic Edge.

If you send your data to a trace observer in one of our EU provider regions, you'll still need a US-based New Relic account because the tail-based sampling data is reported to data centers in the United States. If you have questions, contact your account representative.

Use our Edge app to set up the trace observer. This app also includes some optional configurations: trace observer monitoring and span attribute trace filter. If you're not sure whether you want to configure these, you can set them up later.

Complete the following:

2a. Set up the trace observer endpoint
  1. Go to one.newrelic.com.
  2. In the menu bar, click Apps, expand Your apps, and then click New Relic Edge.
  3. Select an account in the upper-left dropdown.

    If you have access to multiple account hierarchies, make sure you're in the hierarchy where you want a trace observer.

  4. If no trace observers are listed, click New trace observer to add one, insert a descriptive name, select a region, and then click Create.
  5. For the trace observer you want to use, go to the Endpoints dropdown and click the adjacent clipboard icon to copy these values:
    Users Endpoints dropdown value Description
    For all users (Including OpenCensus Python) For other integrations Copy and hold to use later as YOUR_TRACE_OBSERVER_URL in:
    Only for OpenCensus Python For language agents Copy and hold to use later as YOUR_TRACE_OBSERVER_HOST in Step 3. (Infinite Tracing) Configure the integration.
  6. (Optional) If you want to start gathering trace observer metrics, click the toggle for Trace observer monitoring. For details, see trace observer monitoring.
  7. (Optional) If you want to customize the span attribute trace filter, click the gear icon below Trace filters to see the current rules. For details, see span attribute filter.
2b. (Optional) Send test data to the trace observer

This test includes a sample payload with one trace and two spans from the same service: Test Service A. Follow these steps to send a test request:

  1. Get or generate your Insert API key so you can use it later in the test.
  2. Copy the following curl request into a text editor:
    curl request
    curl -i -H "Content-Type: application/json" \
        -H "Api-Key: YOUR_INSERT_API_KEY" \
        -H 'Data-Format: newrelic' \
        -H 'Data-Format-Version: 1' \
        -X POST \
        -d '[
            {
                "common": {
                "attributes": {
                    "environment": "staging"
                }
                },
                "spans": [
                {
                    "trace.id": "123456",
                    "id": "ABC",
                    "attributes": {
                        "duration.ms": 12.53,
                        "host": "host123.test.com",
                        "name": "/home",
                        "service.name": "Test Service A"
                    }
                },
                {
                    "trace.id": "123456",
                    "id": "DEF",
                    "attributes": {
                        "duration.ms": 2.97,
                        "host": "host456.test.com",
                        "error.message": "Invalid credentials",
                        "name": "/auth",
                        "parent.id": "ABC",
                        "service.name": "Test Service B"
                    }
                }
                ]
            }
            ]' \
    'YOUR_TRACE_OBSERVER_URL'
    
  3. Insert your own values into the curl request:
    Value Description
    YOUR_INSERT_API_KEY Replace this with your Insert API key (not the same as your personal API key for NerdGraph API explorer)
    YOUR_TRACE_OBSERVER_URL Replace this with the value you copied in 2a. Set up the trace observer endpoint.
  4. Copy the content of the text editor into a terminal, and then execute the request.
  5. If the test does not return HTTP/1.1 202 Accepted indicating success, check the following and try again:
    • Confirm that you used the Edge app value For other integrations as YOUR_TRACE_OBSERVER_URL.
    • Confirm that you only have single quotes around the value you inserted for YOUR_TRACE_OBSERVER_URL.
    • Check that you are using the Insert API Key (not a license).
  6. If your test returned HTTP/1.1 202 Accepted, go to New Relic One to see a query of your test data using the span attribute service.name = Test Service A. Because the sample payload contains an error attribute, the error sampler will mark it for keeping. If you modify the payload to remove the error attributes, the random sampler may not choose to keep this particular trace.

    Traces may take up to one minute to be processed by both the trace observer and the Trace API.

Step 3. (Infinite Tracing) Configure the integration

To enable Infinite Tracing, configure your integration to send the telemetry data to the trace observer by using the results from Step 2. Find or create a trace observer endpoint.

Find the section below describing these steps for your integration:

Istio adapter

Set the spansHost value with YOUR_TRACE_OBSERVER_URL when deploying the Helm chart.

Kamon reporter

Set the span-ingest-uri value with YOUR_TRACE_OBSERVER_URL in your kamon.newrelic configuration block.

OpenCensus (Go language exporter)

Set the SpansURLOverride field on the Config object with YOUR_TRACE_OBSERVER_URL when creating the Exporter.

OpenCensus (Python language exporter)

Pass the host parameter to the Trace Exporter using YOUR_TRACE_OBSERVER_HOST.

OpenTelemetry (Go language exporter)

Set the SpansURLOverride field on the Config object with YOUR_TRACE_OBSERVER_URL when creating the Exporter.

OpenTelemetry (Java language exporter)

Complete the following:

  1. Create a java.net.URI with YOUR_TRACE_OBSERVER_URL.
  2. Pass the URI to com.newrelic.telemetry.opentelemetry.export.NewRelicSpanExporter builder’s uriOverride method.

See an example where a NewRelicSpanExporter is created.

OpenTelemetry (.NET language exporter)

Configure the TraceUrlOverride parameter with YOUR_TRACE_OBSERVER_URL.

Step 4. View traces

After you configure your integration to send data to New Relic, you are ready to view traces. Here are two alternatives:

View traces that include a specific service

The Entity explorer helps you navigate to a specific service so you can see traces that include that service.

  1. Go to one.newrelic.com.
  2. Click Entity explorer in the top menu bar.
  3. Filter to the service you enabled for Infinite Tracing by typing the service name, and then press Enter.
  4. In the left navigation's Monitor section, click Distributed tracing.
View traces across accounts

This option allows you to search all traces across all New Relic accounts in your organization that you have access to.

  1. Go to one.newrelic.com.
  2. Click Apps in the top menu bar.
  3. Under Favorites click Distributed tracing.
  4. In the Find traces... search, type a search clause to find the service. For example, to query service.name or trace.id:

    service.name = YOUR_SERVICE_NAME
    trace.id = YOUR_TRACE_ID

For more help

If you need more help, check out these support and learning resources: