Language agents: Enable distributed tracing

This document explains how to enable standard distributed tracing or Infinite Tracing with New Relic language agents.

  • To learn more about distributed tracing options before enabling it, see How span sampling works.
  • If you are a current New Relic APM user and want to enable distributed tracing, first read Impacts to APM.

Here's an overview of the steps to enable distributed tracing:

  1. Meet the prerequisites
  2. (Infinite Tracing): Find or create a trace observer endpoint
  3. Configure your agents
  4. View traces in New Relic One

Step 1. Prerequisites

Check the following sections to confirm that the distributed tracing options you want are available for your environment. Your main options are:

To start using Infinite Tracing, please go to our sign-up page.

Once you find the agent version with the features you want, follow the links for help installing or updating your agent. For best results, update existing agents to the latest version.

C SDK

Install (compile) or update to the required C SDK version. For best results, update to the latest C SDK version.

Option C SDK version
Standard distributed tracing

1.1.0 or higher (W3C Trace Context not available)
Infinite Tracing Not available
Go

Install or update to the required Go agent version. For best results, update to the latest Go agent version.

Option Go agent version
Standard distributed tracing

2.1.0 or higher

With W3C Trace Context: 3.1.0 or higher
Infinite Tracing

v3.5.0 (includes W3C Trace Context)

Supported environments: Go 1.9 or higher
Java

Install or update to the required Java agent version. For best results, update to the latest Java agent version.

Type Java agent version
Standard distributed tracing

4.3.0 or higher

With W3C Trace Context: 5.10 or higher
Infinite Tracing

5.12.1 or higher (includes W3C Trace Context)

Supported environments:

  • Java 8: Update 252 or higher
  • All versions of Java 9 or higher

For special considerations, see Infinite Tracing: Configuring SSL for Java 7 and 8.
.NET

Install or update to the required .NET agent version. For best results, update to the latest .NET agent version.

Option .NET agent version
Standard distributed tracing

8.6.45.0 or higher

With W3C Trace Context: 8.27.139.0 or higher
Infinite Tracing

8.26.630.0 (includes W3C Trace Context)

Supported environments:

  • .NET Framework 4.5 or higher
  • .NET Core 2.0 or higher
Node.js

Install or update to the required Node.js agent version. For best results, update to the latest Node.js agent version.

Option Node.js agent version
Standard distributed tracing

4.7.0 or higher

With W3C Trace Context: 6.4 or higher
Infinite Tracing

6.7.0 (includes W3C Trace Context)

Supported environments: Node version 10.10.0 or higher
PHP

Install or update to the required PHP agent version. For best results, update to the latest PHP agent version.

Option PHP agent version
Standard distributed tracing

8.4 or higher

With W3C Trace Context: 9.8 or higher
Infinite Tracing Not available
Python

Install or update to the required Python agent version. For best results, update to the latest Python agent version.

Option Python agent version
Standard distributed tracing

4.2.0.100 or higher

With W3C Trace Context: 5.6 or higher
Infinite Tracing

5.12.0.140 (includes W3C Trace Context)

Supported environments: CPython only (pypy is unsupported)
Ruby

Install or update to the required Ruby agent version. For best results, update to the latest Ruby agent version.

Also install the additional Ruby agent gem for Infinite Tracing.

Option Ruby agent version
Standard distributed tracing

newrelic_rpm 5.3.0.346 or higher

With W3C Trace Context: newrelic_rpm 6.9 or higher
Infinite Tracing

newrelic_rpm 6.11.0.365 or higher (includes W3C Trace Context)

newrelic-infinite_tracing 6.11.0.375 or higher

Step 2. (Infinite Tracing) Find or create a trace observer endpoint

This section only applies to Infinite Tracing. If you are enabling standard distributed tracing, continue to Configure the agents.

Infinite Tracing collects your span data in a trace observer, which runs in a cluster of services in AWS called New Relic Edge. Agents send all your spans to the trace observer so they can be assembled into traces for you to view in New Relic.

Since Infinite Tracing uses a trace observer, you use GraphQL to find an existing trace observer endpoint. If you can't find one, you need to create a new endpoint.

Complete the following trace observer steps in order:

2a. Open NerdGraph API explorer

Prepare to execute some GraphQL:

  1. Open NerdGraph API explorer.

  2. In the API key dropdown next to Tools, select your personal API key for your account under one of these options:

    • Select an existing API key
    • Create a new API key
    Screen capture showing the dropdown options in the NerdGraph API explorer.
2b. Execute a query to find an existing trace observer endpoint

The following steps show you how to execute a GraphQL query to find out if you can use an existing trace observer in your AWS region. If one isn't available, then you can go to Step 2c. Create a new one.

  1. Copy the following query into the middle pane of the NerdGraph API explorer and replace YOUR_ACCOUNT_ID with your account ID (the number next to your account name in NerdGraph API explorer): 
    {
  actor {
    account(id: YOUR_ACCOUNT_ID) {
      edge {
        tracing {
          traceObservers {
            errors {
              message
              type
            }
            traceObservers {
              endpoints {
                agent {
                  host
                  port
                }
                endpointType
                https {
                  host
                  port
                  url
                }
                status
              }
              id
              name
              providerRegion
            }
          }
        }
      }
    }
  }
}
  2. Click the triangle button to execute the query, or press Ctrl+Enter.

  3. Check the right pane showing the results:

2c. If you can't find an existing trace observer endpoint, create a new one

If you didn't find a trace observer endpoint after running the query in the previous section, you need to create one. To do this, execute a GraphQL mutation that passes your configuration details.

  1. Copy the following into the middle pane of the NerdGraph API explorer.

    mutation {
  edgeCreateTraceObserver(accountId: YOUR_ACCOUNT_ID, traceObserverConfigs: {name: "YOUR_DESCRIPTIVE_NAME", providerRegion: YOUR_PROVIDER_REGION}) {
    responses {
      errors {
        message
        type
      }
      traceObserver {
        endpoints {
          agent {
            host
            port
          }
          endpointType
          https {
            host
            port
            url
          }
          status
        }
        id
        name
        providerRegion
      }
    }
  }
}

  2. Insert your own values into the mutation:

    Value Description
    YOUR_ACCOUNT_ID Replace this with your account ID (the number next to your account name in NerdGraph API explorer).
    YOUR_DESCRIPTIVE_NAME Replace this with a name that describes the services that report to this trace observer (for example, production or query services).
    YOUR_PROVIDER_REGION Replace this with the provider region enum value (currently only AWS_US_EAST_1 is available).
  3. Click the triangle button to execute the mutation, or press Ctrl+Enter.

  4. In the right pane showing the results, copy the following endpoints to a text file so you can use them later:

    Screen capture showing agent mutation query to find endpoints.
2d. (Optional) Send test data to the trace observer

This optional test includes a sample payload with one trace and two spans from the same service: Test Service A. Follow these steps to send a test request:

  1. Get or generate your Insert API key so you can use it later in the configuration (Note that the Insert API key is not your license).

    1. Copy the following curl request into a text editor:

      curl request
      curl -i -H "Content-Type: application/json" \
    -H "Api-Key: YOUR_INSERT_API_KEY" \
    -H 'Data-Format: newrelic' \
    -H 'Data-Format-Version: 1' \
    -X POST \
    -d '[
        {
            "common": {
            "attributes": {
                "environment": "staging"
            }
            },
            "spans": [
            {
                "trace.id": "123456",
                "id": "ABC",
                "attributes": {
                    "duration.ms": 12.53,
                    "host": "host123.test.com",
                    "name": "/home",
                    "service.name": "Test Service A"
                }
            },
            {
                "trace.id": "123456",
                "id": "DEF",
                "attributes": {
                    "duration.ms": 2.97,
                    "host": "host456.test.com",
                    "error.message": "Invalid credentials",
                    "name": "/auth",
                    "parent.id": "ABC",
                    "service.name": "Test Service B"
                }
            }
            ]
        }
        ]' \
'YOUR_TRACE_OBSERVER_URL'

  2. Insert your own values into the curl request:

    Value Description
    YOUR_INSERT_API_KEY Replace this with your Insert API key (not your New Relic license key).
    YOUR_TRACE_OBSERVER_URL Replace this with the value under https: url from the previous step.
  3. Copy the content of the text editor into a terminal and execute the request.

  4. If the test does not return HTTP/1.1 202 Accepted indicating success, check the following and try again:

    • Confirm that you substituted the url (not host) value for YOUR_TRACE_OBSERVER_URL.
    • Confirm that you only have single quotes around the value you inserted for YOUR_TRACE_OBSERVER_URL.
    • Check that you are using the Insert API key (not a license key).
  5. If your test returned HTTP/1.1 202 Accepted, go to New Relic One to see a query of your test data using the span attribute service.name = Test Service A. Because the sample payload contains an error attribute, the error sampler will mark it for keeping. If you modify the payload to remove the error attributes, the random sampler may not choose to keep this particular trace.

Traces may take up to one minute to be processed by both the trace observer and the Trace API.

Step 3. Configure the agents

Distributed tracing is enabled through configuration settings. The settings you need to change for distributed tracing depend on the type you want:

  • Standard distributed tracing
  • Infinite Tracing (requires YOUR_TRACE_OBSERVER_HOST from step 2)

Review the following section for your agent to see the required configuration settings. For more information, see Configure the agent.

Server-side configuration is not available for Infinite Tracing.

C SDK

Here's an overview of the settings. For more help with configuration, see Enable distributed tracing for your C applications.

Type Required configuration
Standard distributed tracing

Configuration examples:

  • newrelic_app_config_t structure: 
    newrelic_app_config_t* config;
config = newrelic_create_app_config(app_name, license_key);
config->distributed_tracing.enabled = true;
Infinite Tracing Not available
Go

Here's an overview of the settings. For more help with configuration, see Enable distributed tracing for your Go applications.

Type Required configuration
Standard distributed tracing

Configuration examples:

  • ConfigOption structure:

    newrelic.NewApplication(
  newrelic.ConfigAppName("Example App"),
  newrelic.ConfigLicense(os.Getenv("NEW_RELIC_LICENSE_KEY")),
  newrelic.ConfigDistributedTracerEnabled(true),
)

  • Environment variable:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Configuration examples:

  • newrelic.Config structure:

    app, err := newrelic.NewApplication(
    newrelic.ConfigAppName(YOUR_APP_NAME),
    newrelic.ConfigLicense(YOUR_LICENSE_KEY),
    func(cfg *newrelic.Config) {
        cfg.DistributedTracer.Enabled = true
        cfg.InfiniteTracing.TraceObserver.Host = YOUR_TRACE_OBSERVER_HOST
    },
)

  • Environment variables:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST=YOUR_TRACE_OBSERVER_HOST
Java

Here's an overview of the settings. For more help with configuration, see Java agent configuration: Config file.

Type Required configuration
Standard distributed tracing

Configuration examples:

  • Configuration file (newrelic.yml) (indented 2 spaces under the common stanza):

      distributed_tracing:
    enabled: true

  • Java system property:

    -Dnewrelic.config.distributed_tracing.enabled=true

  • Environment variable:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Configuration examples:

  • Configuration file (newrelic.yml) (indented 2 spaces under the common stanza):

      distributed_tracing:
    enabled: true
  infinite_tracing:
    trace_observer:
      host: "YOUR_TRACE_OBSERVER_HOST"

  • Java system property:

    -Dnewrelic.config.distributed_tracing.enabled=true
-Dnewrelic.config.infinite_tracing.trace_observer.host="YOUR_TRACE_OBSERVER_HOST"

  • Environment variable:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
.NET

Here's an overview of the settings. For more help with configuration, see .NET agent configuration.

Type Required configuration
Standard distributed tracing

Configuration examples:

  • Configuration file (newrelic.config): 
    <configuration . . . >
   <distributedTracing enabled="true" />
</configuration>
  • Environment variable: 
    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Configuration examples:

  • Configuration file (newrelic.config): 
    <configuration . . . >
   <distributedTracing enabled="true" />
   <infiniteTracing>
      <trace_observer host="YOUR_TRACE_OBSERVER_HOST" />
   </infiniteTracing>
</configuration>

  • Environment variable:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
Node.js

Here's an overview of the settings. For more help with configuration, see Node.js agent configuration.

Type Required configuration
Standard distributed tracing

Configuration examples:

Configuration file (newrelic.js):

distributed_tracing: {
  enabled: true
}

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Configuration examples:

Configuration file (newrelic.js):

distributed_tracing: {
  enabled: true
}
infinite_tracing: {
  trace_observer: {
    host: 'YOUR_TRACE_OBSERVER_HOST'
  }
}

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
PHP

Here's an overview of the settings. For more help with configuration, see Distributed tracing for the PHP agent

Type Required configuration
Standard distributed tracing

Configuration examples:

  • Configuration file (newrelic.ini):

    newrelic.distributed_tracing_enabled = true
Infinite Tracing Not available
Python

Here's an overview of the settings. For more help with configuration, see Python agent configuration

Type Required configuration
Standard distributed tracing

Configuration file (newrelic.ini):

distributed_tracing.enabled = true

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Use the following installation command:

pip install newrelic[infinite-tracing]

Configuration file (newrelic.ini):

distributed_tracing.enabled = true
infinite_tracing.trace_observer_host= YOUR_TRACE_OBSERVER_HOST

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
Ruby

Here's an overview of the settings. For more help with configuration, see Ruby agent configuration.

Type Required configuration
Standard distributed tracing

Configuration examples:

  • Configuration file (newrelic.yml):

      distributed_tracing:
        enabled: true
  • Environment variable: 
    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Configuration examples:

  • Configuration file (newrelic.yml):

      distributed_tracing: 
      enabled: true
  infinite_tracing: 
      trace_observer: 
          host: 'YOUR_TRACE_OBSERVER_HOST'

  • Environment variable:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"

Step 4. View traces in New Relic One

After you upgrade and configure your APM agents to send data to your trace observer, you are ready to view traces. On the New Relic One home page, there are two launchers that let you view distributed traces:

Entity explorer

From one.newrelic.com, select the Entity explorer to be able to navigate to a specific service and view traces that include that service.

  1. Go to one.newrelic.com.
  2. Click the Entity explorer launcher.
  3. Filter to the service you enabled for Infinite Tracing by typing the service name, and then press Enter.
  4. In the left navigation's Troubleshoot section, click Distributed tracing.
Distributed tracing

From one.newrelic.com, select the Distributed tracing launcher. This allows you to search all traces across all New Relic accounts in your organization that you have access to use.

  1. Go to one.newrelic.com.
  2. Click the Distributed tracing launcher.

  3. In the Find traces... search, type a search clause to find the service. For example, to query service.name or trace.id:

    service.name = YOUR_SERVICE_NAME
trace.id = YOUR_TRACE_ID

Manual instrumentation (If automatic instrumentation doesn't work)

Recommendation: Before performing any custom instrumentation, read:

If a service is not passing the trace header to other services, you can use the distributed tracing payload APIs to instrument the calling service and the called service. The calling service uses an API call to generate a payload, which is accepted by the called service.

Instrument the calling service

To instrument the calling service:

  1. Ensure the version of the APM agent that monitors the calling service supports distributed tracing.

  2. Invoke the agent API call for generating a distributed trace payload: C SDK | Go | Java | .NET | Node.js | PHP | Python | Ruby.

    To maintain proper ordering of spans in a trace, ensure you generate the payload in the context of the span that sends it.

  3. Add that payload to the call made to the destination service (for example, in a header).

  4. (Optional) Identify the call as an external call:

Instrument the called service

To instrument the called service:

  1. Ensure the version of the APM agent that monitors the called service supports distributed tracing.

  2. If the New Relic agent on the called service does not identify a New Relic transaction, use the agent API to declare a transaction:

    C SDK

    One way to tell that a transaction is not in progress: when newrelic_create_distributed_trace_payload() is called, a NULL pointer is returned. To solve this problem, follow the procedures to create a transaction with the C SDK.

    Go

    One way to tell that a transaction is not in progress: when Transaction.InsertDistributedTraceHeaders(h http.Header) is called, no headers are inserted. To create a transaction, see Instrument Go transactions.

    Java

    One way to tell that a transaction is not in progress: when createDistributedTracePayload() is called, an empty string is returned. To create a transaction, see Java agent transaction-related APIs.

    .NET

    One way to tell that a transaction is not in progress: CreateDistributedTracePayload() returns an empty payload. To create a transaction, see Introduction to .NET custom instrumentation.

    Node.js

    One way to tell that a transaction is not in progress: the Node.js agent logs will report an error similar to this:

    No transaction found when calling Transaction.acceptDistributedTracePayload.

    Use startWebTransaction to create a web transaction or startBackgroundTransaction to capture a non-web transaction.

    PHP

    One way to tell that a transaction is not in progress: newrelic_insert_distributed_trace_headers() returns false. To create a transaction, see newrelic_start_transaction.

    Python

    To tell that a transaction is not in progress: when transaction = current_transaction() is run, transaction is None. Or, if result = accept_distributed_trace_payload(payload) is run, then the result is False.

    Use background_task to report a non-web transaction. For more on Python instrumentation, see Monitor transactions and segments.

    Ruby

    If you are using a Rack-based web framework and have enabled New Relic's Rack instrumentation, the Ruby agent will handle starting a transaction for you. For other use cases, see the add_transaction_tracer API method.

  3. Extract the payload from the call that you received (for example, in a header).

  4. Invoke the call for accepting the payload: C SDK | Go | Java | .NET | PHP | Node.js | Python | Ruby.

For more help

Recommendations for learning more: