Language agents: Enable distributed tracing

Here's what you need to know for enabling either standard distributed tracing or Infinite Tracing with our language agents.

If you want to get more background before getting started, check out these topics:

The process of enabling distributed tracing takes you through these steps:

  1. Meet the prerequisites
  2. (Infinite Tracing) Set up a trace observer
  3. Configure your agents
  4. View traces

Step 1. Prerequisites

Check the following sections to confirm that the distributed tracing options you want are available for your environment. Your main options are:

Once you find the agent version with the features you want, follow the links for help installing or updating your agent. For best results, update existing agents to the latest version.

C SDK

Install (compile) or update to the required C SDK version. For best results, update to the latest C SDK version.

Option C SDK version
Standard distributed tracing

1.1.0 or higher (W3C Trace Context not available)
Infinite Tracing Not available
Go

Install or update to the required Go agent version. For best results, update to the latest Go agent version.

Option Go agent version
Standard distributed tracing

2.1.0 or higher

With W3C Trace Context: 3.1.0 or higher
Infinite Tracing

v3.5.0 (includes W3C Trace Context)

Supported environments: Go 1.9 or higher
Java

Install or update to the required Java agent version. For best results, update to the latest Java agent version.

Type Java agent version
Standard distributed tracing

4.3.0 or higher

With W3C Trace Context: 5.10 or higher
Infinite Tracing

5.12.1 or higher (includes W3C Trace Context)

Supported environments:

  • Java 8: Update 252 or higher
  • All versions of Java 9 or higher

For special considerations, see Infinite Tracing: Configuring SSL for Java 7 and 8.
.NET

Install or update to the required .NET agent version. For best results, update to the latest .NET agent version.

Option .NET agent version
Standard distributed tracing

8.6.45.0 or higher

With W3C Trace Context: 8.27.139.0 or higher
Infinite Tracing

8.30.0 (includes W3C Trace Context)

Supported environments:

  • .NET Framework 4.5 or higher
  • .NET Core 2.0 or higher
Node.js

Install or update to the required Node.js agent version. For best results, update to the latest Node.js agent version.

Option Node.js agent version
Standard distributed tracing

4.7.0 or higher

With W3C Trace Context: 6.4 or higher
Infinite Tracing

6.7.0 (includes W3C Trace Context)

Supported environments: Node version 10.10.0 or higher
PHP

Install or update to the required PHP agent version. For best results, update to the latest PHP agent version.

Option PHP agent version
Standard distributed tracing

8.4 or higher

With W3C Trace Context: 9.8 or higher
Infinite Tracing 9.12.0.268 or higher
Python

Install or update to the required Python agent version. For best results, update to the latest Python agent version.

Option Python agent version
Standard distributed tracing

4.2.0.100 or higher

With W3C Trace Context: 5.6 or higher
Infinite Tracing

5.12.0.140 (includes W3C Trace Context)

Supported environments: CPython only (pypy is unsupported)
Ruby

Install or update to the required Ruby agent version. For best results, update to the latest Ruby agent version.

Also install the additional Ruby agent gem for Infinite Tracing.

Option Ruby agent version
Standard distributed tracing

newrelic_rpm 5.3.0.346 or higher

With W3C Trace Context: newrelic_rpm 6.9 or higher
Infinite Tracing

newrelic_rpm 6.11.0.365 or higher (includes W3C Trace Context)

newrelic-infinite_tracing 6.11.0.375 or higher

Step 2. (Infinite Tracing) Set up a trace observer

This section only applies to Infinite Tracing. If you're enabling standard distributed tracing, skip to Configure the agents.

Infinite Tracing requires you to provision a trace observer that receives your spans. The trace observer examines the spans and keeps the actionable ones. You provision a trace observer in a cluster of services in AWS called New Relic Edge.

If you send your data to a trace observer in one of our EU provider regions, you'll still need a US-based New Relic account because the tail-based sampling data is reported to data centers in the United States. If you have questions, contact your account representative.

Use our Edge app to set up the trace observer. This app also includes some optional configurations: trace observer monitoring and span attribute trace filter. If you're not sure whether you want to configure these, you can set them up later.

Complete the following:

2a. Set up the trace observer endpoint
  1. Go to one.newrelic.com.
  2. In the menu bar, click Apps, expand Your apps, and then click New Relic Edge.
  3. Select an account in the upper-left dropdown.

    If you have access to multiple account hierarchies, make sure you're in the hierarchy where you want a trace observer.

  4. If no trace observers are listed, click New trace observer to add one, insert a descriptive name, select a region, and then click Create.
  5. For the trace observer you want to use, go to the Endpoints dropdown and click the adjacent clipboard icon to copy these values:
    Endpoints dropdown value Description
    For language agents Copy and hold to use later as YOUR_TRACE_OBSERVER_HOST in Step 3. Configure the agents.
    For other integrations (Optional) Copy and hold to use later as YOUR_TRACE_OBSERVER_URL in 2b. Send test data to the trace observer.
  6. (Optional) If you want to start gathering trace observer metrics, click the toggle for Trace observer monitoring. For details, see trace observer monitoring.
  7. (Optional) If you want to customize the span attribute trace filter, click the gear icon below Trace filters to see the current rules. For details, see span attribute filter.
2b. (Optional) Send test data to the trace observer

This test includes a sample payload with one trace and two spans from the same service: Test Service A. Follow these steps to send a test request:

  1. Get or generate your Insert API key so you can use it later in the test.
  2. Copy the following curl request into a text editor:
    curl request
    curl -i -H "Content-Type: application/json" \
  -H "Api-Key: YOUR_INSERT_API_KEY" \
  -H 'Data-Format: newrelic' \
  -H 'Data-Format-Version: 1' \
  -X POST \
  -d '[
      {
          "common": {
          "attributes": {
              "environment": "staging"
          }
          },
          "spans": [
          {
              "trace.id": "123456",
              "id": "ABC",
              "attributes": {
                  "duration.ms": 12.53,
                  "host": "host123.test.com",
                  "name": "/home",
                  "service.name": "Test Service A"
              }
          },
          {
              "trace.id": "123456",
              "id": "DEF",
              "attributes": {
                  "duration.ms": 2.97,
                  "host": "host456.test.com",
                  "error.message": "Invalid credentials",
                  "name": "/auth",
                  "parent.id": "ABC",
                  "service.name": "Test Service B"
              }
          }
          ]
      }
      ]' \
'YOUR_TRACE_OBSERVER_URL'
  3. Insert your own values into the curl request:
    Value Description
    YOUR_INSERT_API_KEY Replace this with your Insert API key (not the same as your personal API key for NerdGraph API explorer)
    YOUR_TRACE_OBSERVER_URL Replace this with the value you copied in 2a. Set up the trace observer endpoint.
  4. Copy the content of the text editor into a terminal, and then execute the request.
  5. If the test does not return HTTP/1.1 202 Accepted indicating success, check the following and try again:
    • Confirm that you used the Edge app value For other integrations as YOUR_TRACE_OBSERVER_URL.
    • Confirm that you only have single quotes around the value you inserted for YOUR_TRACE_OBSERVER_URL.
    • Check that you are using the Insert API Key (not a license).
  6. If your test returned HTTP/1.1 202 Accepted, go to New Relic One to see a query of your test data using the span attribute service.name = Test Service A. Because the sample payload contains an error attribute, the error sampler will mark it for keeping. If you modify the payload to remove the error attributes, the random sampler may not choose to keep this particular trace.

    Traces may take up to one minute to be processed by both the trace observer and the Trace API.

Step 3. Configure the agents

Distributed tracing is enabled through configuration settings. The settings you need to change for distributed tracing depend on the type you want:

  • Standard distributed tracing
  • Infinite Tracing (requires YOUR_TRACE_OBSERVER_HOST from step 2)

Review the following section for your agent to see the required configuration settings. For more information, see Configure the agent.

Server-side configuration is not available for Infinite Tracing.

C SDK

Here's an overview of the settings. For more help with configuration, see Enable distributed tracing for your C applications.

Type Required configuration
Standard distributed tracing

Configuration examples:

  • newrelic_app_config_t structure: 
    newrelic_app_config_t* config;
config = newrelic_create_app_config(app_name, license_key);
config->distributed_tracing.enabled = true;
Infinite Tracing Not available
Go

Here's an overview of the settings. For more help with configuration, see Enable distributed tracing for your Go applications.

Type Required configuration
Standard distributed tracing

Configuration examples:

  • ConfigOption structure:

    newrelic.NewApplication(
  newrelic.ConfigAppName("Example App"),
  newrelic.ConfigLicense(os.Getenv("NEW_RELIC_LICENSE_KEY")),
  newrelic.ConfigDistributedTracerEnabled(true),
)

  • Environment variable:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Configuration examples:

  • newrelic.Config structure:

    app, err := newrelic.NewApplication(
    newrelic.ConfigAppName(YOUR_APP_NAME),
    newrelic.ConfigLicense(YOUR_LICENSE_KEY),
    func(cfg *newrelic.Config) {
        cfg.DistributedTracer.Enabled = true
        cfg.InfiniteTracing.TraceObserver.Host = YOUR_TRACE_OBSERVER_HOST
    },
)

  • Environment variables:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST=YOUR_TRACE_OBSERVER_HOST
Java

Here's an overview of the settings. For more help with configuration, see Java agent configuration: Config file.

Type Required configuration
Standard distributed tracing

Configuration examples:

  • Configuration file (newrelic.yml) (indented 2 spaces under the common stanza):

      distributed_tracing:
    enabled: true

  • Java system property:

    -Dnewrelic.config.distributed_tracing.enabled=true

  • Environment variable:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Configuration examples:

  • Configuration file (newrelic.yml) (indented 2 spaces under the common stanza):

      distributed_tracing:
    enabled: true
  infinite_tracing:
    trace_observer:
      host: "YOUR_TRACE_OBSERVER_HOST"

  • Java system property:

    -Dnewrelic.config.distributed_tracing.enabled=true
-Dnewrelic.config.infinite_tracing.trace_observer.host="YOUR_TRACE_OBSERVER_HOST"

  • Environment variable:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
.NET

Here's an overview of the settings. For more help with configuration, see .NET agent configuration.

Type Required configuration
Standard distributed tracing

Configuration examples:

  • Configuration file (newrelic.config): 
    <configuration . . . >
   <distributedTracing enabled="true" />
</configuration>
  • Environment variable: 
    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Configuration examples:

  • Configuration file (newrelic.config): 
    <configuration . . . >
   <distributedTracing enabled="true" />
   <infiniteTracing>
      <trace_observer host="YOUR_TRACE_OBSERVER_HOST" />
   </infiniteTracing>
</configuration>

  • Environment variable:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
Node.js

Here's an overview of the settings. For more help with configuration, see Node.js agent configuration.

Type Required configuration
Standard distributed tracing

Configuration examples:

Configuration file (newrelic.js):

distributed_tracing: {
  enabled: true
}

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Configuration examples:

Configuration file (newrelic.js):

distributed_tracing: {
  enabled: true
}
infinite_tracing: {
  trace_observer: {
    host: 'YOUR_TRACE_OBSERVER_HOST'
  }
}

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
PHP

Here's an overview of the settings. For more help with configuration, see Distributed tracing for the PHP agent

Type Required configuration
Standard distributed tracing

Configuration examples:

  • Configuration file (newrelic.ini):

    newrelic.distributed_tracing_enabled = true
Infinite Tracing

Configuration examples:

  • Configuration file (newrelic.ini):

    newrelic.distributed_tracing_enabled = true
newrelic.span_events_enabled = true
newrelic.infinite_tracing.trace_observer_host= YOUR_TRACE_OBSERVER_HOST
Python

Here's an overview of the settings. For more help with configuration, see Python agent configuration

Type Required configuration
Standard distributed tracing

Configuration file (newrelic.ini):

distributed_tracing.enabled = true

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Use the following installation command:

pip install newrelic[infinite-tracing]

Configuration file (newrelic.ini):

distributed_tracing.enabled = true
infinite_tracing.trace_observer_host= YOUR_TRACE_OBSERVER_HOST

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
Ruby

Here's an overview of the settings. For more help with configuration, see Ruby agent configuration.

Type Required configuration
Standard distributed tracing

Configuration examples:

  • Configuration file (newrelic.yml):

      distributed_tracing:
        enabled: true
  • Environment variable: 
    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Configuration examples:

  • Configuration file (newrelic.yml):

      distributed_tracing: 
      enabled: true
  infinite_tracing: 
      trace_observer: 
          host: 'YOUR_TRACE_OBSERVER_HOST'

  • Environment variable:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"

If you need help with proxy configuration, see Proxy support.

Step 4. View traces

After you upgrade and configure your APM agents to send data to your trace observer, you are ready to view traces. Here are two alternatives:

View traces that include a specific service

The Entity explorer helps you navigate to a specific service so you can see traces that include that service.

  1. Go to one.newrelic.com.
  2. Click Entity explorer in the top menu bar.
  3. Filter to the service you enabled for Infinite Tracing by typing the service name, and then press Enter.
  4. In the left navigation's Monitor section, click Distributed tracing.
View traces across accounts

This option allows you to search all traces across all New Relic accounts in your organization that you have access to.

  1. Go to one.newrelic.com.
  2. Click Apps in the top menu bar.
  3. Under Favorites click Distributed tracing.

  4. In the Find traces... search, type a search clause to find the service. For example, to query service.name or trace.id:

    service.name = YOUR_SERVICE_NAME
trace.id = YOUR_TRACE_ID

Manual instrumentation (If automatic instrumentation doesn't work)

Recommendation: Before performing any custom instrumentation, read:

If a service is not passing the trace header to other services, you can use the distributed tracing payload APIs to instrument the calling service and the called service. The calling service uses an API call to generate a payload, which is accepted by the called service.

Instrument the calling service

To instrument the calling service:

  1. Ensure the version of the APM agent that monitors the calling service supports distributed tracing.

  2. Invoke the agent API call for generating a distributed trace payload: C SDK | Go | Java | .NET | Node.js | PHP | Python | Ruby.

    To maintain proper ordering of spans in a trace, ensure you generate the payload in the context of the span that sends it.

  3. Add that payload to the call made to the destination service (for example, in a header).

  4. (Optional) Identify the call as an external call:

Instrument the called service

To instrument the called service:

  1. Ensure the version of the APM agent that monitors the called service supports distributed tracing.

  2. If the New Relic agent on the called service does not identify a New Relic transaction, use the agent API to declare a transaction:

    C SDK

    One way to tell that a transaction is not in progress: when newrelic_create_distributed_trace_payload() is called, a NULL pointer is returned. To solve this problem, follow the procedures to create a transaction with the C SDK.

    Go

    One way to tell that a transaction is not in progress: when Transaction.InsertDistributedTraceHeaders(h http.Header) is called, no headers are inserted. To create a transaction, see Instrument Go transactions.

    Java

    One way to tell that a transaction is not in progress: when createDistributedTracePayload() is called, an empty string is returned. To create a transaction, see Java agent transaction-related APIs.

    .NET

    One way to tell that a transaction is not in progress: CreateDistributedTracePayload() returns an empty payload. To create a transaction, see Introduction to .NET custom instrumentation.

    Node.js

    One way to tell that a transaction is not in progress: the Node.js agent logs will report an error similar to this:

    No transaction found when calling Transaction.acceptDistributedTracePayload.

    Use startWebTransaction to create a web transaction or startBackgroundTransaction to capture a non-web transaction.

    PHP

    One way to tell that a transaction is not in progress: newrelic_insert_distributed_trace_headers() returns false. To create a transaction, see newrelic_start_transaction.

    Python

    To tell that a transaction is not in progress: when transaction = current_transaction() is run, transaction is None. Or, if result = accept_distributed_trace_payload(payload) is run, then the result is False.

    Use background_task to report a non-web transaction. For more on Python instrumentation, see Monitor transactions and segments.

    Ruby

    If you are using a Rack-based web framework and have enabled New Relic's Rack instrumentation, the Ruby agent will handle starting a transaction for you. For other use cases, see the add_transaction_tracer API method.

  3. Extract the payload from the call that you received (for example, in a header).

  4. Invoke the call for accepting the payload: C SDK | Go | Java | .NET | PHP | Node.js | Python | Ruby.

For more help

If you need more help, check out these support and learning resources: