Language agents: Enable distributed tracing

Here's what you need to know for enabling either standard distributed tracing or Infinite Tracing with our language agents.

If you want to get more background before getting started, check out these topics:

The process of enabling distributed tracing takes you through these steps:

  1. Meet the prerequisites
  2. (Infinite Tracing) Set up a trace observer
  3. Configure your agents
  4. View traces

Step 1. Prerequisites

Check the following sections to confirm that the distributed tracing options you want are available for your environment. Your main options are:

Once you find the agent version with the features you want, follow the links for help installing or updating your agent. For best results, update existing agents to the latest version.

C SDK

Install (compile) or update to the required C SDK version. For best results, update to the latest C SDK version.

Option C SDK version
Standard distributed tracing

1.1.0 or higher (W3C Trace Context not available)
Infinite Tracing Not available
Go

Install or update to the required Go agent version. For best results, update to the latest Go agent version.

Option Go agent version
Standard distributed tracing

2.1.0 or higher

With W3C Trace Context: 3.1.0 or higher
Infinite Tracing

v3.5.0 (includes W3C Trace Context)

Supported environments: Go 1.9 or higher
Java

Install or update to the required Java agent version. For best results, update to the latest Java agent version.

Your JVM's networkaddress.cache.ttl security setting must not be set to forever (i.e. -1). For more information about this networking property, please visit the Oracle Network Properties docs.

Type Java agent version
Standard distributed tracing

4.3.0 or higher

With W3C Trace Context: 5.10 or higher
Infinite Tracing

5.12.1 or higher (includes W3C Trace Context)

Supported environments:

  • Java 8: Update 252 or higher
  • All versions of Java 9 or higher

For special considerations, see Infinite Tracing: Configuring SSL for Java 7 and 8.
.NET

Install or update to the required .NET agent version. For best results, update to the latest .NET agent version.

Option .NET agent version
Standard distributed tracing

8.6.45.0 or higher

With W3C Trace Context: 8.27.139.0 or higher
Infinite Tracing

8.30.0 (includes W3C Trace Context)

Supported environments:

  • .NET Framework 4.5 or higher
  • .NET Core 2.0 or higher
Node.js

Install or update to the required Node.js agent version. For best results, update to the latest Node.js agent version.

Option Node.js agent version
Standard distributed tracing

4.7.0 or higher

With W3C Trace Context: 6.4 or higher
Infinite Tracing

7.1.1 (includes W3C Trace Context)

Supported environments: Node version 10.10.0 or higher
PHP

Install or update to the required PHP agent version. For best results, update to the latest PHP agent version.

Option PHP agent version
Standard distributed tracing

8.4 or higher

With W3C Trace Context: 9.8 or higher
Infinite Tracing 9.12.0.268 or higher
Python

Install or update to the required Python agent version. For best results, update to the latest Python agent version.

Option Python agent version
Standard distributed tracing

4.2.0.100 or higher

With W3C Trace Context: 5.6 or higher
Infinite Tracing

5.12.0.140 (includes W3C Trace Context)

Supported environments: CPython only (pypy is unsupported)
Ruby

Install or update to the required Ruby agent version. For best results, update to the latest Ruby agent version.

Also install the additional Ruby agent gem for Infinite Tracing.

Option Ruby agent version
Standard distributed tracing

newrelic_rpm 5.3.0.346 or higher

With W3C Trace Context: newrelic_rpm 6.9 or higher
Infinite Tracing

newrelic_rpm 6.11.0.365 or higher (includes W3C Trace Context)

newrelic-infinite_tracing 6.11.0.375 or higher

Step 2. Set up Infinite Tracing (optional)

For all language agents except C SDK, you have the option of enabling Infinite Tracing. If you want to skip this and install standard distributed tracing, skip to Configure the agent.

If you want to enable Infinite Tracing, see Set up trace observer. When you're done, return here to complete the next step of configuring the agent to send data to the trace observer.

Step 3. Configure the agent

Distributed tracing is enabled through configuration settings. Review the following agent-specific sections. If you're enabling Infinite Tracing, there are different configuration options for that. For more about agent configuration, see Configure the agent.

Server-side configuration is not available for Infinite Tracing.

C SDK

Here's an overview of the settings. For more help with configuration, see Enable distributed tracing for your C applications.

Type Required configuration
Standard distributed tracing

Configuration examples:

  • newrelic_app_config_t structure: 
    newrelic_app_config_t* config;
config = newrelic_create_app_config(app_name, license_key);
config->distributed_tracing.enabled = true;
Infinite Tracing Not available
Go

Here's an overview of the settings. For more help with configuration, see Enable distributed tracing for your Go applications.

Type Required configuration
Standard distributed tracing

Configuration examples:

  • ConfigOption structure:

    newrelic.NewApplication(
  newrelic.ConfigAppName("Example App"),
  newrelic.ConfigLicense(os.Getenv("NEW_RELIC_LICENSE_KEY")),
  newrelic.ConfigDistributedTracerEnabled(true),
)

  • Environment variable:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Configuration examples:

  • newrelic.Config structure:

    app, err := newrelic.NewApplication(
    newrelic.ConfigAppName(YOUR_APP_NAME),
    newrelic.ConfigLicense(YOUR_LICENSE_KEY),
    func(cfg *newrelic.Config) {
        cfg.DistributedTracer.Enabled = true
        cfg.InfiniteTracing.TraceObserver.Host = YOUR_TRACE_OBSERVER_HOST
    },
)

  • Environment variables:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST=YOUR_TRACE_OBSERVER_HOST
Java

Here's an overview of the settings. For more help with configuration, see Java agent configuration: Config file.

Type Required configuration
Standard distributed tracing

Configuration examples:

  • Configuration file (newrelic.yml) (indented 2 spaces under the common stanza):

      distributed_tracing:
    enabled: true

  • Java system property:

    -Dnewrelic.config.distributed_tracing.enabled=true

  • Environment variable:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Configuration examples:

  • Configuration file (newrelic.yml) (indented 2 spaces under the common stanza):

      distributed_tracing:
    enabled: true
  infinite_tracing:
    trace_observer:
      host: "YOUR_TRACE_OBSERVER_HOST"

  • Java system property:

    -Dnewrelic.config.distributed_tracing.enabled=true
-Dnewrelic.config.infinite_tracing.trace_observer.host="YOUR_TRACE_OBSERVER_HOST"

  • Environment variable:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
.NET

Here's an overview of the settings. For more help with configuration, see .NET agent configuration.

Type Required configuration
Standard distributed tracing

Configuration examples:

  • Configuration file (newrelic.config): 
    <configuration . . . >
   <distributedTracing enabled="true" />
</configuration>
  • Environment variable: 
    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Configuration examples:

  • Configuration file (newrelic.config): 
    <configuration . . . >
   <distributedTracing enabled="true" />
   <infiniteTracing>
      <trace_observer host="YOUR_TRACE_OBSERVER_HOST" />
   </infiniteTracing>
</configuration>

  • Environment variable:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
Node.js

Here's an overview of the settings. For more help with configuration, see Node.js agent configuration.

Type Required configuration
Standard distributed tracing

Configuration examples:

Configuration file (newrelic.js):

distributed_tracing: {
  enabled: true
}

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Configuration examples:

Configuration file (newrelic.js):

distributed_tracing: {
  enabled: true
}
infinite_tracing: {
  trace_observer: {
    host: 'YOUR_TRACE_OBSERVER_HOST'
  }
}

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
PHP

Here's an overview of the settings. For more help with configuration, see Distributed tracing for the PHP agent

Type Required configuration
Standard distributed tracing

Configuration examples:

  • Configuration file (newrelic.ini):

    newrelic.distributed_tracing_enabled = true
Infinite Tracing

Configuration examples:

  • Configuration file (newrelic.ini):

    newrelic.distributed_tracing_enabled = true
newrelic.span_events_enabled = true
newrelic.infinite_tracing.trace_observer.host= "YOUR_TRACE_OBSERVER_HOST"
Python

Here's an overview of the settings. For more help with configuration, see Python agent configuration

Type Required configuration
Standard distributed tracing

Configuration file (newrelic.ini):

distributed_tracing.enabled = true

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Use the following installation command:

pip install newrelic[infinite-tracing]

Configuration file (newrelic.ini):

distributed_tracing.enabled = true
infinite_tracing.trace_observer_host= YOUR_TRACE_OBSERVER_HOST

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
Ruby

Here's an overview of the settings. For more help with configuration, see Ruby agent configuration.

Type Required configuration
Standard distributed tracing

Configuration examples:

  • Configuration file (newrelic.yml):

      distributed_tracing:
        enabled: true
  • Environment variable: 
    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing

Configuration examples:

  • Configuration file (newrelic.yml):

      distributed_tracing: 
      enabled: true
  infinite_tracing: 
      trace_observer: 
          host: 'YOUR_TRACE_OBSERVER_HOST'

  • Environment variable:

    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"

If you need help with proxy configuration, see Proxy support.

Step 4. View traces

After you configure your APM agents to send data to your trace observer, you're ready to view traces. Here are two alternatives:

View traces that include a specific service

The Entity explorer helps you navigate to a specific service so you can see traces that include that service.

  1. Go to one.newrelic.com.
  2. Click Entity explorer in the top menu bar.
  3. Filter to the service you enabled for Infinite Tracing by typing the service name, and then press Enter.
  4. In the left navigation's Monitor section, click Distributed tracing.
View traces across accounts

This option allows you to search all traces across all New Relic accounts in your organization that you have access to.

  1. Go to one.newrelic.com.
  2. Click Apps in the top menu bar.
  3. Under Favorites click Distributed tracing.

  4. In the Find traces... search, type a search clause to find the service. For example, to query service.name or trace.id:

    service.name = YOUR_SERVICE_NAME
trace.id = YOUR_TRACE_ID

You can also run custom queries and create custom charts of your trace data. See Query trace data.

Manual instrumentation (If automatic instrumentation doesn't work)

Recommendation: Before performing any custom instrumentation, read:

If a service is not passing the trace header to other services, you can use the distributed tracing payload APIs to instrument the calling service and the called service. The calling service uses an API call to generate a payload, which is accepted by the called service.

Instrument the calling service

To instrument the calling service:

  1. Ensure the version of the APM agent that monitors the calling service supports distributed tracing.

  2. Invoke the agent API call for generating a distributed trace payload: C SDK | Go | Java | .NET | Node.js | PHP | Python | Ruby.

    To maintain proper ordering of spans in a trace, ensure you generate the payload in the context of the span that sends it.

  3. Add that payload to the call made to the destination service (for example, in a header).

  4. (Optional) Identify the call as an external call:

Instrument the called service

To instrument the called service:

  1. Ensure the version of the APM agent that monitors the called service supports distributed tracing.

  2. If the New Relic agent on the called service does not identify a New Relic transaction, use the agent API to declare a transaction:

    C SDK

    One way to tell that a transaction is not in progress: when newrelic_create_distributed_trace_payload() is called, a NULL pointer is returned. To solve this problem, follow the procedures to create a transaction with the C SDK.

    Go

    One way to tell that a transaction is not in progress: when Transaction.InsertDistributedTraceHeaders(h http.Header) is called, no headers are inserted. To create a transaction, see Instrument Go transactions.

    Java

    One way to tell that a transaction is not in progress: when Transaction.insertDistributedTraceHeaders(Headers) is called, no headers are inserted (this API requires agent 6.4.0+). To create a transaction, see Java agent transaction-related APIs.

    .NET

    One way to tell that a transaction is not in progress: CreateDistributedTracePayload() returns an empty payload. To create a transaction, see Introduction to .NET custom instrumentation.

    Node.js

    One way to tell that a transaction is not in progress: the Node.js agent logs will report an error similar to this:

    No transaction found when calling Transaction.acceptDistributedTracePayload.

    Use startWebTransaction to create a web transaction or startBackgroundTransaction to capture a non-web transaction.

    PHP

    One way to tell that a transaction is not in progress: newrelic_insert_distributed_trace_headers() returns false. To create a transaction, see newrelic_start_transaction.

    Python

    To tell that a transaction is not in progress: when transaction = current_transaction() is run, transaction is None. Or, if result = accept_distributed_trace_payload(payload) is run, then the result is False.

    Use background_task to report a non-web transaction. For more on Python instrumentation, see Monitor transactions and segments.

    Ruby

    If you are using a Rack-based web framework and have enabled New Relic's Rack instrumentation, the Ruby agent will handle starting a transaction for you. For other use cases, see the add_transaction_tracer API method.

  3. Extract the payload from the call that you received (for example, in a header).

  4. Invoke the call for accepting the payload: C SDK | Go | Java | .NET | PHP | Node.js | Python | Ruby.

For more help

If you need more help, check out these support and learning resources: