Language agents: Enable distributed tracing

Here's what you need to know for enabling either standard distributed tracing or Infinite Tracing with our language agents.

Tip

If you want to get more background before getting started, check out these topics:

How span sampling works explains distributed tracing options.
Impacts to APM tells you what to expect if you are a current APM user.

The process of enabling distributed tracing takes you through these steps:

Meet the prerequisites
(Infinite Tracing) Set up a trace observer
Configure your agents
View traces

Step 1. Prerequisites

Check the following sections to confirm that the distributed tracing options you want are available for your environment. Your main options are:

Standard distributed tracing: Head-based sampling
Infinite Tracing: Tail-based sampling
Support for W3C Trace Context: Available in all Infinite Tracing agents and in standard distributed tracing agents as noted

Once you find the agent version with the features you want, follow the links for help installing or updating your agent. For best results, update existing agents to the latest version.

Install (compile) or update to the required C SDK version. For best results, update to the latest C SDK version.

Option	C SDK version
Standard distributed tracing	1.1.0 or higher (W3C Trace Context not available)
Infinite Tracing	Not available

Install or update to the required Go agent version. For best results, update to the latest Go agent version.

Option	Go agent version
Standard distributed tracing	2.1.0 or higher With W3C Trace Context: 3.1.0 or higher
Infinite Tracing	v3.5.0 (includes W3C Trace Context) Supported environments: Go 1.9 or higher

Option

Go agent version

Standard distributed tracing

2.1.0 or higher

With W3C Trace Context: 3.1.0 or higher

Infinite Tracing

v3.5.0 (includes W3C Trace Context)

Supported environments: Go 1.9 or higher

Install or update to the required Java agent version. For best results, update to the latest Java agent version.

Important

Your JVM's networkaddress.cache.ttl security setting must not be set to forever (i.e. -1). For more information about this networking property, please visit the Oracle Network Properties docs.

Type	Java agent version
Standard distributed tracing	4.3.0 or higher With W3C Trace Context: 5.10 or higher
Infinite Tracing	5.12.1 or higher (includes W3C Trace Context) Supported environments: Java 8: Update 252 or higher All versions of Java 9 or higher Tip For special considerations, see Infinite Tracing: Configuring SSL for Java 7 and 8.

Type

Java agent version

Standard distributed tracing

4.3.0 or higher

With W3C Trace Context: 5.10 or higher

Infinite Tracing

5.12.1 or higher (includes W3C Trace Context)

Supported environments:

Java 8: Update 252 or higher
All versions of Java 9 or higher
Tip
For special considerations, see Infinite Tracing: Configuring SSL for Java 7 and 8.

Install or update to the required .NET agent version. For best results, update to the latest .NET agent version.

Option	.NET agent version
Standard distributed tracing	8.6.45.0 or higher With W3C Trace Context: 8.27.139.0 or higher
Infinite Tracing	8.30.0 (includes W3C Trace Context) Supported environments: .NET Framework 4.5 or higher .NET Core 2.0 or higher

Option

.NET agent version

Standard distributed tracing

8.6.45.0 or higher

With W3C Trace Context: 8.27.139.0 or higher

Infinite Tracing

8.30.0 (includes W3C Trace Context)

Supported environments:

.NET Framework 4.5 or higher
.NET Core 2.0 or higher

Install or update to the required Node.js agent version. For best results, update to the latest Node.js agent version.

Option	Node.js agent version
Standard distributed tracing	4.7.0 or higher With W3C Trace Context: 6.4 or higher
Infinite Tracing	7.1.1 (includes W3C Trace Context) Supported environments: Node version 10.10.0 or higher

Option

Node.js agent version

Standard distributed tracing

4.7.0 or higher

With W3C Trace Context: 6.4 or higher

Infinite Tracing

7.1.1 (includes W3C Trace Context)

Supported environments: Node version 10.10.0 or higher

Install or update to the required PHP agent version. For best results, update to the latest PHP agent version.

Option	PHP agent version
Standard distributed tracing	8.4 or higher With W3C Trace Context: 9.8 or higher
Infinite Tracing	9.12.0.268 or higher

Option

PHP agent version

Standard distributed tracing

8.4 or higher

With W3C Trace Context: 9.8 or higher

Infinite Tracing

9.12.0.268 or higher

Install or update to the required Python agent version. For best results, update to the latest Python agent version.

Option	Python agent version
Standard distributed tracing	4.2.0.100 or higher With W3C Trace Context: 5.6 or higher
Infinite Tracing	5.12.0.140 (includes W3C Trace Context) Supported environments: CPython only (pypy is unsupported)

Option

Python agent version

Standard distributed tracing

4.2.0.100 or higher

With W3C Trace Context: 5.6 or higher

Infinite Tracing

5.12.0.140 (includes W3C Trace Context)

Supported environments: CPython only (pypy is unsupported)

Install or update to the required Ruby agent version. For best results, update to the latest Ruby agent version.

Also install the additional Ruby agent gem for Infinite Tracing.

Option	Ruby agent version
Standard distributed tracing	newrelic_rpm 5.3.0.346 or higher With W3C Trace Context: newrelic_rpm 6.9 or higher
Infinite Tracing	newrelic_rpm 6.11.0.365 or higher (includes W3C Trace Context) newrelic-infinite_tracing 6.11.0.375 or higher

Option

Ruby agent version

Standard distributed tracing

newrelic_rpm 5.3.0.346 or higher

With W3C Trace Context: newrelic_rpm 6.9 or higher

Infinite Tracing

newrelic_rpm 6.11.0.365 or higher (includes W3C Trace Context)

newrelic-infinite_tracing 6.11.0.375 or higher

Step 2. Set up Infinite Tracing (optional)

For all language agents except C SDK, you have the option of enabling Infinite Tracing. If you want to skip this and install standard distributed tracing, skip to Configure the agent.

If you want to enable Infinite Tracing, see Set up trace observer. When you're done, return here to complete the next step of configuring the agent to send data to the trace observer.

Step 3. Configure the agent

Distributed tracing is enabled through configuration settings. Review the following agent-specific sections. If you're enabling Infinite Tracing, there are different configuration options for that. For more about agent configuration, see Configure the agent.

Important

Server-side configuration is not available for Infinite Tracing.

Here's an overview of the settings. For more help with configuration, see Enable distributed tracing for your C applications.

Type	Required configuration
Standard distributed tracing	Configuration examples: `newrelic_app_config_t` structure: `newrelic_app_config_t* config; config = newrelic_create_app_config(app_name, license_key); config->distributed_tracing.enabled = true;`
Infinite Tracing	Not available

Type

Required configuration

Standard distributed tracing

Configuration examples:

newrelic_app_config_t structure:
```
newrelic_app_config_t* config;
config = newrelic_create_app_config(app_name, license_key);
config->distributed_tracing.enabled = true;
```

Infinite Tracing

Not available

Here's an overview of the settings. For more help with configuration, see Enable distributed tracing for your Go applications.

Type	Required configuration
Standard distributed tracing	Configuration examples: `ConfigOption` structure: `newrelic.NewApplication( newrelic.ConfigAppName("Example App"), newrelic.ConfigLicense(os.Getenv("NEW_RELIC_LICENSE_KEY")), newrelic.ConfigDistributedTracerEnabled(true), )` Environment variable: `NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true`
Infinite Tracing	Configuration examples: `newrelic.Config` structure: `app, err := newrelic.NewApplication( newrelic.ConfigAppName(YOUR_APP_NAME), newrelic.ConfigLicense(YOUR_LICENSE_KEY), func(cfg *newrelic.Config) { cfg.DistributedTracer.Enabled = true cfg.InfiniteTracing.TraceObserver.Host = YOUR_TRACE_OBSERVER_HOST }, )` Environment variables: `NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST=YOUR_TRACE_OBSERVER_HOST`

Type

Required configuration

Standard distributed tracing

Configuration examples:

ConfigOption structure:

newrelic.NewApplication(
  newrelic.ConfigAppName("Example App"),
  newrelic.ConfigLicense(os.Getenv("NEW_RELIC_LICENSE_KEY")),
  newrelic.ConfigDistributedTracerEnabled(true),
)

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true

Infinite Tracing

Configuration examples:

newrelic.Config structure:

app, err := newrelic.NewApplication(
    newrelic.ConfigAppName(YOUR_APP_NAME),
    newrelic.ConfigLicense(YOUR_LICENSE_KEY),
    func(cfg *newrelic.Config) {
        cfg.DistributedTracer.Enabled = true
        cfg.InfiniteTracing.TraceObserver.Host = YOUR_TRACE_OBSERVER_HOST
    },
)

Environment variables:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST=YOUR_TRACE_OBSERVER_HOST

Here's an overview of the settings. For more help with configuration, see Java agent configuration: Config file.

Type	Required configuration
Standard distributed tracing	Configuration examples: Configuration file (`newrelic.yml`) (indented 2 spaces under the `common` stanza): `distributed_tracing: enabled: true` Java system property: `-Dnewrelic.config.distributed_tracing.enabled=true` Environment variable: `NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true`
Infinite Tracing	Configuration examples: Configuration file (`newrelic.yml`) (indented 2 spaces under the `common` stanza): `distributed_tracing: enabled: true infinite_tracing: trace_observer: host: "YOUR_TRACE_OBSERVER_HOST"` Java system property: `-Dnewrelic.config.distributed_tracing.enabled=true -Dnewrelic.config.infinite_tracing.trace_observer.host="YOUR_TRACE_OBSERVER_HOST"` Environment variable: `NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"`

Type

Required configuration

Standard distributed tracing

Configuration examples:

Configuration file (newrelic.yml) (indented 2 spaces under the common stanza):
```
distributed_tracing:
    enabled: true
```

Java system property:

-Dnewrelic.config.distributed_tracing.enabled=true

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true

Infinite Tracing

Configuration examples:

Configuration file (newrelic.yml) (indented 2 spaces under the common stanza):

distributed_tracing:
    enabled: true
  infinite_tracing:
    trace_observer:
      host: "YOUR_TRACE_OBSERVER_HOST"

Java system property:

-Dnewrelic.config.distributed_tracing.enabled=true
-Dnewrelic.config.infinite_tracing.trace_observer.host="YOUR_TRACE_OBSERVER_HOST"

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"

Here's an overview of the settings. For more help with configuration, see .NET agent configuration.

Type	Required configuration
Standard distributed tracing	Configuration examples: Configuration file (`newrelic.config`): `<configuration . . . > <distributedTracing enabled="true" /> </configuration>` Environment variable: `NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true`
Infinite Tracing	Configuration examples: Configuration file (`newrelic.config`): `<configuration . . . > <distributedTracing enabled="true" /> <infiniteTracing> <trace_observer host="YOUR_TRACE_OBSERVER_HOST" /> </infiniteTracing> </configuration>` Environment variable: `NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"`

Type

Required configuration

Standard distributed tracing

Configuration examples:

Configuration file (newrelic.config):

<configuration . . . >
   <distributedTracing enabled="true" />
</configuration>

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true

Infinite Tracing

Configuration examples:

Configuration file (newrelic.config):

<configuration . . . >
   <distributedTracing enabled="true" />
   <infiniteTracing>
      <trace_observer host="YOUR_TRACE_OBSERVER_HOST" />
   </infiniteTracing>
</configuration>

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"

Here's an overview of the settings. For more help with configuration, see Node.js agent configuration.

Type	Required configuration
Standard distributed tracing	Configuration examples: Configuration file (`newrelic.js`): `distributed_tracing: { enabled: true }` Environment variable: `NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true`
Infinite Tracing	Configuration examples: Configuration file (`newrelic.js`): `distributed_tracing: { enabled: true } infinite_tracing: { trace_observer: { host: 'YOUR_TRACE_OBSERVER_HOST' } }` Environment variable: `NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"`

Type

Required configuration

Standard distributed tracing

Configuration examples:

Configuration file (newrelic.js):

distributed_tracing: {
  enabled: true
}

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true

Infinite Tracing

Configuration examples:

Configuration file (newrelic.js):

distributed_tracing: {
  enabled: true
}
infinite_tracing: {
  trace_observer: {
    host: 'YOUR_TRACE_OBSERVER_HOST'
  }
}

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"

Here's an overview of the settings. For more help with configuration, see Distributed tracing for the PHP agent

Type	Required configuration
Standard distributed tracing	Configuration examples: Configuration file (`newrelic.ini`): `newrelic.distributed_tracing_enabled = true`
Infinite Tracing	Configuration examples: Configuration file (`newrelic.ini`): `newrelic.distributed_tracing_enabled = true newrelic.span_events_enabled = true newrelic.infinite_tracing.trace_observer.host= "YOUR_TRACE_OBSERVER_HOST"`

Type

Required configuration

Standard distributed tracing

Configuration examples:

Configuration file (newrelic.ini):

newrelic.distributed_tracing_enabled = true

Infinite Tracing

Configuration examples:

Configuration file (newrelic.ini):

newrelic.distributed_tracing_enabled = true
newrelic.span_events_enabled = true
newrelic.infinite_tracing.trace_observer.host= "YOUR_TRACE_OBSERVER_HOST"

Here's an overview of the settings. For more help with configuration, see Python agent configuration

Type	Required configuration
Standard distributed tracing	Configuration file (`newrelic.ini`): `distributed_tracing.enabled = true` Environment variable: `NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true`
Infinite Tracing	Use the following installation command: `pip install newrelic[infinite-tracing]` Configuration file (`newrelic.ini`): `distributed_tracing.enabled = true infinite_tracing.trace_observer_host= YOUR_TRACE_OBSERVER_HOST` Environment variable: `NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"`

Type

Required configuration

Standard distributed tracing

Configuration file (newrelic.ini):

distributed_tracing.enabled = true

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true

Infinite Tracing

Use the following installation command:

pip install newrelic[infinite-tracing]

Configuration file (newrelic.ini):

distributed_tracing.enabled = true
infinite_tracing.trace_observer_host= YOUR_TRACE_OBSERVER_HOST

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"

Here's an overview of the settings. For more help with configuration, see Ruby agent configuration.

Type	Required configuration
Standard distributed tracing	Configuration examples: Configuration file (`newrelic.yml`): `distributed_tracing: enabled: true` Environment variable: `NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true`
Infinite Tracing	Configuration examples: Configuration file (`newrelic.yml`): `distributed_tracing: enabled: true infinite_tracing: trace_observer: host: 'YOUR_TRACE_OBSERVER_HOST'` Environment variable: `NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"`

Type

Required configuration

Standard distributed tracing

Configuration examples:

Configuration file (newrelic.yml):

distributed_tracing:
        enabled: true

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true

Infinite Tracing

Configuration examples:

Configuration file (newrelic.yml):

distributed_tracing: 
      enabled: true
  infinite_tracing: 
      trace_observer: 
          host: 'YOUR_TRACE_OBSERVER_HOST'

Environment variable:

NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"

Tip

If you need help with proxy configuration, see Proxy support.

Step 4. View traces

After you configure your APM agents to send data to your trace observer, you're ready to view traces. Here are two alternatives:

This option allows you to search all traces across all New Relic accounts in your organization that you have access to.

Go to one.newrelic.com.
Click Apps in the top menu bar.
Under Favorites click Distributed tracing.
In the Find traces... search, type a search clause to find the service. For example, to query service.name or trace.id:
```
service.name = YOUR_SERVICE_NAME
trace.id = YOUR_TRACE_ID
```

You can also run custom queries and create custom charts of your trace data. See Query trace data.

Manual instrumentation (If automatic instrumentation doesn't work)

Recommendation: Before performing any custom instrumentation, read:

If a service is not passing the trace header to other services, you can use the distributed tracing payload APIs to instrument the calling service and the called service. The calling service uses an API call to generate a payload, which is accepted by the called service.

To instrument the calling service:

Ensure the version of the APM agent that monitors the calling service supports distributed tracing.
Invoke the agent API call for generating a distributed trace payload: C SDK | Go | Java | .NET | Node.js | PHP | Python | Ruby.
Important
To maintain proper ordering of spans in a trace, ensure you generate the payload in the context of the span that sends it.
Add that payload to the call made to the destination service (for example, in a header).
(Optional) Identify the call as an external call:
- C SDK
- Go
- Java
- .NET: n/a
- Node.js
- PHP: n/a
- Python
- Ruby

To instrument the called service:

For more help

If you need more help, check out these support and learning resources:

Browse the Explorers Hub to get help from the community and join in discussions.
Find answers on our sites and learn how to use our support portal.
Run New Relic Diagnostics, our troubleshooting tool for Linux, Windows, and macOS.
Review New Relic's data security and licenses documentation.

Language agents: Enable distributed tracing

Tip

Step 1. Prerequisites

C SDK

Go

Java

.NET

Node.js

PHP

Python

Ruby

Step 2. Set up Infinite Tracing (optional)

Step 3. Configure the agent

Important

C SDK

Go

Java

.NET

Node.js

PHP

Python

Ruby

Tip

Step 4. View traces

View traces that include a specific service

View traces across accounts

Manual instrumentation (If automatic instrumentation doesn't work)

Instrument the calling service

Instrument the called service

.css-s3mb0o{fill:none;stroke:currentColor;stroke-width:2;stroke-linecap:round;stroke-linejoin:round;}.css-1bvd50p{width:1rem;height:1rem;fill:none;stroke:currentColor;stroke-width:2;stroke-linecap:round;stroke-linejoin:round;}For more help

For more help