Here's what you need to know for enabling either standard distributed tracing or Infinite Tracing with our language agents.
If you want to get more background before getting started, check out these topics:
- How span sampling works explains distributed tracing options.
- Impacts to APM tells you what to expect if you are a current APM user.
The process of enabling distributed tracing takes you through these steps:
Step 1. Prerequisites
Check the following sections to confirm that the distributed tracing options you want are available for your environment. Your main options are:
- Standard distributed tracing: Head-based sampling
- Infinite Tracing: Tail-based sampling
- Support for W3C Trace Context: Available in all Infinite Tracing agents and in standard distributed tracing agents as noted
Once you find the agent version with the features you want, follow the links for help installing or updating your agent. For best results, update existing agents to the latest version.
- C SDK
-
Install (compile) or update to the required C SDK version. For best results, update to the latest C SDK version.
Option C SDK version Standard distributed tracing 1.1.0 or higher (W3C Trace Context not available)
Infinite Tracing Not available - Go
-
Install or update to the required Go agent version. For best results, update to the latest Go agent version.
Option Go agent version Standard distributed tracing 2.1.0 or higher
With W3C Trace Context: 3.1.0 or higher
Infinite Tracing v3.5.0 (includes W3C Trace Context)
Supported environments: Go 1.9 or higher
- Java
-
Install or update to the required Java agent version. For best results, update to the latest Java agent version.
Your JVM's
networkaddress.cache.ttl
security setting must not be set to forever (i.e.-1
). For more information about this networking property, please visit the Oracle Network Properties docs.Type Java agent version Standard distributed tracing 4.3.0 or higher
With W3C Trace Context: 5.10 or higher
Infinite Tracing 5.12.1 or higher (includes W3C Trace Context)
Supported environments:
- Java 8: Update 252 or higher
- All versions of Java 9 or higher
For special considerations, see Infinite Tracing: Configuring SSL for Java 7 and 8.
- .NET
-
Install or update to the required .NET agent version. For best results, update to the latest .NET agent version.
Option .NET agent version Standard distributed tracing 8.6.45.0 or higher
With W3C Trace Context: 8.27.139.0 or higher
Infinite Tracing 8.30.0 (includes W3C Trace Context)
Supported environments:
- .NET Framework 4.5 or higher
- .NET Core 2.0 or higher
- Node.js
-
Install or update to the required Node.js agent version. For best results, update to the latest Node.js agent version.
Option Node.js agent version Standard distributed tracing 4.7.0 or higher
With W3C Trace Context: 6.4 or higher
Infinite Tracing 7.1.1 (includes W3C Trace Context)
Supported environments: Node version 10.10.0 or higher
- PHP
-
Install or update to the required PHP agent version. For best results, update to the latest PHP agent version.
Option PHP agent version Standard distributed tracing 8.4 or higher
With W3C Trace Context: 9.8 or higher
Infinite Tracing 9.12.0.268 or higher - Python
-
Install or update to the required Python agent version. For best results, update to the latest Python agent version.
Option Python agent version Standard distributed tracing 4.2.0.100 or higher
With W3C Trace Context: 5.6 or higher
Infinite Tracing 5.12.0.140 (includes W3C Trace Context)
Supported environments: CPython only (pypy is unsupported)
- Ruby
-
Install or update to the required Ruby agent version. For best results, update to the latest Ruby agent version.
Also install the additional Ruby agent gem for Infinite Tracing.
Option Ruby agent version Standard distributed tracing newrelic_rpm 5.3.0.346 or higher
With W3C Trace Context: newrelic_rpm 6.9 or higher
Infinite Tracing newrelic_rpm 6.11.0.365 or higher (includes W3C Trace Context)
newrelic-infinite_tracing 6.11.0.375 or higher
Step 2. Set up Infinite Tracing (optional)
For all language agents except C SDK, you have the option of enabling Infinite Tracing. If you want to skip this and install standard distributed tracing, skip to Configure the agent.
If you want to enable Infinite Tracing, see Set up trace observer. When you're done, return here to complete the next step of configuring the agent to send data to the trace observer.
Step 3. Configure the agent
Distributed tracing is enabled through configuration settings. Review the following agent-specific sections. If you're enabling Infinite Tracing, there are different configuration options for that. For more about agent configuration, see Configure the agent.
Server-side configuration is not available for Infinite Tracing.
- C SDK
-
Here's an overview of the settings. For more help with configuration, see Enable distributed tracing for your C applications.
Type Required configuration Standard distributed tracing Configuration examples:
newrelic_app_config_t
structure:newrelic_app_config_t* config; config = newrelic_create_app_config(app_name, license_key); config->distributed_tracing.enabled = true;
Infinite Tracing Not available - Go
-
Here's an overview of the settings. For more help with configuration, see Enable distributed tracing for your Go applications.
Type Required configuration Standard distributed tracing Configuration examples:
-
ConfigOption
structure:newrelic.NewApplication( newrelic.ConfigAppName("Example App"), newrelic.ConfigLicense(os.Getenv("NEW_RELIC_LICENSE_KEY")), newrelic.ConfigDistributedTracerEnabled(true), )
-
Environment variable:
NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing Configuration examples:
-
newrelic.Config
structure:app, err := newrelic.NewApplication( newrelic.ConfigAppName(YOUR_APP_NAME), newrelic.ConfigLicense(YOUR_LICENSE_KEY), func(cfg *newrelic.Config) { cfg.DistributedTracer.Enabled = true cfg.InfiniteTracing.TraceObserver.Host = YOUR_TRACE_OBSERVER_HOST }, )
-
Environment variables:
NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST=YOUR_TRACE_OBSERVER_HOST
-
- Java
-
Here's an overview of the settings. For more help with configuration, see Java agent configuration: Config file.
Type Required configuration Standard distributed tracing Configuration examples:
-
Configuration file (
newrelic.yml
) (indented 2 spaces under thecommon
stanza):distributed_tracing: enabled: true
-
Java system property:
-Dnewrelic.config.distributed_tracing.enabled=true
-
Environment variable:
NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing Configuration examples:
-
Configuration file (
newrelic.yml
) (indented 2 spaces under thecommon
stanza):distributed_tracing: enabled: true infinite_tracing: trace_observer: host: "YOUR_TRACE_OBSERVER_HOST"
-
Java system property:
-Dnewrelic.config.distributed_tracing.enabled=true -Dnewrelic.config.infinite_tracing.trace_observer.host="YOUR_TRACE_OBSERVER_HOST"
-
Environment variable:
NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
-
- .NET
-
Here's an overview of the settings. For more help with configuration, see .NET agent configuration.
Type Required configuration Standard distributed tracing Configuration examples:
- Configuration file (
newrelic.config
):<configuration . . . > <distributedTracing enabled="true" /> </configuration>
- Environment variable:
NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing Configuration examples:
- Configuration file (
newrelic.config
):<configuration . . . > <distributedTracing enabled="true" /> <infiniteTracing> <trace_observer host="YOUR_TRACE_OBSERVER_HOST" /> </infiniteTracing> </configuration>
-
Environment variable:
NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
- Configuration file (
- Node.js
-
Here's an overview of the settings. For more help with configuration, see Node.js agent configuration.
Type Required configuration Standard distributed tracing Configuration examples:
Configuration file (
newrelic.js
):distributed_tracing: { enabled: true }
Environment variable:
NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing Configuration examples:
Configuration file (
newrelic.js
):distributed_tracing: { enabled: true } infinite_tracing: { trace_observer: { host: 'YOUR_TRACE_OBSERVER_HOST' } }
Environment variable:
NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
- PHP
-
Here's an overview of the settings. For more help with configuration, see Distributed tracing for the PHP agent
Type Required configuration Standard distributed tracing Configuration examples:
-
Configuration file (
newrelic.ini
):newrelic.distributed_tracing_enabled = true
Infinite Tracing Configuration examples:
-
Configuration file (
newrelic.ini
):newrelic.distributed_tracing_enabled = true newrelic.span_events_enabled = true newrelic.infinite_tracing.trace_observer.host= "YOUR_TRACE_OBSERVER_HOST"
-
- Python
-
Here's an overview of the settings. For more help with configuration, see Python agent configuration
Type Required configuration Standard distributed tracing Configuration file (
newrelic.ini
):distributed_tracing.enabled = true
Environment variable:
NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing Use the following installation command:
pip install newrelic[infinite-tracing]
Configuration file (
newrelic.ini
):distributed_tracing.enabled = true infinite_tracing.trace_observer_host= YOUR_TRACE_OBSERVER_HOST
Environment variable:
NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
- Ruby
-
Here's an overview of the settings. For more help with configuration, see Ruby agent configuration.
Type Required configuration Standard distributed tracing Configuration examples:
-
Configuration file (
newrelic.yml
):distributed_tracing: enabled: true
- Environment variable:
NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
Infinite Tracing Configuration examples:
-
Configuration file (
newrelic.yml
):distributed_tracing: enabled: true infinite_tracing: trace_observer: host: 'YOUR_TRACE_OBSERVER_HOST'
-
Environment variable:
NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST"
-
If you need help with proxy configuration, see Proxy support.
Step 4. View traces
After you configure your APM agents to send data to your trace observer, you're ready to view traces. Here are two alternatives:
- View traces that include a specific service
-
The Entity explorer helps you navigate to a specific service so you can see traces that include that service.
- Go to one.newrelic.com.
- Click Entity explorer in the top menu bar.
- Filter to the service you enabled for Infinite Tracing by typing the service name, and then press Enter.
- In the left navigation's Monitor section, click Distributed tracing.
- View traces across accounts
-
This option allows you to search all traces across all New Relic accounts in your organization that you have access to.
- Go to one.newrelic.com.
- Click Apps in the top menu bar.
- Under Favorites click Distributed tracing.
-
In the Find traces... search, type a search clause to find the service. For example, to query
service.name
ortrace.id
:service.name = YOUR_SERVICE_NAME trace.id = YOUR_TRACE_ID
You can also run custom queries and create custom charts of your trace data. See Query trace data.
Manual instrumentation (If automatic instrumentation doesn't work)
Recommendation: Before performing any custom instrumentation, read:
If a service is not passing the trace header to other services, you can use the distributed tracing payload APIs to instrument the calling service and the called service. The calling service uses an API call to generate a payload, which is accepted by the called service.
- Instrument the calling service
-
To instrument the calling service:
-
Ensure the version of the APM agent that monitors the calling service supports distributed tracing.
-
Invoke the agent API call for generating a distributed trace payload: C SDK | Go | Java | .NET | Node.js | PHP | Python | Ruby.
To maintain proper ordering of spans in a trace, ensure you generate the payload in the context of the span that sends it.
- Add that payload to the call made to the destination service (for example, in a header).
-
(Optional) Identify the call as an external call:
-
- Instrument the called service
-
To instrument the called service:
-
Ensure the version of the APM agent that monitors the called service supports distributed tracing.
-
If the New Relic agent on the called service does not identify a New Relic transaction, use the agent API to declare a transaction:
- C SDK
-
One way to tell that a transaction is not in progress: when
newrelic_create_distributed_trace_payload()
is called, aNULL
pointer is returned. To solve this problem, follow the procedures to create a transaction with the C SDK. - Go
-
One way to tell that a transaction is not in progress: when
Transaction.InsertDistributedTraceHeaders(h http.Header)
is called, no headers are inserted. To create a transaction, see Instrument Go transactions. - Java
-
One way to tell that a transaction is not in progress: when
Transaction.insertDistributedTraceHeaders(Headers)
is called, no headers are inserted (this API requires agent 6.4.0+). To create a transaction, see Java agent transaction-related APIs. - .NET
-
One way to tell that a transaction is not in progress:
CreateDistributedTracePayload()
returns an empty payload. To create a transaction, see Introduction to .NET custom instrumentation. - Node.js
-
One way to tell that a transaction is not in progress: the Node.js agent logs will report an error similar to this:
No transaction found when calling Transaction.acceptDistributedTracePayload.
Use
startWebTransaction
to create a web transaction orstartBackgroundTransaction
to capture a non-web transaction. - PHP
-
One way to tell that a transaction is not in progress:
newrelic_insert_distributed_trace_headers()
returnsfalse
. To create a transaction, see newrelic_start_transaction. - Python
-
To tell that a transaction is not in progress: when
transaction = current_transaction()
is run,transaction
isNone
. Or, ifresult = accept_distributed_trace_payload(payload)
is run, then the result isFalse
.Use
background_task
to report a non-web transaction. For more on Python instrumentation, see Monitor transactions and segments. - Ruby
-
If you are using a Rack-based web framework and have enabled New Relic's Rack instrumentation, the Ruby agent will handle starting a transaction for you. For other use cases, see the
add_transaction_tracer
API method.
- Extract the payload from the call that you received (for example, in a header).
-
Invoke the call for accepting the payload: C SDK | Go | Java | .NET | PHP | Node.js | Python | Ruby.
-