Distributed tracing allows you to see the entire journey of your requests throughout a distributed system. For the Python agent, we offer two types of distributed tracing. For more details, see How span sampling works:
Standard (head-based sampling): Before any traces arrive, we determine a set percentage of traces to accept and analyze. This gives you a solid starting point to see how tracing can help you. It's turned on by default in the New Relic Python agent.
Infinite Tracing (tail-based sampling): Our cloud-based service accepts all your traces and then sorts through them to find the most important. Infinite tracing analyzes all of your traces and gives you configuration options to sample the traces that matter most to you.
Whether you just want to try out standard distributed tracing (head-based sampling) or also want to set up Infinite Tracing (tail-based sampling), you need to start by setting up standard tracing.
Standard distributed tracing
This is the best approach to set up standard distributed tracing if you haven't installed any APM agents for your services yet.
Tip
When you install the New Relic Python agent, standard distributed tracing is turned on by default. If you prefer to turn it off, see our configuration guide.
Identify services
Figure out which services you want to instrument so they each send trace data to New Relic.
Instrument each service with an APM agent
For each service involved in your transactions, you'll perform separate installations of the agent. If some of your services use other languages, simply repeat the installation steps for those languages.
To start the installation routine, click the tile below. When you're finished installing each agent, return here to see tips for viewing your traces.
After you instrument each of your services with APM agents, generate some traffic in your application so we can capture some traces. Here are two ways to view your traces in the UI:
Here's one way you can see traces for a particular service:
In the left pane's Monitor section, click Distributed tracing.
For details, click on an individual trace. If Group similar traces is on in the top menu, click on a trace group, and then click on an individual trace.
If you don't see the traces you want, you can filter by the trace.id.
For details, click on an individual trace. If Group similar traces is on in the top menu, click on a trace group, and then click on an individual trace.
If you don't see the traces you want, you can filter by the trace.id.
You can bring your logs and tracing details together to make troubleshooting easier and faster. With logs in context, you can see log messages alongside traces in the New Relic UI.
Standard distributed tracing for APM agents captures up to 10% of your traces, but if you want us to analyze all your data and find the most relevant traces, you can set up Infinite Tracing.
Complete the setup for standard distributed tracing
The Infinite Tracing setup builds on standard distributed tracing. So, make sure you've completed the steps above, and then continue with the trace observer setup.
Set up the trace observer
The trace observer is a New Relic AWS-based service that collects and analyzes all your traces. Follow the instructions in Set up trace observer. When you're done, return here with your trace observer information and continue with the next step to configure the agent.
Configure the agent for Infinite Tracing
Infinite Tracing configuration settings include the standard distributed tracing plus information about the trace observer. Note that server-side configuration is not available for Infinite Tracing.
As of v8.7.0, the Python agent supports both batching and compression. Both settings are defaulted to true and can be toggled with the following options:
Configuration file (newrelic.ini):
infinite_tracing.batching=true # Send spans to the trace observer in batches rather than one by one
infinite_tracing.compression=true # Enable gzip compression for data sent via Infinite Tracing
Environment variables:
NEW_RELIC_INFINITE_TRACING_BATCHING=true
NEW_RELIC_INFINITE_TRACING_COMPRESSION=true
Tip
If you need help with proxy configuration, see Proxy support.
(Optional) Customize Infinite Tracing
After you add the agent configuration settings, you should start seeing data in the New Relic UI. After you spend some time analyzing your data, you may want to adjust some of the features of Infinite Tracing:
If a service is not passing the trace header to other services, you can use the distributed tracing payload APIs to instrument the calling service and the called service. The calling service uses an API call to generate a payload, which is accepted by the called service.
To instrument the calling service:
Ensure the version of the APM agent that monitors the calling service supports distributed tracing.
Invoke the agent API call for generating a distributed trace payload (see the Python agent API).
Important
To maintain proper ordering of spans in a trace, ensure you generate the payload in the context of the span that sends it.
Add that payload to the call made to the destination service (for example, in a header).
(Optional) Identify the call as an external call (see the Python agent API).
To instrument the called service:
Ensure the version of the APM agent that monitors the called service supports distributed tracing.
If the New Relic agent on the called service does not identify a New Relic transaction, use the agent API to declare a transaction. Here's one way to tell that a transaction is not in progress: when transaction = current_transaction() is run, transaction is None. Or, if result = accept_distributed_trace_payload(payload) is run, then the result is False.