Enable distributed tracing

New Relic’s distributed tracing monitors requests across a distributed system. This document explains how to enable distributed tracing for supported New Relic agents and how to manually implement distributed tracing.

Compatibility and requirements

An APM Pro subscription is required for distributed tracing. In addition, make sure your APM agent version supports this feature:

  • Go: 2.1.0 or higher
  • Java: 4.3.0 or higher
  • .NET: 8.6.45.0 or higher
  • Node.js​: 4.7.0 or higher
  • PHP: 8.4 or higher
  • Python: 4.2.0.100 or higher
  • Ruby: 5.3.0.346 or higher

Enable distributed tracing

Before you enable distributed tracing, read the transition guide for information about effects on APM features and recommendations on system-wide rollout.

For best results, enable distributed tracing for as many services and applications in your system as possible. This will maximize the level of detail in your traces and will help you analyze issues.

To enable distributed tracing:

  1. Update APM agents to versions that support distributed tracing.
  2. For each app using distributed tracing, use the agent's config file or other available configuration options to enable the feature:

Configure proxies that block tracing

New Relic's distributed tracing works by propagating header information from service to service in a request path. Some services may communicate through a proxy that does not automatically propagate the header. In that case, you will need to configure that proxy so that it allows the newrelic header value to be propagated from source to destination. Consult the proxy's documentation for specific instructions.

What's next?

Once you've enabled distributed tracing, here are some potential next steps:

Distributed tracing payload APIs

When you enable distributed tracing for your New Relic-monitored applications and services, for the most part, you will see complete and detailed traces. However, for some services or setups, you may need to do some manual instrumentation to get more detail or to see connections between services.

There are two main ways to manually add instrumentation:

  • Add detail within a service: use APM custom instrumentation to report transactions and segments.
  • Connect services: if two services are not seeing each other in the distributed tracing UI, you will need to manually instrument them to pass trace context data via headers using the distributed tracing payload APIs (below).

Use the distributed tracing payload APIs to instrument both the calling service and the called service. The calling service uses an API call to generate a payload, which is accepted by an API call placed in the service being called.

Instrument the calling service

To instrument the calling service:

  1. Ensure that your APM agent monitoring the calling service supports distributed tracing.

  2. Invoke the agent API call for generating a distributed trace payload. In order to maintain proper ordering of spans in a trace, be sure to generate the payload in the context of the span that sends it.

  3. Add that payload to the call made to the destination service (for example, in a header).
  4. Optional: Identify that call as an external call:

Instrument the called service

To instrument the called service:

  1. Ensure that your APM agent monitoring the called service supports distributed tracing.

  2. If the New Relic agent on the called service does not identify a New Relic transaction, use the agent API to declare a transaction:

    Go

    One way to tell that a transaction is not in progress: when CreateDistributedTracePayload() is called, an empty string is returned. To create a transaction, see Instrument Go transactions.

    Java

    One way to tell that a transaction is not in progress: when createDistributedTracePayload() is called, an empty string is returned. To create a transaction, see Java agent transaction-related APIs.

    .NET

    One way to tell that a transaction is not in progress: CreateDistributedTracePayload() returns an empty payload. To create a transaction, see Introduction to .NET custom instrumentation.

    Node.js

    One way to tell that a transaction is not in progress: the Node.js agent logs will report an error similar to:

    No transaction found when calling Transaction.acceptDistributedTracePayload.

    Use startWebTransaction to create a web transaction or startBackgroundTransaction to capture a non-web transaction.

    PHP

    One way to tell that a transaction is not in progress: newrelic_create_distributed_trace_payload() returns an empty payload. To create a transaction, see newrelic_start_transaction.

    Python

    To tell that a transaction is not in progress: when transaction = current_transaction() is run, transaction is None. Or, if result = accept_distributed_trace_payload(payload) is run, then the result is False.

    Use background_task to report a non-web transaction. For more on Python instrumentation, see Monitor transactions and segments.

    Ruby

    If you are using a Rack-based web framework and have enabled New Relic's Rack instrumentation, the Ruby agent will handle starting a transaction for you. For other use cases, see the add_transaction_tracer API method.

  3. Extract the payload from the call that you received (for example, in a header).
  4. Invoke the call for accepting the payload:

For more help

Recommendations for learning more: