Enable distributed tracing

This document explains:

In addition to APM data, our distributed tracing feature supports these sources of data:

APM compatibility and requirements

To get New Relic APM data in the distributed tracing UI, you must have an APM Pro subscription.

Before enabling distributed tracing for New Relic APM agents, read the transition guide to learn about effects on existing APM features and set-up recommendations.

Distributed tracing is supported by the following APM agent versions. For best results, update existing agents to the latest version:

Enable distributed tracing for APM agents

For best results, enable distributed tracing for as many services and applications as possible. This will maximize the level of detail in your traces and help you better analyze issues.

To enable distributed tracing for an APM agent:

  1. Read the transition guide to learn about:
    • Effects of enabling distributed tracing on existing APM features.
    • Recommendations for setting up distributed tracing.
  2. Update your APM agent to versions that support distributed tracing.
  3. For each application that will use distributed tracing, use the agent's config file or other available configuration options to enable the feature: C SDK | Go | Java | .NET | Node.js | PHP | Python | Ruby.

To enable other sources of data, see:

Configure intermediaries that block tracing

New Relic's distributed tracing works by propagating header information from service to service in a request path. Some services may communicate through a proxy or other intermediary service that does not automatically propagate the header. In that case, you will need to configure that proxy so that it allows the newrelic header value to be propagated from source to destination.

For more information on intermediaries, see Troubleshooting.

What's next?

Once you've enabled distributed tracing, here are some potential next steps:

Manual instrumentation: Use API to pass distributed tracing header

Before performing any custom instrumentation, we recommend reading:

If a service is not passing the trace header to other services, you can use the distributed tracing payload APIs to instrument the calling service and/or the called service. The calling service uses an API call to generate a payload, which is accepted by the called service.

To instrument the calling service:

  1. Ensure the version of the APM agent that monitors the calling service supports distributed tracing.

  2. Invoke the agent API call for generating a distributed trace payload: C SDK | Go | Java | .NET | Node.js | PHP | Python | Ruby.

    To maintain proper ordering of spans in a trace, ensure you generate the payload in the context of the span that sends it.

  3. Add that payload to the call made to the destination service (for example, in a header).
  4. Optional: Identify the call as an external call:

To instrument the called service:

  1. Ensure the version of the APM agent that monitors the called service supports distributed tracing.

  2. If the New Relic agent on the called service does not identify a New Relic transaction, use the agent API to declare a transaction:

    C SDK

    One way to tell that a transaction is not in progress: when newrelic_create_distributed_trace_payload() is called, a NULL pointer is returned. To solve this problem, follow the procedures to create a transaction with the C SDK.

    Go

    One way to tell that a transaction is not in progress: when Transaction.InsertDistributedTraceHeaders(h http.Header) is called, no headers are inserted. To create a transaction, see Instrument Go transactions.

    Java

    One way to tell that a transaction is not in progress: when createDistributedTracePayload() is called, an empty string is returned. To create a transaction, see Java agent transaction-related APIs.

    .NET

    One way to tell that a transaction is not in progress: CreateDistributedTracePayload() returns an empty payload. To create a transaction, see Introduction to .NET custom instrumentation.

    Node.js

    One way to tell that a transaction is not in progress: the Node.js agent logs will report an error similar to:

    No transaction found when calling Transaction.acceptDistributedTracePayload.

    Use startWebTransaction to create a web transaction or startBackgroundTransaction to capture a non-web transaction.

    PHP

    One way to tell that a transaction is not in progress: newrelic_create_distributed_trace_payload() returns an empty payload. To create a transaction, see newrelic_start_transaction.

    Python

    To tell that a transaction is not in progress: when transaction = current_transaction() is run, transaction is None. Or, if result = accept_distributed_trace_payload(payload) is run, then the result is False.

    Use background_task to report a non-web transaction. For more on Python instrumentation, see Monitor transactions and segments.

    Ruby

    If you are using a Rack-based web framework and have enabled New Relic's Rack instrumentation, the Ruby agent will handle starting a transaction for you. For other use cases, see the add_transaction_tracer API method.

  3. Extract the payload from the call that you received (for example, in a header).
  4. Invoke the call for accepting the payload: C SDK | Go | Java | .NET | PHP | Node.js | Python | Ruby.

For more help

Recommendations for learning more: