Enable distributed tracing

New Relic’s distributed tracing monitors requests across a distributed system. This document explains how to enable distributed tracing for New Relic APM agents, and how to manually implement distributed tracing when it does not automatically work for your application.

Compatibility and requirements

Distributed tracing requires a New Relic APM Pro subscription.

Before enabling distributed tracing, read the transition guide to learn about effects on existing APM features and set-up recommendations.

Distributed tracing is supported by the following APM agent versions. For best results, update your existing agent to the latest version:

New Relic monitoring for AWS Lambda supports distributed tracing. For details, see Lambda monitoring requirements.

Enable distributed tracing

For best results, enable distributed tracing for as many services and applications as possible. This will maximize the level of detail in your traces and help you better analyze issues.

To enable distributed tracing:

  1. Read the transition guide to learn about:
    • Effects of enabling distributed tracing on existing APM features.
    • Recommendations for setting up distributed tracing.
  2. Update your APM agent to versions that support distributed tracing.
  3. For each application that will use distributed tracing, use the agent's config file or other available configuration options to enable the feature: C SDK | Go | Java | .NET | Node.js | PHP | Python | Ruby.

Configure proxies that block tracing

New Relic's distributed tracing works by propagating header information from service to service in a request path. Some services may communicate through a proxy that does not automatically propagate the header. In that case, you will need to configure that proxy so that it allows the newrelic header value to be propagated from source to destination. For more information, see the proxy's documentation.

What's next?

Once you've enabled distributed tracing, here are some potential next steps:

Use API to pass distributed tracing header

Before performing any custom instrumentation, we recommend reading:

If a service is not passing the trace header to other services, you can use the distributed tracing payload APIs to instrument the calling service and/or the called service. The calling service uses an API call to generate a payload, which is accepted by the called service.

To instrument the calling service:

  1. Ensure the version of the APM agent that monitors the calling service supports distributed tracing.

  2. Invoke the agent API call for generating a distributed trace payload: C SDK | Go | Java | .NET | Node.js | PHP | Python | Ruby.

    To maintain proper ordering of spans in a trace, ensure you generate the payload in the context of the span that sends it.

  3. Add that payload to the call made to the destination service (for example, in a header).
  4. Optional: Identify the call as an external call:

To instrument the called service:

  1. Ensure the version of the APM agent that monitors the called service supports distributed tracing.

  2. If the New Relic agent on the called service does not identify a New Relic transaction, use the agent API to declare a transaction:

    C SDK

    One way to tell that a transaction is not in progress: when newrelic_create_distributed_trace_payload() is called, a NULL pointer is returned. To solve this problem, follow the procedures to create a transaction with the C SDK.

    Go

    One way to tell that a transaction is not in progress: when CreateDistributedTracePayload() is called, an empty string is returned. To create a transaction, see Instrument Go transactions.

    Java

    One way to tell that a transaction is not in progress: when createDistributedTracePayload() is called, an empty string is returned. To create a transaction, see Java agent transaction-related APIs.

    .NET

    One way to tell that a transaction is not in progress: CreateDistributedTracePayload() returns an empty payload. To create a transaction, see Introduction to .NET custom instrumentation.

    Node.js

    One way to tell that a transaction is not in progress: the Node.js agent logs will report an error similar to:

    No transaction found when calling Transaction.acceptDistributedTracePayload.

    Use startWebTransaction to create a web transaction or startBackgroundTransaction to capture a non-web transaction.

    PHP

    One way to tell that a transaction is not in progress: newrelic_create_distributed_trace_payload() returns an empty payload. To create a transaction, see newrelic_start_transaction.

    Python

    To tell that a transaction is not in progress: when transaction = current_transaction() is run, transaction is None. Or, if result = accept_distributed_trace_payload(payload) is run, then the result is False.

    Use background_task to report a non-web transaction. For more on Python instrumentation, see Monitor transactions and segments.

    Ruby

    If you are using a Rack-based web framework and have enabled New Relic's Rack instrumentation, the Ruby agent will handle starting a transaction for you. For other use cases, see the add_transaction_tracer API method.

  3. Extract the payload from the call that you received (for example, in a header).
  4. Invoke the call for accepting the payload: C SDK | Go | Java | .NET | PHP | Node.js | Python | Ruby.

For more help

Recommendations for learning more: