Troubleshooting: Missing trace data

Problem

You have enabled distributed tracing but do not see data you expect to see in the distributed tracing UI.

Solution

Before performing troubleshooting, we recommend reading How distributed tracing works.

Here are some reasons you might not see or be able to find expected data in the distributed tracing UI:

Service doesn't have distributed tracing enabled

In order for distributed tracing to report details for all nodes in a trace, each application must be monitored by a New Relic agent that has had distributed tracing enabled.

If an application's New Relic account has not had distributed tracing enabled, it would have these issues:

  • Its distributed tracing UI page wouldn't have data.
  • It wouldn't report data to other account's distributed traces.
A proxy (or other intermediary) isn't passing headers

Some services may communicate through a proxy or other intermediary that doesn't automatically propagate the distributed tracing header. If that's the case, you must configure that component to allow the header to be passed from source to destination. For instructions, consult the documentation for that intermediary component.

A service may require manual instrumentation

When you enable distributed tracing for applications and services that New Relic automatically instruments, you'll usually see complete and detailed data for those nodes in the distributed tracing UI.

However, you may notice that some services or applications are missing from traces, or that there are some internal spans you expect to see that are missing. If that's the case, you may want to implement custom instrumentation of applications or specific transactions to see more detail in traces. Some examples of when you may need to do this:

  • Transactions not automatically instrumented. To ensure your application is automatically instrumented, read the compatibility and requirements documentation for the New Relic agent you're using. If an application isn't automatically instrumented, or if you'd like to add instrumentation of specific activity, see Custom instrumentation.
  • All Go applications. The Go agent, unlike other agents, requires manual instrumentation of your code. For instructions, see Instrument a Go application.
  • A service doesn't use HTTP. If a service doesn't communicate via HTTP, the New Relic agent won't send distributed tracing headers. This may be the case for some non-web applications or message queues. To remedy this, use the distributed tracing APIs to instrument either the calling or called application.
Details may be obfuscated for accounts you don't have access to

If a trace contains data from applications monitored by multiple New Relic accounts, and your user permissions don't allow you to access those accounts, some of the span and service details will be obfuscated in the UI.

Sampling process means not all activity will be traced

Distributed tracing uses adaptive sampling. This means that a percentage of calls to a service will be reported as part of a distributed trace. Specific calls to your service might not have been selected to be sampled sampled as part of a distributed trace.

Span limits may be maxed out

There are limits on the number of spans that can be collected and displayed. If an application generates a very large number of spans for a single call, it might exceed the agent's span-collection limit for that harvest cycle. This could result in missing spans and significantly limit the number of traces the agent can completely sample and report.

Search by entity.name not finding associated app names

Potential cause: For applications that have multiple app names, the entity.name attribute will be associated only with the primary app name. To search by other app names, search using the appName attribute.

For more help

Recommendations for learning more: