• /
  • Log in
  • Free account

Understand and use the distributed tracing UI

Distributed tracing helps you monitor and analyze the behavior of your distributed system. After you enable distributed tracing, you can use our UI tools to search for traces and analyze them.

For example, let's say you are an engineer troubleshooting errors in a complex transaction spanning many services. Here's what you can do in our UI:

  1. Open the distributed tracing UI page.
  2. Sort through your traces using a filter to find that specific request and show only traces containing errors.
  3. On the trace details page, you review the span along the request route that originated the error.
  4. Noting the error class and message, navigate to the service from its span in the trace so you can see that the error is occurring at a high rate.

Read on to explore the options in the distributed tracing UI.

See distributed tracing UI

Here's how you can access the distributed tracing UI, depending on the type of search you want to do:

Sort through your traces

The opening distributed tracing page is populated with a list of traces, and you can quickly refine this list using these tools:


In addition to these tools, you can also use other options mentioned in Query distributed trace data.

Filter using the query bar

The Find traces query bar is a quick way to narrow your search for traces. You can either start typing in the query bar or use the dropdown to create a compound query.

Query returns are based on span attributes, not on trace attributes. You define spans that have certain criteria, and the search displays traces that contain those spans.

If you use a multi-attribute filter, it is affected by first attribute selected. Distributed tracing reports on two types of data: transaction events and spans. When you select an attribute in the filter, the data type that attribute is attached to dictates the available attributes. For example, if you filter on an attribute that is attached to a transaction event, only transaction event attributes are available when you attempt to add filter on additional attribute values.

Queries for traces are similar to NRQL (our query language). Here are the main exceptions:

  • String values don't require quote marks (for example, you can use either appName = MyApp or appName = 'MyApp')
  • The like operator doesn’t require % (for example, you can use either appName like product or appName like %product%).


Some queries that return a large number of results may return false positives. The trace list limits these incorrect results to 10% of the returned results. False positives may also result in histogram chart results that are not displayed in the trace list.

Here are two query bar examples:

Filter using the scatter plot

The trace scatter plot on the opening page of distributed tracing is a quick way to search for outlying traces. You can move the cursor across the chart to view trace details and you can click individual points to get details.

Screenshot showing the distributed tracing scatter plot chart.

Go to one.newrelic.com > Apps > Favorites > Distributed tracing or go to one.newrelic.com > APM > (select an app) > Monitor > Distributed tracing.

When viewing the trace scatter plot, you can select the duration type in View by, and you can select one of these options in Group traces by:

  • Errors: Group by whether traces contain errors or not.
  • Root service: Group by the name of the first service in traces. In a trace where Service A calls Service B and Service B calls Service C, the root service would be Service A.
  • Root entry span: Group by the root transaction, which is the root service's endpoint. In a trace where Service A calls Service B and Service B calls Service C, the root entry span is Service A's endpoint. For example: "Service A - GET /user/%".
  • Service entry span: Group by the span name of the service currently being viewed in APM. For example, for a trace where Service A calls Service B and Service B calls Service C, if you are viewing Service B in APM and select this grouping, the traces will be represented by their Service B span names. If a service has multiple spans in a trace, this grouping option will use that service's first entry point.

Filter traces using histograms


Trace histogram charts are available only on the global distributed tracing page.

The histogram charts on the global distributed tracing page give you a quick understanding of trace distribution for important values, like duration. You can use the sliders under the charts to control what traces are displayed. For example, you can drag the Trace duration chart slider to show only traces over 500 ms, as shown in the histogram example below.

New Relic One distributed tracing - histogram


Some queries that produce many results may result in false positives in histograms. This could manifest as histograms showing trace results that are not in the trace list.

Trace details UI page

When you select a trace from the trace list, you see that trace's timeline and spans:

New Relic distributed tracing UI - trace details page

one.newrelic.com > APM > (select an application) > Monitor > Distributed tracing > (select a trace) > (select a span): See the spans in a trace. Examine individual span details and see notifications for spans with anomalous behavior.

The UI indicates some span properties with icons:

Span property




New Relic distributed tracing service icon

This icon represents a span that's a service's entry point.


New Relic distributed tracing in-process span icon

This icon represents an in-process span, which is a span that takes place within a process (as opposed to a cross-process span). Examples: middleware instrumentation, user-created spans.


New Relic distributed tracing datastore span icon

This icon represents a span call to a datastore.


New Relic distributed tracing external span icon

This icon represents category representing a call to an external service made via HTTP.

Browser app

New Relic distributed tracing browser span icon

This icon represents a browser application span.


New Relic distributed tracing external span icon

This icon represents a span from a Lambda function.

Some spans will have additional indicators:

Span property



Type of connection

New Relic distributed tracing connecting lines image

Solid lines indicate a direct parent-child relationship; in other words, one process or function directly calling another. A dotted line indicates a non-direct relationship. For more on relationships between spans, see Trace structure.


New Relic distributed tracing error icon

A span with an error. See How to understand span errors.


New Relic distributed tracing datastore span icon

This icon represents the detection of an anomalous span.

Orphaned spans

New Relic distributed tracing orphaned span icon

Some spans may be "orphaned," or separated, from the trace. These spans will appear at the bottom of the trace. For more details, see Fragmented traces.

Multiple app names

New Relic distributed tracing multiple app names indicator

When beside a span name, this represents an entity that has had multiple app names set. Select this to see all app names it reports to. To search trace data by alternate app names, use the appName attribute.

Client/server time difference

New Relic distributed tracing client-server time difference indicator

If a span's duration indicator is not completely colored in (like in this example), it means that there is a time discrepancy between the server-side duration and the client-side duration for that activity. For details on this, see Client/server time difference.

For more on the trace structure and how span properties are determined, see Trace structure.

Span details pane

When you select a span, a pane opens up with span details. These details can be helpful for troubleshooting performance issues. Details include:

What a span displays is based on its span type. For example, a datastore span's name attribute will contain the datastore query.

View related logs

If you are using our logs in context feature together with our log management, you can see any logs that are linked to your traces:

  1. Go to the trace details page by clicking on a trace.
  2. Click See logs in the upper-right corner.
  3. For details related to an individual log message, click directly on the message.

Additional UI details

Here are some additional distributed tracing UI details, rules, and limits:

For more help

If you need more help, check out these support and learning resources:

Create issueEdit page
Copyright © 2021 New Relic Inc.