Distributed tracing helps you monitor and analyze the behavior of your distributed system. After you enable distributed tracing, you can use our UI tools to search for traces and analyze them.
For example, let's say you are an engineer troubleshooting errors in a complex transaction spanning many services. Here's what you can do in our UI:
- Open the distributed tracing UI page.
- Sort through your traces using a filter to find that specific request and show only traces containing errors.
- On the trace details page, you review the span along the request route that originated the error.
- Noting the error class and message, navigate to the service from its span in the trace so you can see that the error is occurring at a high rate.
Read on to explore the options in the distributed tracing UI.
Open the distributed tracing UI
Here's how you can access the distributed tracing UI, depending on the type of search you want to do:
Find traces that are useful
We have a variety of tools to help you find traces and spans so you can resolve issues. The opening distributed tracing page is populated with a default list of traces, and you can quickly refine this list using these tools:
In addition to these UI tools, you can also use other options mentioned in Query distributed trace data.
Filter using the query bar
The Find traces query bar is a quick way to narrow your search for traces. You can either start typing in the query bar or use the dropdown to create a compound query.
Query returns are based on span attributes, not on trace attributes. You define spans that have certain criteria, and the search displays traces that contain those spans.
If you use a multi-attribute filter, it is affected by first attribute selected. Distributed tracing reports on two types of data: transaction events and spans. When you select an attribute in the filter, the data type that attribute is attached to dictates the available attributes. For example, if you filter on an attribute that is attached to a transaction event, only transaction event attributes are available when you attempt to add filter on additional attribute values.
Queries for traces are similar to NRQL (our query language). Here are the main exceptions:
- String values don't require quote marks (for example, you can use either
appName = MyAppor
appName = 'MyApp')
likeoperator doesn’t require
%(for example, you can use either
appName like productor
appName like %product%).
Some queries that return a large number of results may return false positives. The trace list limits these incorrect results to 10% of the returned results. False positives may also result in histogram chart results that are not displayed in the trace list.
Here are two query bar examples:
Besides the query bar at the top of the page, you can use a variety of filters in the left pane to find traces you're interested in.
Infinite tracing data: Select this only to show traces related to the Infinite Tracing feature.
Multi span traces only: Use this to hide single-span traces.
Error filters: Under Errors select the errors to filter by.
Histogram filters: Below Errors at the bottom of the left pane, you can click More filters to show histogram filters. Drag the sliders to change the values, such as Trace duration:
- Drag the left end of the slider for greater-than comparisons.
- Drag the right end of the slider for less-than comparisons.
- Drag each end of the slider toward the center to filter by a range.
When you drag the slider, it changes both the list of traces and what's shown in the trace charts.
The default view of distributed tracing shows traces grouped by the same root entry span. In other words, traces are grouped by the span where New Relic began recording the request. You can slide the toggle Group similar traces to turn this on and off.
With trace groups you get a high-level view of traces so you can understand request behavior for groups of similar traces. This helps you understand dips or spikes in trace count, duration, and errors. When you click on one of the trace groups, you get all the standard details in context of the specific trace group you selected.
You can control what you see in the trace groups list by using left-pane filters.
Filter using the scatter plot
The trace scatter plot is a quick way to search for outlying traces. This is available on the opening page of distributed tracing if you turn off the Group similar traces toggle at the top of the page.
In the scatter plot, you can move the cursor across the chart to view trace details and you can click individual points to get details:
Here's how you can control what's displayed in the scatter plot:
- In the View by dropdown, select the duration type:
- Backend duration
- Root span duration
- Trace duration
- In Facet traces by, select one of these options:
- Root entry span: Group by the root transaction, which is the root service's endpoint. In a trace where Service A calls Service B and Service B calls Service C, the root entry span is Service A's endpoint. For example: "Service A - GET /user/%".
- Root entity: Group by the name of the first entity in traces. In a trace where Service A calls Service B and Service B calls Service C, the root entity would be Service A.
- Errors: Group by whether or not traces contain errors.
- For tips about how to change the filters on the scatter plot, see Left-pane filters.
When you open distributed tracing, you'll see three charts at the top (if Group similar traces is turned on). These charts show you the trace-count, 95th percentile duration, and error-count for each of the trace groups listed in the table below the charts.
To change the filters on these charts, see Left-pane filters.
Some queries that produce many results may result in false positives in charts. This could manifest as charts showing trace results that are not in the trace list.
Trace details UI page
When you select a trace from the trace list, you see an entity map at the top followed by a waterfall view displaying that trace's spans.
When you select a trace from the list of traces, you see details about it.
If a trace involves more than one entity, and we have information about those entities, the trace page includes a map.
The waterfall view in the bottom section of the page displays a detailed flow of all the operations that make up the distributed trace. This allows you to see the exact context from which latency and errors arise.
The waterfall view has a number of controls to make it easier to find spans that you care about:
- Expand all/Collapse all: Use this option to toggle between showing every span (including in-process spans) and showing collapsed spans.
- Standard/Manual: In standard mode, you get a semi-expanded view that includes the first span in each process. In manual mode, you only see what you expand.
- Both Standard and Manual are overridden by Expand all, which displays all spans.
- If you're in standard or manual mode, and you click Collapse all, the waterfall is reset to the default for the mode you're using.
- Focus on: If your span has errors or anomalies, you'll see checkboxes for each of these types. If you check these boxes, your waterfall view will collapse all other types of spans and only show color-coded error or anomalous spans. This is a quick way to find problematic spans.
- Maximize: This allows you to see more spans on the screen by expanding the waterfall and hiding the entity map.
Waterfall span properties
The UI indicates span properties with these icons:
This icon represents a span that's a service's entry point.
This icon represents an in-process span, which is a span that takes place within a process (as opposed to a cross-process span). Examples: middleware instrumentation, user-created spans.
This icon represents a span call to a datastore.
This icon represents category representing a call to an external service made via HTTP.
This icon represents a browser application span.
This icon represents a span from a Lambda function.
Some spans will have additional indicators:
Type of connection
Solid lines indicate a direct parent-child relationship; in other words, one process or function directly calling another. A dotted line indicates a non-direct relationship. For more on relationships between spans, see Trace structure.
A span with an error. See How to understand span errors.
This icon represents the detection of an anomalous span.
Some spans may be "orphaned," or separated, from the trace. These spans will appear at the bottom of the trace. For more details, see Fragmented traces.
Multiple app names
When beside a span name, this represents an entity that has had multiple app names set. Select this to see all app names it reports to. To search trace data by alternate app names, use the
Client/server time difference
If a span's duration indicator is not completely colored in (like in this example), it means that there is a time discrepancy between the server-side duration and the client-side duration for that activity. For details on this, see Client/server time difference.
For more on the trace structure and how span properties are determined, see Trace structure.
Span details pane
When you select a span, a pane opens up with span details. These details can be helpful for troubleshooting performance issues. This page has three tabs:
- Performance: You’ll see charts showing the average duration and throughput for that span operation, as well as how the performance of this specific span compares to the average.
- Attributes: You can learn more about attributes from our data dictionary.
- Details: You can view details, such as the span's full name and data source.
What a span displays is based on its span type. For example, the datastore span's details will include the database query.
View related logs
If you are using our logs in context feature, you can see any logs that are linked to your traces:
- Go to the trace details page by clicking on a trace.
- In the upper-left corner of the span details page, click the Logs tab.
- For details related to an individual log message, click directly on the message.
Additional UI details
Here are some additional distributed tracing UI details, rules, and limits: