--- title: Understand the trace details UI page source: https://docs.newrelic.com/docs/distributed-tracing/ui-data/trace-details --- Once you select a trace from the trace list, you'll be directed to the trace details page in the UI. Use trace details to understand the flow of operations that make up a distributed trace, and adjust the view to find the information most important to you. ![New Relic distributed tracing UI - trace details page](https://docs.newrelic.com/images/distributed-tracing-trace-details-page.webp "Trace details page") Select a trace from the list of traces to see details about it. Check out ways to display your traces' spans: ## Timeline view The timeline view is a high-density view of all the spans in the trace. This view allows you to quickly scroll through the entire trace and see how latency and errors propagate through the trace and across entity boundaries. You can select spans from this view to see span details in the waterfall view below it. ## Latency view The latency view shows you how much latency is contributed by each entity in the trace, as well as the number of error spans per entity. The latency contributed by an entity is the wall-clock time that it had one or more processes running but was **not** making any external or database calls. Here are two examples: - If an entity has a process lasting 500ms, but 400ms of that time it's making an external call, it's shown as contributing 100ms of latency to the trace. - If an entity has three processes running in parallel for one second, and none of them made external calls, it's shown as contributing one second of latency to the trace. ![Screenshot showing the distributed tracing latency view](https://docs.newrelic.com/images/distributed-tracing-trace-details-page-latency-view.webp "Latency view") ## Waterfall view The waterfall view in the bottom section of the page displays a detailed flow of all the operations that make up the distributed trace. This allows you to see the exact context from which latency and errors arise. ### Waterfall controls [#waterfall-controls] The waterfall view has a number of controls to make it easier to find spans that you care about: - **Expand all/Collapse all**: Use this option to toggle between showing every span (including in-process spans) and showing collapsed spans. - **Standard/Manual**: In standard mode, you get a semi-expanded view that includes the first span in each process. In manual mode, you only see what you expand. - Both **Standard** and **Manual** are overridden by **Expand all**, which displays all spans. - If you're in standard or manual mode, and you click **Collapse all**, the waterfall is reset to the default for the mode you're using. - **Focus on**: If your span has errors or anomalies, you'll see checkboxes for each of these types. If you check these boxes, your waterfall view will collapse all other types of spans and only show color-coded error or anomalous spans. This is a quick way to find problematic spans. ![Screenshot showing the color coding of focus on](https://docs.newrelic.com/images/distributed-tracing_screenshot-crop_focus-on-example.webp "Screenshot showing the color coding of focus on") - **Maximize**: This allows you to see more spans on the screen by expanding the waterfall and hiding the entity map. ### Waterfall span properties [#span-icons] The UI indicates span properties with these icons: | Span property | Indicator | Description | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Service | ![New Relic distributed tracing service icon](https://docs.newrelic.com/images/new-relic-distributed-tracing-service-icon.webp "new-relic-distributed-tracing-service-span-icon.png") | This icon represents a span that's a service's entry point. | | In-process | ![New Relic distributed tracing in-process span icon ](https://docs.newrelic.com/images/new-relic-distributed-tracing-inprocess-icon.webp "new-relic-distributed-tracing-in-process-span-icon.png") | This icon represents an [in-process span](#in-process), which is a span that takes place within a process (as opposed to a cross-process span). Examples: middleware instrumentation, user-created spans. | | Datastore | ![New Relic distributed tracing datastore span icon](https://docs.newrelic.com/images/new-relic-distributed-tracing-databases-icon.webp "new-relic-distributed-tracing-database-span-icon.png") | This icon represents a span call to a datastore. | | External | ![New Relic distributed tracing external span icon](https://docs.newrelic.com/images/new-relic-distributed-tracing-external-icon.webp "new-relic-distributed-tracing-external-span-icon.png") | This icon represents category representing a call to an external service made via HTTP. | | Browser app | ![New Relic distributed tracing browser span icon](https://docs.newrelic.com/images/distributed-tracing-browser-span-icon.webp "distributed-tracing-browser-span-icon.png") | This icon represents a [browser application span](https://docs.newrelic.com/docs/browser/new-relic-browser/browser-pro-features/browser-data-distributed-tracing). | | Lambda | ![New Relic distributed tracing external span icon](https://docs.newrelic.com/images/new-relic-distributed-tracing-lambda-icon.webp "new-relic-distributed-tracing-lambda-icon.png") | This icon represents a span from a [Lambda function](https://docs.newrelic.com/docs/serverless-function-monitoring/aws-lambda-monitoring/get-started/introduction-new-relic-monitoring-aws-lambda). | Some spans will have additional indicators: | Span property | Indicator | Description | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Type of connection | ![New Relic distributed tracing connecting lines image](https://docs.newrelic.com/images/new-relic-distributed-tracing-lines-image.webp "new-relic-distributed-tracing-lines-image.png") | Solid lines indicate a direct parent-child relationship; in other words, one process or function directly calling another. A dotted line indicates a non-direct relationship. For more on relationships between spans, see [Trace structure](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/get-started/how-new-relic-distributed-tracing-works#trace-structure). | | Errors | ![New Relic distributed tracing error icon](https://docs.newrelic.com/images/new-relic-distributed-tracing-error-icon.webp "new-relic-distributed-tracing-error-icon.png") | A span with an error. See [How to understand span errors](#error-tips). | | Anomalous | ![New Relic distributed tracing datastore span icon](https://docs.newrelic.com/images/new-relic-distributed-tracing-anomalous-icon.webp "new-relic-distributed-tracing-anomalous-icon.png") | This icon represents the detection of an [anomalous span](#anomalous-spans). | | Orphaned spans | ![New Relic distributed tracing orphaned span icon](https://docs.newrelic.com/images/new-relic-distributed-tracing-orphan-icon.webp "new-relic-distributed-tracing-fragmented-icon.png") | Some spans may be "orphaned," or separated, from the trace. These spans will appear at the bottom of the trace. For more details, see [Fragmented traces](#fragmented-traces). | | Multiple app names | ![New Relic distributed tracing multiple app names indicator](https://docs.newrelic.com/images/new-relic-distributed-tracing-multiapp-icon.webp "new-relic-distributed-tracing-multiple-app-names-indicator.png") | When beside a span name, this represents an entity that has had [multiple app names set](https://docs.newrelic.com/docs/agents/manage-apm-agents/app-naming/use-multiple-names-app). Select this to see all app names it reports to. To search trace data by alternate app names, use the `appName` attribute. | | Client/server time difference | ![New Relic distributed tracing client-server time difference indicator](https://docs.newrelic.com/images/new-relic-distributed-tracing-client-server-time-indicator.webp "new-relic-distributed-tracing-client-server-time-indicator.png") | If a span's duration indicator is not completely colored in (like in this example), it means that there is a time discrepancy between the server-side duration and the client-side duration for that activity. For details on this, see [Client/server time difference](#client-server-time). | ## Span details pane [#span-details] When you select a span, a pane opens up with span details. These details can be helpful for troubleshooting performance issues. This page has three tabs: - **Performance**: You'll see charts showing the average duration and throughput for that span operation, as well as how the performance of this specific span compares to the average. - **Attributes**: You can learn more about attributes from our [data dictionary](/attribute-dictionary/?dataSource=Distributed+Tracing&event=Span). - **Details**: You can view details, such as the [span's full name](https://docs.newrelic.com/docs/distributed-tracing/ui-data/understand-use-distributed-tracing-ui/#prettified-span-names) and data source. - **Span links**: If the span has [span links](#span-links), you'll see this tab. It shows causal relationships with spans from other traces. [Learn more about span links](#span-links). What a span displays is based on its span type. For example, the datastore span's details will include the database query. For more on the trace structure and how span properties are determined, see [Trace structure](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/get-started/how-new-relic-distributed-tracing-works#trace-structure). ## Understanding span links [#span-links] Span links help you understand causal relationships between spans across different traces. When traces get split due to asynchronous communication or long-running processes, span links allow you to navigate between related traces and see the complete picture of your distributed system. ### When traces get split [#why-traces-split] In modern distributed systems, traces can appear as separate, disconnected traces in the following scenarios: - **Long-running workflows**: Background jobs or workflows that exceed trace duration limits (90+ sec for head-based sampling, 20+ sec for Infinite Tracing) or run for extended periods. When tracing across message queues, it's generally recommended to use span links to associate traces from the producer(s) and consumer(s). Examples include: - **General asynchronous message queues**: A service publishes messages to a queue (like AWS SQS, RabbitMQ, or Kafka), and another service consumes them later - **Batch-processing**: Workers that process multiple messages in batches, where each message originated from a different trace - **Fan-in pattern**: Multiple services independently produce messages that one consumer aggregates - **Fan-out pattern**: Multiple consumers consume from one message queue - each starting its own trace When traces are split like this, traditional parent-child relationships don't capture the full picture. Span links fill this gap by creating explicit causal connections. ### What are span links? [#what-are-span-links] Span links are OpenTelemetry attributes that create causal relationships between spans that don't have a direct parent-child connection. They allow you to: - Navigate from a consuming span back to the producing span - Understand the complete request flow even when traces are split - Debug issues that span across asynchronous boundaries A span link includes the trace ID and span ID of the related span, allowing New Relic to connect the traces for you. ### Identifying traces with span links [#identify-span-links] In the trace details UI, traces with span links are marked in several ways: 1. **Span links filter badge**: In the trace filter bar, you'll see a badge showing the number of spans with links (for example, `span links (2)`). 2. **Span count indicator**: When viewing the list of spans, each span shows a link count if it has span links. ![Screenshot showing the span links filter badge in the trace details UI](https://docs.newrelic.com/images/distributed-tracing-span-links-filter-badge.webp "Distributed tracing span links filter") The span links filter badge appears in the trace filter bar when a trace contains spans with links. ### Viewing span links [#view-span-links] To view and navigate span links: 1. From the trace details page, select a span from the span links dropdown. 2. In the span details pane, click the **Span links** tab. 3. You'll see a list of linked traces with the following information: - **Direction badge**: `Forward` indicates a span that occurred later in time (successor), while `Backward` indicates a span that occurred earlier (predecessor). - **Trace ID**: The ID of the linked trace (truncated for display). - **Timestamp**: When the linked span occurred. - **Duration**: How long the linked span took. - **Errors**: Number of errors in the linked trace. 4. Click on a linked trace to navigate to it. Use your browser's back button to return to the original trace. ![Screenshot showing the Span links tab with forward and backward linked traces](https://docs.newrelic.com/images/distributed-tracing-span-links-tab-detail.webp "Span links tab showing linked traces") The Span links tab displays linked traces with direction badges (Forward/Backward), trace IDs, timestamps, durations, and error counts. > #### 💡 TIP > > **Understanding forward vs. backward links** > > - **Forward links**: Point to traces that happened as a result of the current span. Example: A message was published by this span and processed later by another trace. > - **Backward links**: Point to traces that caused the current span. Example: This span is processing a message that was published by an earlier trace. ### When span links don't appear [#span-links-troubleshooting] If you expect to see span links but they're not appearing, check the following: ![Screenshot showing the empty state message when a trace has no span links](https://docs.newrelic.com/images/distributed-tracing-span-links-empty-state.webp "Empty state when trace has no span links") When a trace has no span links, you'll see this message in the Span links tab. **Trace context not propagated** Span links require proper trace context propagation through your messaging system: - Ensure W3C trace context headers (`traceparent`, `tracestate`) are included in message headers or metadata - Verify your messaging library or framework supports context propagation - When instrumenting manually, make sure you're extracting context from messages and creating span links programmatically For implementation examples, see [OpenTelemetry span links best practices](https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-best-practices-traces/#span-links). **Sampling considerations** Both the original trace and the linked trace must be sampled for span links to appear: - If either trace is dropped by sampling, the span link won't be visible - Consider using head-based or tail-based sampling strategies that preserve important traces - For critical workflows with span links, you may want to adjust sampling rates **Data not yet processed** There may be a delay between when trace data is sent and when span links appear: - Allow 1-2 minutes for data to be processed and indexed - Check that both linked traces have been successfully ingested - Verify that you're not hitting any data limits or rate limits For more information on implementing span links in your instrumentation, see our [OpenTelemetry best practices guide for traces](https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-best-practices-traces/#span-links). ## Span attributes [#span-attributes] If you'd like to learn more about `Span` data: - Explore your span data using the [query builder](https://docs.newrelic.com/docs/chart-builder/use-chart-builder/get-started/introduction-chart-builder). - To see the default attributes attached to span data, use the [data dictionary](/attribute-dictionary/?event=Span). - Check out these [example queries](https://docs.newrelic.com/docs/apm/distributed-tracing/ui-data/example-insights-queries-distributed-trace-data) for help with NRQL queries using these attributes. ## View related logs [#view-your-logs] If you are using our [logs in context](https://docs.newrelic.com/docs/logs/logs-context/configure-logs-context-apm-agents/) feature, you can see any logs that are linked to your traces: 1. Go to the trace details page by clicking on a trace. 2. In the upper-left corner of the span details page, click the **Logs** tab. 3. For details related to an individual log message, click directly on the message.