Trace API: Decorate spans with attributes

This document will explain how to add attributes to trace data sent to the Trace API so that spans display specific properties in the UI.

Why decorate your spans?

When you send data to our Trace API, you can add custom attributes to spans. For example, you might decide to add attributes like customer.id or user.id in order to help you analyze your trace data.

Some expected attribute values cause our distributed tracing UI to display some specific trace properties and details. For example, if a span has an attribute with an error. prefix, the UI displays that span with an error. For another example, a span with an attribute that has a db. prefix will be displayed as a datastore span in the UI, and will have its datastore query highlighted.

Decorating your spans to show specific properties in our UI can help you:

Better understand the structure of your trace data.
More easily troubleshoot problems.

How to decorate your spans with attributes

This table explains how to get spans sent to the Trace API to show up with specific properties in the UI. For property descriptions, see span properties.

Desired span property	UI indicator	How to add property
Errors		Use an attribute with an `error.` prefix. For example: `error.message`.
External		Use an attribute with an `http.` prefix. For example: `http.method`. A span will also appear as an external if it has a child that comes from a different entity. In order to be indexed on the External services page, either `http.url` or `net.peer.name` must be provided.
Datastore		Use an attribute with an `db.` prefix. For example: `db.statement`.
Service		This cannot be done with an attribute. A span is classified as a service span if it's the root span, or if its parent is from a different entity. If a span has several properties, service span classification takes precedence in the UI.
In-process		This cannot be done with an attribute. A span is classified as in-process if it hasn't been classified as a service span, datastore span, or external span.

For more on how these span properties are determined and stored, see Trace structure.

Tips for adding attributes:

You can add any attribute to a span. For example: you might add an attribute like customer.id so that you could search traces globally for traces containing a specific customer.
A span can be in multiple categories. For example, external is a more general category than is datastore, so if a span is classified as both external and datastore, it will be indicated as a datastore span in the UI.

JSON examples

Here are JSON examples showing how to use attributes to set span properties:

New Relic-format JSON with multiple types of attributes added. The significance of the custom attributes is described in customAttribute.

[
  {
    "common": {
      "attributes": {
        "hostname": "cattle456.example.com",
        "environment": "staging"
      }
    },
    "spans": [
      {
        "id": "1-abcdefg",
        "trace.id": "abc123-xyz789",
        "attributes": {
          "service.name": "Order Service",
          "duration.ms": 3.5,
          "error.name": "StackOverflowException",
          "name": "/placeOrder [POST]",
          "customer.id": "datanerd@newrelic.com",
          "description": "This span is the root of the whole trace. It has no parent.id. Custom attributes like 'customer.id' can have any name. Using these kinds of attributes will allow you to do a global search across all traces for desired traces."
        }
      },
      {
        "id": "2-abcdefg",
        "trace.id": "abc123-xyz789",
        "attributes": {
          "service.name": "Order Service",
          "duration.ms": 1,
          "parent.id": "1-abcdefg",
          "db.query": "foo selection",
          "db.statement": "SELECT FOO FROM BAR",
          "name": "DB Span",
          "description": "This is a datastore span. The presence of one or more attributes prefixed with db. makes this display as a datastore span in the UI."
        }
      },
      {
        "id": "3-abcdefg",
        "trace.id": "abc123-xyz789",
        "attributes": {
          "service.name": "Order Service",
          "parent.id": "1-abcdefg",
          "duration.ms": 1.5,
          "http.method": "POST",
          "name": "HTTP Span",
          "description": "An external (HTTP) span. Spans with one or more attributes prefixed with http. are treated as external spans."
        }
      },
      {
        "id": "4-abcdefg",
        "trace.id": "abc123-xyz789",
        "attributes": {
          "service.name": "Order Service",
          "duration.ms": 1.2,
          "error.text": "404 file not found",
          "parent.id": "1-abcdefg",
          "http.method": "GET",
          "http.statusCode": 404,
          "name": "Error Http Span",
          "description": "Spans with error. prefixed attributes are displayed in red text in the UI. Errors can coexist with other span categories."
        }
      },
      {
        "id": "5-abcdefg",
        "trace.id": "abc123-xyz789",
        "attributes": {
          "service.name": "Order Service",
          "error.message": "404 file not found",
          "duration.ms": 1.2,
          "parent.id": "1-abcdefg",
          "http.method": "GET",
          "http.statusCode": 404,
          "db.query": "SELECT FOO FROM BAR",
          "name": "Error Http DB Span",
          "description": "Spans can have multiple properties. Relevant attributes are highlighted when you select a span to view its details."
        }
      },
      {
        "id": "6-abcdefg",
        "trace.id": "abc123-xyz789",
        "attributes": {
          "service.name": "Order Service",
          "duration.ms": 1.6,
          "parent.id": "1-abcdefg",
          "http.method": "GET",
          "db.query": "SELECT FOO FROM BAR",
          "name": "Http DB Span",
          "description": "External (HTTP) is a more general category than is datastore, so a span with both http.- and db.-prefixed attributes is displayed as a datastore span in the UI."
        }
      },
      {
        "id": "7-abcdefg",
        "trace.id": "abc123-xyz789",
        "attributes": {
          "service.name": "Order Service",
          "duration.ms": 2.0,
          "parent.id": "1-abcdefg",
          "description": "Spans with no explicit types that belong to the same entity as its parent and children are considered in-process spans.",
          "name": "In-process span 1"
        }
      },
      {
        "id": "8-abcdefg",
        "trace.id": "abc123-xyz789",
        "attributes": {
          "service.name": "Order Service",
          "duration.ms": 1.7,
          "parent.id": "7-abcdefg",
          "name": "In-process span 2",
          "description": "In-process spans can represent a breakdown of work being done within a process."
        }
      },
      {
        "id": "9-abcdefg",
        "trace.id": "abc123-xyz789",
        "attributes": {
          "service.name": "Order Service",
          "duration.ms": 1.0,
          "parent.id": "8-abcdefg",
          "name": "In-process span 3",
          "description": "The number and granularity of in-process spans vary depending on instrumentation and frameworks being used."
        }
      },
      {
        "id": "10-abcdefg",
        "trace.id": "abc123-xyz789",
        "attributes": {
          "service.name": "Order Service",
          "duration.ms": 2.2,
          "parent.id": "1-abcdefg",
          "name": "In-process span"
        }
      },
      {
        "id": "11-abcdefg",
        "trace.id": "abc123-xyz789",
        "attributes": {
          "service.name": "Order Service",
          "duration.ms": 2.2,
          "parent.id": "10-abcdefg",
          "name": "External determined by entity change",
          "description": "A span that’s a parent to a span from another entity is displayed as an external span in the UI."
        }
      },
      {
        "id": "12-abcdefg",
        "trace.id": "abc123-xyz789",
        "attributes": {
          "service.name": "Order Notification Service",
          "duration.ms": 1.8,
          "parent.id": "11-abcdefg",
          "name": "Entry span determined by entity change",
          "description": "The attribute 'service.name' is used to detect process boundaries in the UI. For compatibility with data from Lambda monitoring and APM agents, the attribute 'entity.name' can be used to search across all traces."
        }
      }
    ]
  }
]

Zipkin-format JSON with multiple attribute types added. The significance of the attributes (key-value pairs) is described in customAttribute.

[
  {
    "traceId": "zipkinSampleTrace",
    "id": "1",
    "kind": "SERVER",
    "name": "Error Span",
    "duration": 35000,
    "localEndpoint": {
      "serviceName": "sampleApp",
      "ipv4": "127.0.0.1",
      "port": 8080
    },
    "tags": {
      "customAttribute": "This span is the root of the whole trace. It has no parent.id"
    }
  },
  {
    "traceId": "zipkinSampleTrace",
    "id": "2",
    "parentId": "1",
    "kind": "SERVER",
    "name": "post",
    "duration": 10000,
    "localEndpoint": {
      "serviceName": "sampleApp",
      "ipv4": "127.0.0.1",
      "port": 8080
    },
    "tags": {
      "customAttribute": "This is a datastore span. The presence of one or more attributes prefixed with db. makes this display as a datastore span in the UI."
    }
  },
  {
    "traceId": "zipkinSampleTrace",
    "id": "2",
    "parentId": "1",
    "kind": "CLIENT",
    "name": "DB Span",
    "duration": 12000,
    "localEndpoint": {
      "serviceName": "sampleApp",
      "ipv4": "127.0.0.1",
      "port": 8080
    },
    "tags": {
      "db.query": "foo selection",
      "db.statement": "SELECT FOO FROM BAR",
      "customAttribute": "This is a datastore span. The presence of one or more attributes prefixed with db. makes this display as a datastore span in the UI."
    }
  },
  {
    "traceId": "zipkinSampleTrace",
    "id": "3",
    "parentId": "1",
    "kind": "CLIENT",
    "name": "HTTP Span",
    "duration": 15000,
    "localEndpoint": {
      "serviceName": "sampleApp",
      "ipv4": "127.0.0.1",
      "port": 8080
    },
    "tags": {
      "http.method": "POST",
      "customAttribute": "AAn external (HTTP) span. Spans with one or more attributes prefixed with http. are treated as external spans."
    }
  },
  {
    "traceId": "zipkinSampleTrace",
    "id": "4",
    "parentId": "1",
    "kind": "CLIENT",
    "name": "Error Span",
    "duration": 12000,
    "localEndpoint": {
      "serviceName": "sampleApp",
      "ipv4": "127.0.0.1",
      "port": 8080
    },
    "tags": {
      "error.message": "404 file not found",
      "http.method": "GET",
      "http.statusCode": 404,
      "customAttribute": "Spans with error. prefixed attributes are displayed in red text in the UI. Errors can coexist with other span categories."
    }
  },
  {
    "traceId": "zipkinSampleTrace",
    "id": "5",
    "parentId": "1",
    "kind": "CLIENT",
    "name": "HTTP Error DB Span",
    "duration": 12000,
    "localEndpoint": {
      "serviceName": "sampleApp",
      "ipv4": "127.0.0.1",
      "port": 8080
    },
    "tags": {
      "error.message": "404 file not found",
      "http.method": "GET",
      "http.statusCode": 404,
      "db.query": "SELECT FOO FROM BAR",
      "customAttribute": "Spans can have multiple properties. Relevant attributes are highlighted when you select a span to view its details."
    }
  },
  {
    "traceId": "zipkinSampleTrace",
    "id": "6",
    "parentId": "1",
    "kind": "CLIENT",
    "name": "HTTP DB Span",
    "duration": 12000,
    "localEndpoint": {
      "serviceName": "sampleApp",
      "ipv4": "127.0.0.1",
      "port": 8080
    },
    "tags": {
      "http.method": "GET",
      "db.query": "SELECT FOO FROM BAR",
      "customAttribute": "External (HTTP) is a more general category than is datastore, so a span with both http.- and db.-prefixed attributes is displayed as a datastore span in the UI."
    }
  },
  {
    "traceId": "zipkinSampleTrace",
    "id": "7",
    "parentId": "1",
    "kind": "SERVER",
    "name": "In process span 1",
    "duration": 12000,
    "localEndpoint": {
      "serviceName": "sampleApp",
      "ipv4": "127.0.0.1",
      "port": 8080
    },
    "tags": {
      "customAttribute": "Spans with no explicit types that belong to the same entity as its parent and children are considered in-process spans."
    }
  },
  {
    "traceId": "zipkinSampleTrace",
    "id": "8",
    "parentId": "7",
    "kind": "SERVER",
    "name": "In process span 2",
    "duration": 12000,
    "localEndpoint": {
      "serviceName": "sampleApp",
      "ipv4": "127.0.0.1",
      "port": 8080
    },
    "tags": {
      "customAttribute": "In-process spans can represent a breakdown of work being done within a process."
    }
  },
  {
    "traceId": "zipkinSampleTrace",
    "id": "9",
    "parentId": "8",
    "kind": "SERVER",
    "name": "In process span 2",
    "duration": 12000,
    "localEndpoint": {
      "serviceName": "sampleApp",
      "ipv4": "127.0.0.1",
      "port": 8080
    },
    "tags": {
      "customAttribute": "The number and granularity of in-process spans vary depending on instrumentation and frameworks being used."
    }
  },
  {
    "traceId": "zipkinSampleTrace",
    "id": "10",
    "parentId": "1",
    "kind": "CLIENT",
    "name": "In process remote parent",
    "duration": 12000,
    "localEndpoint": {
      "serviceName": "sampleApp",
      "ipv4": "127.0.0.1",
      "port": 8080
    },
    "tags": {
      "name": "in process remote parent"
    }
  },
  {
    "traceId": "zipkinSampleTrace",
    "id": "10",
    "parentId": "1",
    "kind": "CLIENT",
    "name": "In process remote parent",
    "duration": 12000,
    "localEndpoint": {
      "serviceName": "sampleApp",
      "ipv4": "127.0.0.1",
      "port": 8080
    },
    "tags": {
      "customAttribute": "A span that is a parent to a span from another entity will be displayed as an external span."
    }
  },
  {
    "traceId": "zipkinSampleTrace",
    "id": "11",
    "parentId": "10",
    "kind": "SERVER",
    "name": "Downstream entry span",
    "duration": 12000,
    "localEndpoint": {
      "serviceName": "downstreamSampleApp",
      "ipv4": "127.0.0.1",
      "port": 8080
    },
    "tags": {
      "customAttribute": "The attribute 'service.name' is used to detect process boundaries in the UI. For compatibility with data from Lambda monitoring and APM agents, the attribute 'entity.name' can be used to search across all traces."
    }
  }
]

Trace API: Decorate spans with attributes

Why decorate your spans?

How to decorate your spans with attributes

JSON examples

New Relic-format attribute examples

Zipkin-format attribute examples

Trace API: Decorate spans with attributes

Why decorate your spans? .css-21sua1{background:none;border:none;width:0;padding:0;}

How to decorate your spans with attributes

JSON examples

Zipkin-format attribute examples

Why decorate your spans?