Tags are key-value pairs, for example team: operations, added to your monitored entities, or to your dashboards or workloads. We make some important attributes available as tags: this includes app metadata, like the app name and the language, and host metadata like the host name and the cloud provider region. You can also add your own custom tags.
You can use tags in the UI to filter down to relevant entities. Here's an example:
Here you can see an accountId tag being used to filter workloads to a specific account.
The benefits of tags
Tags help you:
Organize data coming from a large number of sources and/or to a large number of New Relic accounts.
Identify teams, roles, environments, or regions to know who's responsible for what. See some examples.
Tags are useful for organizing your data at a high level. If you want to add more fine-grained detail, like capturing user names or other high-cardinality values, we recommend using custom attributes or custom events.
Automatically reported tags
Tags are automatically applied to your monitored entities in some cases, from the following sources:
New Relic account metadata
Agent configuration
Host/environment
OpenTelemetry resource attributes
Tags from these sources can be removed or changed only by modifying the source of the tag, not via the UI or API.
You can also programmatically automate tags using our API or the New Relic CLI.
We automatically apply certain tags to entities that include account information. These tags include:
account
accountId
New Relic agents can be configured to apply tags to the entities created from their data. Configuration options are somewhat agent-specific, but generally use the labels key. Some examples:
Tags applied from agent configuration can result in multiple values for a given tag key on a given entity, because each deployed instance of the agent could be configured to send a different value.
New Relic agents automatically apply certain tags to service entities based on the environment. These include (but are not limited to):
For service/application entities
language: (for example, java or python)
For host entities
awsRegion, when applicable
If you're using one of our cloud provider integrations (AWS, Azure, or Google Cloud Platform), we recommend installing the infrastructure agent on your hosts: this will add cloud environment tags to your monitored host entities in New Relic.
Certain OpenTelemetry resource attributes are automatically applied as entity tags. When the following attributes are present on span data associated with a service entity, they will be attached to that entity:
k8s.cluster.name
k8s.deployment.name
k8s.namespace.name
telemetry.sdk.language: Will be stored as a tag named language
telemetry.sdk.version
service.namespace
This automated tagging is done using New Relic entity synthesis, which is documented on GitHub. In the case of OpenTelemetry services, the tagging definitions are on ext-service entities.
Manage tags
Some important attributes are available by default to be used as tags. For example, the account name, the account ID, and some basic host and server metadata. You can't remove these available-by-default tags.
When you add tags via the UI or API, this occurs at the entity level. This means that, for example, if you have multiple APM agents monitoring a single entity and apply a tag via the UI, that tag is available for all data coming from that entity.
Add tags
Go to one.newrelic.com and select a monitored entity, like a monitored app or host.
Near the entity's name at the top of the page, click on the tags button.
In the Add a new tag... field of the tags widget, add a tag with this format: accountId: 123456789.
For APM agents, if you want to use those tags in NRQL queries, restart your service.
Remove tags
Go to one.newrelic.com and select a monitored entity, like a monitored app or host.
Near the entity's name at the top of the page, click on the tags button.
Click the next to the tag you want to remove.
For APM agents, if you want to use those tags in NRQL queries, restart your service.
Navigate to a condition.
Near the condition's name at the top of the form, click Manage tags.
In the menu that pops up, add or delete a tag.
You can also add tags to an alert condition as an entity via the NerdGraph API (utilize the condition's entity GUID from the 'manage tags' screen).
Click the Filter by ... field to see a dropdown of available attributes and tags or enter a tag you want to filter by, such as accountId: 123456789.
Best practices
Follow these best practices to get the most out of your tags.
The UI has limits on the total number of tags it can display per entity:
Limit of 100 for tags reported by default or added via agent/integration configuration.
Limit of 100 for custom tags added via UI or API.
The following are maximum character length for tags:
Key: 128 characters
Value: 256 characters
Allowed characters:
Characters must be UTF-8.
When using NerdGraph to add tags, a dash - in a tag key is interpreted as a minus symbol. If your tag key has a dash, use back ticks around it, like `key-name`.
Some tips on keeping tags simple:
Start by adding only tags you know you'll definitely use. Unused tags create noise and may add confusion.
Try to use short tags. Shorter tags are easier to parse, and also the UI may sometimes truncate longer tags. (See character limits.)
When possible, use keys and values that are human-readable (for example, region: EMEA is better than Param8323 : 1229072).
Avoid including several values like regions: EMEA | US | LATAM in a single tag. We recommend using three different tags for that, like region: emea, region: us, and region: latam. If you want to match multiple tags, you can do that using the advanced options in the filter UI.
A caveat for alert incidents: these only support one key name instance. If you do use multiple key names, one will be randomly selected to be added to that incident.
Try to be consistent with tag language across your teams and entities.
Be consistent with naming. For example, avoid using both region: emea and reg: emea.
Be consistent with format. For example, avoid using both camelCase and kebab-case.
Although tag searching is not case-sensitive in the UI and API, try to be consistent with capitalization. For example, avoid using both env: staging and env: Staging.
Tags help improve observability and cost allocation. For this reason, responsibility for tag implementation is often assigned to an observability team, SREs, a group of architects, or a cross-team task force.
We recommend the people responsible for implementing tags meet and create an internal policy to describe how the tags are defined and which conventions are to be used. Then:
Keep this reference manual up to date.
Automate the definition of tags when you deploy New Relic agents, at the cloud provider, or through New Relic automation tools, such as the API or Terraform.
Create recurring reports that identify entities that are non-compliant with your tagging standards.
Tag examples
Here are some examples of common ways to use tags to organize data.
Creating tags for team names can be helpful to understand which team, group, department, or region was responsible for a change that led to a performance issue.
### Team tags
team: backend
team: frontend
team: db
### Role tags
roles: architecture
roles: devops
roles: pm
### Region tags
region: emea
region: america
region: asia
You can create entities for the environment they belong to. For example:
env: production
env: qa
env: development
env: staging
You can create tags related to criticality levels, which lets you easier track the most critical issues. For example:
layer: level1
layer: level2
layer: level3
Custom queries, dashboards, and alerts
Different features handle tags differently. Here are some details about how you can use NRQL to query tag data, or create NRQL condition alerts.
In the New Relic UI: On the entity's infrastructure page, create a tag and assign it a value. To query these tags, you need to add the tags. prefix to the name of the tag when you're quering for it in NRQL.
If you want to turn off infrastructure tag propagation, which means that you won't be able to query them with NRQL, see how to turn it off with Nerdgraph.
Query examples
Given a service that is distributed across nine shards, each shard assigned as an environment, we can plot the transaction throughput on all shards using a wildcard query like:
FROM Metric WHERE appName LIKE'MyApp (%'AND transactionType ='Other'
FACET tags.Environment TIMESERIES
To see the number of transactions per service for a specific team, we might use a query like:
FROMTransactionSELECTcount(*)
WHERE tags.Team ='team-a'
FACET tags.Project TIMESERIES
You can use a query for your services' error rate for setting an alert without having to create a rule for each service. Here is the error rate for all services on a team called team-b. And this alert will automatically monitor any new app names added with the team tag.
FROM Metric SELECTcount(apm.service.error.count)/count(apm.service.transaction.duration)
WHERE tags.Team ='team-b' FACET appName
Related to this, you might create a general rule for a specific service deployed across several environments, so that we have an alarm for a single service monitoring each environment individually:
From Metric SELECTcount(apm.service.error.count)/count(apm.service.transaction.duration)
WHERE tags.Project ='MyProject' FACET tags.Environment
To find internal teams that own entities named, for example, SensitiveEntity, run the following:
FROM SystemSample SELECT internalOwningTeam WHERE entityName ='SensitiveEntity'
Let's say you want to do some load testing on your different infrastructure entities with CPUs. You can apply one treatment to entities in a group called A, apply a different treatment to entities in a group called B, and keep one group of entities with no treatments, called control group.
You can create tags called experimentGroup in the New Relic UI, mapped to the experimental group that each entity belongs to. Since these are tags created in the New Relic UI, you need to add the tags. prefix to the tag name, unlike with custom attributes created in the newrelic-infra.yml configuration file.
Here's the NRQL query to select cpuPercent, broken down by the different experimental groups using a FACET CASES (...) clause:
FROM SystemSample SELECT cpuPercent FACET CASES (WHERE tags.experimentGroup ='control'AS'control group',WHERE tags.experimentGroup ='A'AS'given treatment A',WHERE tags.experimentGroup ='B'AS'given treatment B') SINCE 1DAY AGO
To turn off tag propagation for infrastructure tags, run the following in NerdGraph:
mutation{
dataManagementUpdateFeatureSettings(
accountId:YOUR_ACCOUNT_ID
setting:{
enabled:false
featureSetting:{key:"infra_tag_propagation"}
locked:false
}
){
enabled
key
locked
name
}
}
That call should return the following answer:
{
"data":{
"dataManagementUpdateFeatureSettings":{
"enabled":false,
"key":"infra_tag_propagation",
"locked":false,
"name":"Infra tag propagation"
}
}
}
Query and alert on attributes
Some important attributes are used as tags. You can query all attributes and create custom charts using NRQL, and create NRQL alert conditions.