Alert custom incident descriptions

For some types of conditions, you can define a custom incident description that passes useful information downstream for better incident responses or for use by other systems. You can create a custom incident description in the UI or with GraphQL.

Requirements

The custom incident description feature is available for:

• NRQL alert conditions
• Infrastructure monitoring alert conditions

The custom incident description field has a maximum character size of 4000.

What is a custom incident description?

For some types of conditions, you can create a custom incident description that passes information about a condition and its incidents downstream. Using a description, you can:

• Capture the reason for the condition
• Define the signal being monitored
• Define next steps
• Add metadata to downstream systems

The Custom incident description field allows you to use a general template structure that, when an incident is generated, substitutes the generic sections with specific values. For example, let's say you used this in a description:

This is my condition name : {{conditionName}}
The hostname is : {{tags.fullHostname}}

An incident resulting from this condition would fill in the details and you'd receive a notification like this:

This is my condition name : My Great Alert Condition
The hostname is : ip-123-45-67-89.us-west-1.compute.internal

Create a custom incident description

You can create a custom incident description using the dedicated field for NRQL alerts, or the Describe this condition section for infrastructure alerts.

Custom incident description template example

Here's a custom incident description template example:

this is my condition name : {{conditionName}}
The hostname is : {{tags.fullHostname}}
Owning Team: {{tags.label.owning_team}}
Product: {{tags.label.product}}
ec2VpcId: {{tags.aws.ec2VpcId}}
Service name : {{tags.label.Name}}
AWS Region : {{tags.aws.awsRegion}}
AWS Availability Zone: {{tags.aws.awsAvailabilityZone}}
Department : {{tags.label.department}}
Environment: {{tags.label.environment}}
Cluster: {{tags.clusterName}}
Cluster Role: {{tags.clusterRole}}
EC2 Instance Type: {{tags.instanceType}}
EC2 InstanceID: {{tags.aws.ec2InstanceId}}
EC2 AmiId: {{tags.aws.ec2AmiId}}
EC2 Root Device Type: {{tags.aws.ec2RootDeviceType}}

How to use attributes

The attributes you can use in a custom incident description are a subset of attributes attached to the incident event. Here are some details about the available attributes and how to use them:

• Incident event attributes: For any of the standard attributes available for use in a custom incident description, use the format {{AN_ATTRIBUTE}}. For example:

  The target name is : {{targetName}}

  Copy

• Tag attributes: For tags, you would use the format {{tags.TAG_NAME}}. There are two types of tags:

  • Entity-related tags: For infrastructure conditions only. You can find available entity-related tags by going to the entity explorer and looking under a service's Metadata and tags, or by viewing incident details. Here's an example of using tags in a custom incident description:

    The AWS region is : {{tags.aws.awsRegion}}
    Responsible team : {{tags.label.owning_team}}

    Copy

  • Facet clause tags: For NRQL conditions only. If a NRQL condition uses a FACET clause, you can use tags formatting to use those values in your custom incident description. For example, if the NRQL query included FACET hostName, cluster, you could then use this:

    The host is : {{tags.hostName}}
    The cluster is : {{tags.cluster}}

    Copy

Please note that tag names cannot include whitespace. The expanded values can include whitespace, but not the names of the tags themselves.

Create a custom incident description with GraphQL

You can create a custom incident description on a NRQL alert condition through a GraphQL mutation with that description.

Here's an empty mutation template:

mutation {
  alertsNrqlConditionStaticUpdate(accountId: YOUR_ACCOUNT_ID, id: "YOUR_CONDITION_ID", condition: {description: ""}) {
    YOUR_CUSTOM_INCIDENT_DESCRIPTION
  }
}

Here's an example mutation with a custom incident description:

mutation {
  alertsNrqlConditionStaticUpdate(accountId: 123456, id: "123456", condition: {description: "timestamp : {{timestamp}} \n accountId : {{accountId}} \n type : {{type}} \n event : {{event}} \n description : {{description}} \n policyId : {{policyId}} \n policyName: {{policyName}} \n conditionName : {{conditionName}} \n conditionId : {{conditionId}} \n product : {{product}} \n conditionType : {{conditionType}} \n RunbookUrl : {{runbookUrl}} \n nrqlQuery : {{nrqlQuery}} \n nrqlEventType : {{nrqlEventType}} \n targetID : {{targetId}} \n targetName : {{targetName}} \n commandLine : {{tags.commandLine}} \n entityGuid : {{tags.entityGuid}} \n entityName : {{tags.entityName}} \n fullHostname : {{tags.fullHostname}} \n instanceType : {{tags.instanceType}} \n processDisplayName : {{tags.processDisplayName}}"}) {
    description
  }
}

• For more information on NRQL conditions and using the NerdGraph API, go to NerdGraph API: NRQL condition alerts.
• For infrastructure monitoring conditions, you can create the custom incident description when creating an Infrastructure monitoring alert condition.
• To learn about how to structure a custom incident description, see the example template and the attribute/tag instructions.

Mentioning users or notifying channel in Slack

When creating a description that you know will be sent to Slack, you may wish to @mention a user or generate a channel-wide notification (for example, @here or @channel). To achieve this, simply include the User ID or a channel-wide notification in the description surrounded by < and > characters.

Here's an example for mentioning a user:

Attention <@LewCirne>

Here's an example for generating a channel-wide notification:

Attention <!channel>