Get started with Incident Intelligence

New Relic AI's Incident Intelligence provides you with an overview of all your incidents, their sources, and related events together.

diagram-applied-intelligence-workflow.png
Your sources supply the data to New Relic AI, which looks for potential anomolies for you to review.

Get started with Incident Intelligence

Enabling Incident Intelligence only requires three steps:

1. Configure sources

To get started, determine the sources for your incidents. A source refers to a data input to New Relic AI. Currently, available sources for New Relic AI are To get started, determine the sources for your incidents. A source refers to a data input to New Relic AI. Currently, you can get data from the the following available sources:

New Relic AI supports integrating your New Relic Alerts violations, so you can get enhanced context and correlation for the monitors you’ve already configured. Adding New Relic Alerts as a Source will not affect your current configuration or notifications.

To get data from Alerts:

  1. From the Sources page, click on New Relic Alerts. You’ll see a list of available New Relic Alerts policies, along with a search option to help you find specific policies.
  2. Select the policies you wish to connect to New Relic AI, and click Connect.
  3. Optional: If you want to add or remove policies from the New Relic Alerts Source, click on New Relic Alerts from the Sources page and click Add Policy, or hover over a connected policy and click Remove Policy.

New Relic AI supports integrating directly with your PagerDuty services, to consume incident data in real time from whatever tools you are currently using with PagerDuty. This can save your team hours in integrating individual tools, allowing New Relic AI to ingest, process, and enhance all of your PagerDuty incidents with only a few clicks.

To get data from PagerDuty:

  1. From the Sources page, click on PagerDuty.
  2. Enter your PagerDuty API key (which must have write access).
  3. Select the PagerDuty services you wish to connect to New Relic AI, and click Connect.
  4. Optional: If you want to add or remove services from the PagerDuty Source, click on PagerDuty again from the Sources page and click Add a Service, or hover over a connected policy and click Remove Service.

Connecting services to New Relic AI will not affect your current PagerDuty services or notifications.

New Relic AI supports integrating with Splunk log monitoring to consume infrastructure log data in real time for specific searches and key term reports, and to correlate alerts and search reports with your other metrics and incidents.

New Relic AI currently supports Splunk Light, Splunk Cloud, and Splunk Enterprise version 6.3 and above.

To get data from Splunk:

  1. In your Splunk console, start a search for the relevant events.
  2. Save your search as an alert, configure your alert conditions, and choose the webhook as the delivery method.
  3. From the Sources page in New Relic AI, click on Splunk. Copy the collector URL and paste it into the webhook endpoint back in the Splunk console.

New Relic AI supports Splunk metadata to enrich the alert data. By using the Splunk tokens, it is possible to leverage any search data, including search metadata and values from the first row of search results.

To access search metadata, use the format $<fieldname>$ - for example, $app$ for the App context for the search.

To access field values from the first result row that a search returns, use the format $result.<fieldname>$ - for example, $result.host$ for the host value and $result.sourcetype$ for the source type.

You can leverage any of the Selected Fields in the Splunk search and add any interesting fields to the Selected Fields to make the data available as a variable. The following fields will automatically provide hints to the New Relic AI correlation engine:

  • app: will be automatically parsed as APPLICATION_NAME
  • application: will be automatically parsed as APPLICATION_NAME
  • application_name: will be automatically parsed as APPLICATION_NAME
  • cluster: will be automatically parsed as CLUSTER_NAME
  • computer: will be automatically parsed as HOST_NAME
  • Dc: will be automatically parsed as DATACENTER_NAME
  • datacenter: will be automatically parsed as DATACENTER_NAME
  • host: will be automatically parsed as HOST_NAME
  • host_name: will be automatically parsed as HOST_NAME
  • hostname: will be automatically parsed as HOST_NAME
  • transaction: will be automatically parsed as EVENT_ID
  • Transaction_id: will be automatically parsed as EVENT_ID
  • user: will be automatically parsed as USER_NAME

Alerting rules in Prometheus send alerts to an Alertmanager, which distributes them accordingly. New Relic AI supports integrating with Prometheus Alertmanager to receive and correlate your Prometheus alerts with events from other sources.

To integrate with Prometheus Alertmanager:

  1. Set up your Alertmanager config file by running:
    ./alertmanager -config.file=simple.yml
  2. From the Sources page in New Relic AI, click on Prometheus Alertmanager and copy the URL. Paste this URL in the In the <webhook_config>/url section of the Alertmanager config file.

  3. Reload the Prometheus Alertmanager configuration by sending a SIGHUP to the process or by sending an HTTP POST request to the /-/reload endpoint.

New Relic AI integrates with Grafana’s alert notifications to provide fast correlation and root cause detection for events happening across your applications and infrastructure.

To integrate Grafana with New Relic AI:

  1. Log into your Grafana portal using Admin permissions and choose the Alerting option.
  2. On the Grafana Notification Channels page, click the New Channel button to configure a new Notification Channel.
  3. Choose webhook. The webhook notification is a simple way to send information about a state change over HTTP to a custom endpoint.
  4. From the Sources page in New Relic AI, choose Grafana and copy the URL. Paste this URL inside your new Grafana webhook.

New Relic AI integrates with Amazon CloudWatch to provide smarter incident management for all of your AWS services.

To integrate Amazon CloudWatch:

  1. From the Sources page in New Relic AI, choose AWS and copy the URL.
  2. Create a new Amazon SNS topic.
  3. Set CloudWatch to forward all Alarms state changes to that topic:
    • In the Amazon CloudWatch UI, click “Events” -> “Event Pattern”
    • In the “Service Name” drop down menu, select “CloudWatch”
    • In the “Event Type” drop down menu, select “CloudWatch Alarm State Change”
    • Under “Targets,” select “SNS Topic” and choose the one you created in step 2
  4. Create a new subscription:
    • In the Amazon AWS UI, click “Create a Subscription”
    • Select the SNS topic you created in step 2
    • Under “Protocol,” choose HTTPS
    • Under “Endpoint,” paste the URL from step 1

New Relic AI supports a dedicated REST API interface so you can easily integrate with additional systems, allowing instrumentation of your code or other monitoring solutions to report any kind of metric or event. A metric can be a raw data point such as CPU, memory, disk utilization, or business KPI. An event can be a monitoring alert, deployment event, incident, exceptions or any other change in state that you want to describe.

New Relic AI is totally agnostic to the event type, so you can send any type of data straight from your own systems or applications. The REST API supports secure token-based authentication and accepts JSON content as input.

For information on authentication and the full API reference, see REST API for New Relic AI.

2. Configure destinations

Destinations are the data outputs to New Relic AI, where you can view your automatically correlated incidents. Currently available destinations for New Relic AI include PagerDuty and webhooks.

Send data to PagerDuty

To configure the PagerDuty destination:

  1. From the Destinations page, click on PagerDuty. Enter your PagerDuty API key.

  2. Select the PagerDuty services you wish to connect to New Relic AI and click Connect.

We suggest you create a new PagerDuty service to use as the destination for New Relic AI.

Since PagerDuty services can also act as a source for New Relic AI, creating one or more new services for the correlated, streamlined incident output by New Relic AI can help keep the differentiation between your original and correlated/enhanced incidents clear.

When your team is ready, you can simply add on-call escalation policies for the PagerDuty destination service(s) you’ve configured in New Relic AI (or transfer the policies over from your existing services, leaving those services as NR AI sources if needed).

Review and edit the permissions granted to New Relic AI for the PagerDuty service(s) you chose. When you’re ready, click Authorize, and you’re ready to go!

If you want to add or remove services from the PagerDuty destination, click on PagerDuty again from the destinations page and click Add a Service, or hover over a connected policy and click Delete Service. You can also click Edit preferences to change the permissions granted to New Relic AI for any service.

The preferences/configuration settings are:

Configuration setting Description
Trigger new incidents Required. New Relic AI will trigger correlated parent incidents, so you can identify issues faster.
Edit incident titles Required. New Relic AI will automatically alter incident titles to help you orient and understand issues.
Add new integrations Required. New Relic AI will automatically add integrations to enable incident creation for selected services.
Add webhook extensions to sync user actions New Relic AI will automatically add webhook extensions to sync user actions in PagerDuty to New Relic, in real-time, and update correlated issue state.
Auto resolve correlated incidents when parent incident is resolved When enabled, New Relic AI will automatically close correlated children incidents when the parent incident is resolved.
Select user which will be used to take the following actions in PagerDuty on behalf of New Relic AI

New Relic AI requires a user to be able to enable the deep integration with PagerDuty, and provide additional context. You have full control of what actions we can take for you.

  • Automatically add notes to incidents (required): PagerDuty incident notes are used by New Relic AI to enrich incidents with additional context.

  • Automatically acknowledge triggered incidents: When enabled, New Relic AI will acknowledge newly triggered incidents in PagerDuty to allow them to be correlated before you're being paged.

  • Use the original escalation policy: When enabled, New Relic AI will apply the escalation policy of the original source service to each incident.

Send data via Webhook

New Relic AI will send the event body, in JSON format, via HTTPS POST. The system expects the endpoint to return a successful HTTP code (2xx).

To configure New Relic AI to send data via webhook:

  1. From the Destinations page, click on Webhook.
  2. Configure the webhook key. This is mandatory and unique, used in New Relic AI to refer to this webhook configuration and its specific settings.
  3. Configure the destination endpoint. This is mandatory - the endpoint where the webhook payload will be sent.
  4. Configure custom headers. These are optional key:value pairs of headers to be sent with the request. Example: “Authentication” “Bearer <bearer token>”
  5. Configure a custom payload. This is optional; this template can be leveraged to map New Relic fields to match the destination tool’s expected name and format.

    Both template examples are formatted in Jinja2.

    Suggested VictorOps template example
     
    {
    	"monitoring_tool": {{ sources }},
    	{% if state == 'closed' %}
    		"message_type":  "OK",
    	{% else %}
    		"message_type":  {{ priority }},
    	{% endif %}
    ​
    	"custom_fields": {
    		"issue_url": {{ url }},
    		"description": {{ description }},
    		"state": {{ state }},
    		"is_correlated": {{ is_correlated }},
    		"created_on": {{ created_on }},
    		"modified_on": {{ modified_on }},
    		"activated_on": {{ active_since }},
    		"closed_on": {% if closed_on is defined %} {{ closed_on }} {% else %} None {% endif %},
    		"related_incidents": [
    			{% for incident in incidents %}
    			{
    				"id": {{ incident.id }},
    				"events_count": {{ incident.events_count }},
    				"labels": {{ incident.labels }},
    				"title": {{ incident.title }},
    				"description": {{ incident.description }},
    				"state": {{ incident.state }},
    				"sources": {{ incident.sources }},
    				"modified_on": {{ incident.modified_on }},
    				"opened_on": {{ incident.opened_on }},
    				"closed_on": {{ incident.closed_on }}
    			}
    			{% if not loop.last %},{% endif %}
    			{% endfor %}
    		]
    	},
    	"state_message": {{ description }},
    	"entity_id": {{ id }},
    	"entity_display_name": {{ ui_name }},
    	"vo_annotate.u.NRAI_Link": {{ url }}
    }
    
    Suggested OpsGenie template example
    {
    	"alias":  {{ id }},
    	"message": {{ ui_name }},
    	"source": '{{ sources }}',
    	"priority": {{ priority }},
    	"details": {
    		"self_url": {{ url }},
    		"state": {{ state }},
    		"is_correlated": {{ is_correlated }},
    		"created_on": {{ created_on }},
    		"modified_on": {{ modified_on }},
    		"activated_on": {{ active_since }},
    		"closed_on": {% if closed_on is defined %} {{ closed_on }} {% else %} None {% endif %},
    		"is_correlated": {{ is_correlated }}
    	},
    	"description": 'Incidents [
    		{% for incident in incidents %}
    		{
    			"id": {{ incident.id }},
    			"events_count": {{ incident.events_count }},
    			"labels": {{ incident.labels }},
    			"title": {{ incident.title }},
    			"description": {{ incident.description }},
    			"state": {{ incident.state }},
    			"sources": {{ incident.sources }},
    			"modified_on": {{ incident.modified_on }},
    			"opened_on": {{ incident.opened_on }},
    			"closed_on": {{ incident.closed_on }}
    		}
    		{% if not loop.last %},{% endif %}{% endfor %}
    	]'
    }
    
    	      	
  6. Configure priority mapping. This optional section allows you to map New Relic’s priorities to the priorities expected at the destination.

    New Relic’s priorities are Critical, High, Medium, and Low.

New Relic AI will send the event body, in JSON format, via HTTPS POST. The system expects the endpoint to return a successful HTTP code (2xx). If you're configuring via webhook, this is the webhook body format:

Webhook format and JSON example
Name Type Description

url

Href

A link to the UI where the issue can be seen

id

String

Unique identifier for the issue

title

String

description

String

priority

Enum (Critical, High, Medium, Low)

state

Enum (Active, Closed, Acknowledged)

is_correlated

Boolean

True if the issue is correlated

created_on

String (ISO format)

modified_on

String (ISO format)

activated_on

String (ISO format)

closed_on

String (ISO format)

sources

List[String]

List of the different sources that were used to send the events into New Relic AI (for example: PagerDuty)

pathways

List[Pathways]

List of pathways that are associated with the issue. Each pathway contains an id and name: [{“id”: “pathway id”, “name”: “pathway name”}]

pathways[].id

String

pathways[].name

String

incidents

List[Incident]

List of incidents that are attached to the issue. The list will contain only the latest 100 incidents.

incidents[].id

String

incidents[].events_count

Integer

The number of events used to create the incident

incidents[].title

String

incidents[].description

String

incidents[].labels

Dictionary[String, String]

A string to string mapping of the incident labels. Labels represent the unique entities that are used to describe the incident.

incidents[].priority

Enum (Critical, High, Medium, Low)

incidents[].sources

List[String]

incidents[].state

Enum(open, closed)

incidents[].opened_on

String (ISO format)

incidents[].closed_on

String (ISO format)

incidents[].modified_on

String (ISO format)

JSON schema example

New Relic AI will send the event body, in JSON format, via HTTPS POST. The system expects the endpoint to return a successful HTTP code (2xx).

JSON schema:

 
{
 "$schema": "http://json-schema.org/draft-04/schema#",
 "definitions": {
   "Labels": {
     "type": "object",
     "properties": {},
     "additionalProperties": {
       "type": "string",
       "maxLength": 10000
     }
   },
   "LabelsSet": {
     "type": "array",
     "properties": {},
     "additionalProperties": {
       "type": "array",
       "items": {
         "$ref": "#/definitions/Labels"
       },
       "uniqueItems": true
     },
   },
   "Annotations": {
     "type": "object",
     "title": "Additional contextual information about the Entity",
     "description": "Fields like title, description and priority can be described as annotations since\nthey represent some additional information about the Incident.\nFor example, the rule that creates the entity can have some logic that will create a title annotation field.\nThe title will change according to the latest event received by the entity and as more\nevent will keep coming the field may change, all this as a decision by the rule.\nA different rule may decide not to populate the title at all.\nSince a title can also be produced from the labels themselves.",
     "properties": {
       "title": {
         "type": "string",
         "maxLength": 10000
       },
       "description": {
         "type": "string",
         "maxLength": 10000
       }
     },
     "patternProperties": {
       ".*": {
         "type": "string",
         "maxLength": 10000
       }
     },
     "additionalProperties": {
       "type": "string",
       "maxLength": 10000
     }
   },
   "Accumulations": {
     "type": "object",
     "title": "Accumulated data about the Incident",
     "description": "For example an entity with the label: host_name: foo will include a accumulated list of the\nrelevant process names that are relevant to the entity. Accumulations will be derived from the events",
     "properties": {},
     "additionalProperties": {
       "type": "array",
       "items": {
         "type": "string",
         "maxLength": 10000
       }
     }
   },
   "IssueState": {
     "type": "string",
     "enum": [
       "created",
       "active",
       "acknowledged",
       "closed"
     ],
     "properties": {},
     "additionalProperties": false
   },
   "Priority": {
     "type": "string",
     "enum": [
       "critical",
       "high",
       "medium",
       "low"
     ],
     "properties": {},
     "additionalProperties": false
   },
   "IssueClosingReason": {
     "type": "string",
     "enum": [
       "nacked",
       "resolved",
       "system"
     ],
     "properties": {},
     "additionalProperties": false
   },
   "IssueAction": {
     "type": "string",
     "enum": [
       "ack",
       "snooze",
       "nack",
       "end_snooze",
       "expire_snooze",
       "star",
       "un_snooze",
       "unstar",
       "archive",
       "unarchive",
       "resolve",
       "reopen",
       "open",
       "feedback",
       "feedback_file",
       "open_ticket",
       "merge_issues",
       "active",
       "create",
       "close",
       "incident_updated",
       "confidence_changed",
       "diagnoses_changed",
       "rule_properties_changed",
       "flapping",
       "notification",
       "incident_events_limit_reached",
       "follow",
       "unfollow",
       "priority_updated",
       "comment",
       "incident_appended",
       "diagnosis_feedback",
       "flapping_delay_close"
     ],
     "properties": {},
     "additionalProperties": false
   },
   "StringKeyValue": {
     "type": "object",
     "properties": {},
     "additionalProperties": {
       "type": "string"
     }
   },
   "JavaOptionalUuid": {
     "type": "string",
     "properties": {},
     "additionalProperties": false,
     "format": "string-uuid"
   }
 },
 "type": "object",
 "properties": {
   "id": {
     "type": "string",
     "format": "string-uuid"
   },
   "customer_id": {
     "type": "integer"
   },
   "labels_set": {
     "$ref": "#/definitions/LabelsSet"
   },
   "annotations": {
     "$ref": "#/definitions/Annotations"
   },
   "user_annotations": {
     "$ref": "#/definitions/Annotations"
   },
   "user_annotations_version": {
     "type": "integer"
   },
   "accumulations": {
     "$ref": "#/definitions/Accumulations"
   },
   "state": {
     "$ref": "#/definitions/IssueState"
   },
   "previous_state": {
     "$ref": "#/definitions/IssueState"
   },
   "priority": {
     "$ref": "#/definitions/Priority"
   },
   "closing_reason": {
     "oneOf": [
       {
         "$ref": "#/definitions/IssueClosingReason"
       },
       {
         "type": "null"
       }
     ]
   },
   "modified_on": {
     "type": "integer"
   },
   "created_on": {
     "type": "integer"
   },
   "closed_on": {
     "oneOf": [
       {
         "type": "integer"
       },
       {
         "type": "null"
       }
     ]
   },
   "acknowledged_on": {
     "oneOf": [
       {
         "type": "integer"
       },
       {
         "type": "null"
       }
     ]
   },
   "active_since": {
     "oneOf": [
       {
         "type": "integer"
       },
       {
         "type": "null"
       }
     ]
   },
   "assigned_to": {
     "oneOf": [
       {
         "type": "array",
         "items": {
           "type": "string",
           "minLength": 1
         },
         "uniqueItems": true
       },
       {
         "type": "null"
       }
     ]
   },
   "created_by": {
     "oneOf": [
       {
         "type": "array",
         "items": {
           "type": "string",
           "minLength": 1
         },
         "uniqueItems": true
       },
       {
         "type": "null"
       }
     ]
   },
   "progress_report": {
     "type": "array",
     "default": [],
     "items": {
       "type": "object",
       "properties": {
         "origin_id": {
           "oneOf": [
             {
               "type": "string",
               "minLength": 1
             },
             {
               "type": "null"
             }
           ]
         },
         "timestamp": {
           "type": "integer"
         },
         "action_timestamp": {
           "oneOf": [
             {
               "type": "integer"
             },
             {
               "type": "null"
             }
           ]
         },
         "origin_type": {
           "type": "string",
           "enum": [
             "user",
             "system"
           ]
         },
         "event_type": {
           "$ref": "#/definitions/IssueAction"
         },
         "issue_id": {
           "oneOf": [
             {
               "type": "string",
               "minLength": 1,
               "format": "string-uuid"
             },
             {
               "type": "null"
             }
           ]
         },
         "attributes": {
           "oneOf": [
             {
               "$ref": "#/definitions/StringKeyValue"
             },
             {
               "type": "null"
             }
           ]
         },
         "team_id": {
           "$ref": "#/definitions/JavaOptionalUuid"
         }
       },
       "required": [
         "timestamp",
         "origin_type",
         "event_type"
       ],
       "additionalProperties": false
     },
     "uniqueItems": true
   },
   "rule_id": {
     "oneOf": [
       {
         "type": "string",
         "minLength": 1,
         "maxLength": 255
       },
       {
         "type": "null"
       }
     ]
   },
   "incident_ids": {
     "oneOf": [
       {
         "type": "array",
         "items": {
           "type": "string",
           "format": "string-uuid"
         },
         "uniqueItems": true
       },
       {
         "type": "null"
       }
     ]
   },
   "confidence": {
     "type": "number",
     "default": 1.0,
     "minimum": 0.0,
     "maximum": 1.0
   }
 },
 "required": [
   "id",
   "customer_id",
   "labels_set",
   "annotations",
   "priority",
   "modified_on",
   "created_on",
   "confidence"
 ],
 "additionalProperties": true
}
		

3. Configure pathways

Pathways tell New Relic AI when and where you want to receive notifications, giving you control over the specific logic used to direct alerts to the correct team members.

From the Pathways page, click in the query builder box to start creating a pathway.

  1. Specify or select an attribute, such as application/name. You can choose from a list that includes all the attributes available in PagerDuty incidents and New Relic Alert violations, or add your own attributes.

  2. Choose a logical operator—for example, contains.

  3. Enter a specific value to complete the logical expression.

You can also use the AND/OR operators to build more complex logic, or just choose Send everything, which will include all issues created by the sources you configured earlier (this is a good option if your team uses only one PagerDuty service to manage all incidents).

Finally, choose one or more of the destinations you configured.

You can add or configure pathways from the Pathways page (hover over an existing pathway to see options).

Getting started with decisions

Use AI decisions to further our correlation logic. Recommendations are automatically provided to you, and you can create decision logic to provide feedback and train our AI system over time.

To start working with decisions, click on Decisions from the New Relic AI UI. You’ll see a list of active decisions and some information about them:

You can click on any decision to see the detailed rule logic that creates correlations between your issues. Click the like/dislike buttons to give feedback about any decision. You can also enable, disable, edit, or delete decisions from this view.

Using the Decision builder

Once you’ve checked out decisions and are ready to create your own, click on Add a decision in the upper right corner of the screen.

  1. Tell us what data to compare: define up to two segments of incoming data (from any of the sources you’ve configured) to compare based on any attribute - choose from the drop down list or add your own. A broad range of comparison operators is supported, from basic “strict equals” to regular expression matching to many advanced similarity algorithms.

  2. Tell us what to correlate: now that you’ve chosen the data to be compared, add rules to define when those issues should be correlated (merged). You can correlate issues based on attributes (eg. “source equals aws”) or based on a similarity algorithm that assesses the entire incident.

  3. Give it a name: Once you’ve configured your decision logic, provide an easily-readable name and description that will be used to identify it in the future.

  4. Configure advanced settings (optional): You can also change some of the other settings for the decision:

    • Time window: Set the maximum time between incidents for them to be eligible for correlation with this decision.

    • Issue priority: The default setting is to “inherit priority” (whichever is higher), but you can override this to add higher or lower urgency to correlated issues.

    • Frequency: Change this setting to modify the minimum number of incidents that need to meet the rule logic in order for the decision to trigger.

    • Similarity: If you’re using similar to comparison operators in your rule logic, you can choose from a list of algorithms to use and set their sensitivity. The algorithm and threshold you choose will apply to all similar to operators in your rule logic. Hover over each algorithm name in the drop down menu to learn more about when it should be used.

When you’re done adding and fine-tuning logic, save the decision. All new decisions are enabled by default, so check back in awhile to see how many incidents were correlated with your rule logic.

With proactive detection, New Relic AI can deliver real-time failure warnings when an anomaly is detected with your app data. For more information, see Proactive detection.

EU and US datacenter and New Relic AI

Whether you store your data in New Relic’s U.S. region data center or our EU region data center, because New Relic AI’s services are performed by services operating solely in the U.S., by using New Relic AI, you agree that New Relic may move your data to, and process your data in, the U.S. region.

For more help

Recommendations for learning more: