Get started with Incident Intelligence

New Relic AI's Incident Intelligence provides you with an overview of all your incidents, their sources, and related events together.

Your sources supply the data to New Relic AI, which looks for potential anomalies for you to review.

Get started with Incident Intelligence

Enabling Incident Intelligence only requires three steps:

1. Configure sources

To get started, determine the sources for your incidents. A source refers to a data input to New Relic AI. You can get data from the following sources:

By integrating New Relic AI with your New Relic Alerts violations, you can get enhanced context and correlation for the monitors you've already configured. Adding New Relic Alerts as a source will not affect your current configuration or notifications.

To get data from Alerts:

  1. Go to > New Relic AI > Incident Intelligence > Sources > New Relic Alerts.
  2. Search or select the policies you want to connect to New Relic AI, and click Connect.

From the Sources > New Relic Alerts page, you can also add more Alerts policies or remove existing connected policies as a source.

To save your team hours in integrating individual tools that you already use with PagerDuty, you can integrate New Relic AI directly with your PagerDuty services. This allows New Relic AI to ingest, process, and enhance all of your PagerDuty incidents with a few clicks.

Connecting services to New Relic AI will not affect your current PagerDuty services or notifications.

To get data from PagerDuty:

  1. Make sure your PagerDuty API key has write access.
  2. Go to > New Relic AI > Incident Intelligence > Sources > PagerDuty.
  3. Enter your PagerDuty API key. This should be either a personal or general access API key with write access; if it's created by a user, the user should be an admin.
  4. Select the PagerDuty services you want to connect to New Relic AI, and click Connect.

From the Sources > PagerDuty page, you can also add more or remove existing connected services as a source.

New Relic AI supports Splunk Light, Splunk Cloud, and Splunk Enterprise version 6.3 and higher. By integrating with your Splunk log monitoring, New Relic AI:

  • Consumes your environment's log data in real time for specific searches and key term reports.
  • Correlates alerts and search reports with your other metrics and incidents.

To get data from Splunk:

  1. In your Splunk console, start a search for the relevant events.
  2. Save your search as an alert, configure your alert conditions, and choose the webhook as the delivery method.
  3. Go to > New Relic AI > Incident Intelligence > Sources > Splunk.
  4. Copy the collector URL, and paste it into the webhook endpoint in the Splunk console.
  5. Optional: Use Splunk tokens to enrich alert data with Splunk metadata.

To enrich alert data with your Splunk metadata, use Splunk tokens. You can leverage any search data, including search metadata and values from the first row of search results.

If you want to... Do this...
Access search data

Use the format $<fieldname>$. For example, use $app$ for the app context for the search.

Access field values

To access field values from the first result row that a search returns, use the format $result.<fieldname>$. For example, use $$ for the host value and $result.sourcetype$ for the source type.

Use variables

You can leverage any of the Selected fields in the Splunk search and add any interesting fields to the Selected fields to make the data available as a variable.

The following fields automatically provide hints to the New Relic AI correlation engine:

  • app: will be automatically parsed as APPLICATION_NAME
  • application: will be automatically parsed as APPLICATION_NAME
  • application_name: will be automatically parsed as APPLICATION_NAME
  • cluster: will be automatically parsed as CLUSTER_NAME
  • computer: will be automatically parsed as HOST_NAME
  • Dc: will be automatically parsed as DATACENTER_NAME
  • datacenter: will be automatically parsed as DATACENTER_NAME
  • host: will be automatically parsed as HOST_NAME
  • host_name: will be automatically parsed as HOST_NAME
  • hostname: will be automatically parsed as HOST_NAME
  • transaction: will be automatically parsed as EVENT_ID
  • Transaction_id: will be automatically parsed as EVENT_ID
  • user: will be automatically parsed as USER_NAME

Alerting rules in Prometheus send alerts to an Alertmanager, which distributes them accordingly. By integrating New Relic AI with Prometheus Alertmanager, you can receive and correlate your Prometheus alerts with events from other sources.

To integrate with Prometheus Alertmanager:

  1. Set up your Alertmanager config file by running:

    ./alertmanager -config.file=simple.yml
  2. Go to > New Relic AI > Incident Intelligence > Sources > Prometheus Alertmanager.
  3. Copy the Prometheus Alertmanager URL, and paste it in the <webhook_config>/url section of your Alertmanager config file.

  4. Reload the Prometheus Alertmanager configuration by sending a SIGHUP to the process or by sending an HTTP POST request to the /-/reload endpoint.

Use the New Relic AI integration with Grafana's alert notifications to provide fast correlation and root cause detection for events happening across your applications and environment. The webhook notification is a simple way to send information about a state change over HTTP to a custom endpoint.

To integrate Grafana with New Relic AI as a new webhook notification channel:

  1. Log into your Grafana portal using Admin permissions, and choose Alerting.
  2. On the Grafana Notification Channels page, click New Channel > Webhook.
  3. Go to > New Relic AI > Incident Intelligence > Sources > Grafana.
  4. Copy the URL, and paste it in your new Grafana webhook.

New Relic AI integrates with Amazon CloudWatch to provide smarter incident management for all of your AWS services.

To integrate Amazon CloudWatch:

  1. Go to > New Relic AI > Incident Intelligence > Sources > AWS.
  2. Copy the URL.
  3. Create a new Amazon SNS topic.
  4. Set CloudWatch to forward all Alarms state changes to that topic:

    • In the Amazon CloudWatch UI, click Events > Event Pattern.
    • Select Service Name > CloudWatch.
    • Select Event Type > CloudWatch Alarm State Change.
    • Select Targets > SNS Topic, and select your new Amazon SNS topic.
  5. Create a new subscription:

    • In the Amazon AWS UI, click Create a Subscription.
    • Select your new Amazon SNS topic.
    • Select Protocol > choose HTTPS.
    • In Endpoint, paste the URL you previously copied from the New Relic AI Sources.

New Relic AI supports a dedicated REST API interface so you can easily integrate with additional systems. This allows instrumentation of your code or other monitoring solutions to report any kind of metric or event.

  • A metric can be a raw data point such as CPU, memory, disk utilization, or business KPI.
  • An event can be a monitoring alert, deployment event, incident, exceptions or any other change in state that you want to describe.

New Relic AI is totally agnostic to the event type, so you can send any type of data straight from your own systems or applications. The REST API supports secure token-based authentication and accepts JSON content as input.

For more information on authentication and the full API reference, see REST API for New Relic AI.

2. Configure destinations

Destinations are the data outputs to New Relic AI, where you can view your automatically correlated incidents. Available destinations for New Relic AI include PagerDuty and webhooks. For examples of destination templates, webhook formats, and JSON schema, see the Incident Intelligence destination examples.

Send data to PagerDuty

Recommendation: Create a new PagerDuty service to use as the destination for New Relic AI. PagerDuty services can also act as a source for New Relic AI. Creating one or more new services for the correlated, streamlined incident output by New Relic AI can help keep the differentiation clear between your original and correlated, enhanced incidents.

To configure the PagerDuty destination:

  1. Go to > New Relic AI > Incident Intelligence > Destinations > PagerDuty.
  2. Enter your PagerDuty API key. This should be either a personal or general access API key with write access; if it's created by a user, the user should be an admin. This can be the same API key used for the PagerDuty source (if you configured this source).
  3. Select the PagerDuty services you want to connect to New Relic AI, and click Connect.

When your team is ready, you can simply add on-call escalation policies for one or more PagerDuty destination services configured in New Relic AI. You can also transfer the policies over from your existing services, and leave those services as New Relic AI sources if needed.

From the Destinations > PagerDuty page, you can also:

  • Review and edit the permissions granted to New Relic AI for the PagerDuty services you chose. When you're ready, click Authorize.
  • Add more or delete existing connected services from the PagerDuty destination.
  • Edit permissions granted to New Relic AI for any service.

The following are settings for preferences and other configuration settings. For examples of destination templates, see the Incident Intelligence destination examples.

Configuration setting Description
Trigger new incidents Required. New Relic AI will trigger correlated parent incidents, so you can identify issues faster.
Edit incident titles Required. New Relic AI will automatically alter incident titles to help you orient and understand issues.
Add new integrations Required. New Relic AI will automatically add integrations to enable incident creation for selected services.
Add webhook extensions to sync user actions New Relic AI will automatically add webhook extensions to sync user actions in PagerDuty to New Relic, in real-time, and update correlated issue state.
Auto-resolve correlated incidents when parent incident is resolved When enabled, New Relic AI will automatically close correlated children incidents when the parent incident is resolved.
Select user that takes actions in PagerDuty on behalf of New Relic AI

New Relic AI requires a user to be able to enable the deep integration with PagerDuty, and provide additional context. You have full control of what actions we can take for you. These include:

  • Automatically add notes to incidents (required): PagerDuty incident notes are used by New Relic AI to enrich incidents with additional context.

  • Automatically acknowledge triggered incidents: When enabled, New Relic AI will acknowledge newly triggered incidents in PagerDuty to allow them to be correlated before you're being paged.

  • Use the original escalation policy: When enabled, New Relic AI will apply the escalation policy of the original source service to each incident.

Send data via webhook

New Relic AI will send the event body in JSON format by HTTPS POST. The system expects the endpoint to return a successful HTTP code (2xx).

To configure New Relic AI to send data via webhook:

  1. Go to > New Relic AI > Incident Intelligence > Destinations > Webhook.
  2. Required: Configure the unique webhook key, used in New Relic AI to refer to this webhook configuration and its specific settings.
  3. Required: Configure the destination endpoint where the webhook payload will be sent.
  4. Optional: Configure custom headers, which are key:value pairs of headers to be sent with the request. Example: "Authentication" "Bearer" <bearer token>
  5. Optional: Configure a custom payload template that can be used to map New Relic fields to match the destination tool's expected name and format.
  6. Optional: Configure priority mapping (critical, high, medium, or low), used to map New Relic's priorities to the priorities expected at the destination.

For examples of webhook formats and JSON schema, see the Incident Intelligence destination examples.

3. Configure pathways

Pathways tell New Relic AI when and where you want to receive notifications. This gives you control over the specific logic used to direct alerts to the correct team members.

  1. Go to > New Relic AI > Incident Intelligence > Pathways, and select Add a pathway.
  2. In the query builder box, specify or select an attribute, such as application/name. Select from the list that includes all the attributes available in PagerDuty incidents and New Relic Alert violations, or add your own attributes.
  3. Select a logical operator; for example, contains.
  4. Enter a specific value to complete the logical expression.

    • To include all issues created by the sources you configured, select Send everything. (Recommended if your team uses only one PagerDuty service to manage all incidents.)
    • To build more complex logic, use the AND/OR operators.
  5. Select one or more of the destinations you configured.

To edit or remove existing pathways, mouse over the pathway's name on the Pathways page.

Using the Issue feed

After you configure sources, destinations, and pathways, Issues will begin to appear in the New Relic AI Issue feed. This gives you a quick overview of all of the Issues in your environment, along with some helpful information about them.

From left to right in the screenshot below, you'll see:

  • The last time the issue was updated - for example: was opened or closed, had new incidents added to it, was correlated with another issue. The issue table is sorted by last update by default.

  • An icon to indicate if the issue is correlated with other issues.

  • A human-readable title - this is created automatically by New Relic AI's natural language processing engine, and updated dynamically as the issue evolves.

  • The issue's state (created, active, acknowledged, or closed).

  • Its priority (determined by New Relic; this can also be dynamically updated with decision logic).

  • The path the issue has taken - data source(s) and destination(s).

  • The issue's duration - time from creation to now (if it's active) or creation to close.

  • The number of incidents included in an issue (includes incidents in the parent issue and, if applicable, incidents in correlated child issues).

This image shows the AI issue feed in New Relic One.

You can also click into any issue to see more details about it, including an analysis summary with an automatically generated symptom classification, a log of the major events that have taken place involving the issue, details about correlated issues, and links to related issues that occurred in the past.

What's the difference between an issue, incident, and event? The best way to think about these terms is as building blocks: events are raw data, transformed for consistency, from your sources; incidents are made up of one or more events; and issues are composed of one or more incidents.

In more detail:

  • Events usually indicate a state change or trigger in criteria defined by your monitoring systems (sources). An event will typically contain information about the affected entity and a description of what happened, and events are almost always triggered automatically by the monitoring system.

  • Incidents represent the observed state and evolution of a failing and recovery criteria in your monitoring system - in other words, groups of events that together describe what happened in the system over time. You can think of incidents as “symptoms” of possible problems in the monitored environment; these symptoms are detected by configurations in different monitoring tools which evaluate multiple types of data streams and events.

  • Issues are groups of incidents, and represent the underlying problem that's causing the symptoms observed in the incidents. When a new incident is created, New Relic AI opens an issue containing this incident and evaluates the other open issues to look for correlations. In addition to incident data and correlated issue information, issues also include a progress report, which details actions and events involving the issue over time, as well as other metadata like the issue's classification and related activity.

Get started with decisions

Use AI decisions to further the AI correlation logic. Recommendations are automatically provided to you, and you can create decision logic to provide feedback and train our AI system over time.

New Relic One: AI decisions
Here is an example of a New Relic AI decision that can be used to create correlations between issues.

To start working with decisions:

  1. Go to > New Relic AI > Incident Intelligence > Decisions.
  2. Review the list of active decisions and information about them.
  3. To see the detailed rule logic that creates correlations between your issues, click the decision.
  4. To give feedback about any decision, click the like [like icon] or dislike [dislike icon] icons.
  5. Use any of the other available options to enable, disable, edit, or delete decisions.

To create your own decision:

  1. Go to > New Relic AI > Incident Intelligence > Decisions, and click Add a decision.
  2. Use the following decision builder guidelines to build the AI decision.
  3. When you're done adding and fine-tuning logic, save the decision.

All new decisions are enabled by default, so check back later to see how many incidents were correlated with your rule logic.

Decision builder Guidelines
1. Tell us what data to compare

Define up to two segments of incoming data (from any of the sources you've configured) to compare based on any attribute. Choose from the dropdown, or add your own.

A broad range of comparison operators is supported, from basic “strict equals” to regular expression matching to many advanced similarity algorithms.

2. Tell us what to correlate.

Now that you've chosen the data to be compared, add rules to define when those issues should be correlated (merged). You can correlate issues based on attributes (for example, source equals aws) or on a similarity algorithm that assesses the entire incident.

3. Give it a name.

After you configure your decision logic, provide an easily-readable name and description that will be used to identify it later.

4. Optional: Configure advanced settings.

You can also change some of the other settings for the decision:

  • Time window: Set the maximum time between incidents for them to be eligible for correlation with this decision.

  • Issue priority: The default setting is to inherit priority (whichever is higher), but you can override this to add higher or lower urgency to correlated issues.

  • Frequency: Change this setting to modify the minimum number of incidents that need to meet the rule logic in order for the decision to trigger.

  • Similarity: If you're using similar to comparison operators in your rule logic, you can choose from a list of algorithms to use and set their sensitivity. The algorithm and threshold you choose will apply to all similar to operators in your rule logic. To learn more about when an algorithm should be used, mouse over its name.

For technical details of some of the correlation algorithms that are available to choose from in the New Relic AI Decision engine, see the follwoing options:

Levenshtein distance
Details Description
What it is

Useful for comparing short strings with static schema and fixed length, like host names.

One of the similarity measures used in the New Relic AI Decision engine is Levenshtein distance, otherwise known as edit distance. Common applications of Levenshtein distance include spell checkers, computational biology, and speech recognition.

The default similarity threshold for new New Relic AI Decisions is an edit distance of 3 - you can change this in the Advanced mode of the Decision builder.

How it works

The Levenshtein distance between two strings is the minimum number of single-character edits to get from one string to the other. Allowed edit operations are deletion, insertion, and substitution.


number/bumble: 3 (number → bumber → bumblr → bumble)

trying/lying: 2 (trying → rying → lying)

strong/through: 4 (strong → htrong → throng → throug → through)

When to use it

Levenshtein distance is most useful for comparing relatively short strings with static schema and fixed length, such as host names - for example, the Levenshtein distance between us-a1 and us-a2 is only 1, providing strong support for correlation.

Potential drawbacks

Levenshtein distance is not normalized by default to take into account string lengths. 'a' and 'q' have a Levenshtein distance of 1, whereas "the quick brown fox jumps over the lazy dog" and "the quick brown rat jumps over the lazy dog" have a Levenshtein distance of 3.

Fuzzy score
Details Description
What it is

Useful for same-length strings where the same prefix would be a good indicator of correlation. Between two strings, a high fuzzy score indicates high similarity.

How it works

The fuzzy score algorithm works by allocating "points" for character matches between strings:

  • One point for each matching character

  • Two bonus points for subsequent matches


Example: Decisions / dcsions

d: 1

c: 1

i 1

s: 2

o: 1

n: 1

si: 2

io: 2

on: 2

ns: 2

= 15 points

When to use it

Fuzzy score is most useful for strings that have the same, and relatively short (ideally <5 character), prefixes. A minimum guaranteed score would be (length(expected prefix) * 3) - 2.

Potential drawbacks

If the first character of the first string can't be found in the second string, no points are awarded. For example, the fuzzy score for us01 and cn01 is 0. For "ru01" and the second is "us01", the score is 1, since fuzzy score finds "u" at position 1, but cannot find an "s" afterwards.

Fuzzy wuzzy ratio

Useful for comparing strings of similar length.

Details Description
What it is

The fuzzy wuzzy family of similarity measures was developed by SeatGeek to help find tickets for the same event that have different labels across multiple platforms. The fuzzy wuzzy ratio for two strings is expressed as a percentage, where a higher number indicates a more similar string. It's based on the SequenceMatcher algorithm in Python's difflib.

When to use it

Fuzzy wuzzy ratio is effective for very short (eg. hostname) or very long (eg. event description) strings, especially in comparing strings of similar length.

Potential drawbacks

This algorithm is too sensitive to be used effectively for 3-10 word strings; one of the other modifications to fuzzy wuzzy (see below) may be a better choice.

Fuzzy Wuzzy partial ratio

Useful for comparing strings of different length.

Details Description
What it is

This modification to the fuzzy wuzzy algorithm helps address the effective length limitation described previously. With fuzzy wuzzy partial ratio, the shorter string is compared to each substring of the same length within the longer string. The score of the “best matching” substring is used to determine the fuzzy wuzzy partial ratio.


For example, between the following strings:

DevOps and SRE teams


…”DevOps” (the shorter string, length = 6) would be compared to each substring with length 6 within “DevOps and SRE teams.” Since one of those substrings (“DevOps”) is a perfect match, the fuzzy wuzzy partial ratio for these two strings will be high.

When to use it

Fuzzy wuzzy partial ratio is especially effective for the types of comparisons the basic fuzzy wuzzy algorithm fails at: 3-10 word strings where some significant substrings are likely to be overlapping.

Potential drawbacks

Where fuzzy wuzzy may be too conservative, fuzzy wuzzy partial match may be more liberal than expected with correlations - adjust the threshold in the Decision builder according to your needs.

Fuzzy wuzzy token set rato

Useful for comparing strings where the information may not be in the same order, and of possible different lengths. Works best for sentences such as messages, descriptions, etc.

Details Description
How it works

The token set ratio algorithm follows a few steps to compare strings:

  1. Tokenize each string (eg. “DevOps and SRE teams” → “DevOps” “and” “SRE” “teams”; “SRE team and DevOps engineers” → “SRE” “teams” “and” “DevOps” “engineers”)

  2. Combine intersecting tokens into a new string, leaving the remaining tokens. (eg. intersecting: ”DevOps”, “and”, “SRE”; remainder1: “teams”; remainder2: “team”, “engineers”)

  3. Alphabetize each token group (eg. “and, DevOps, SRE”, “teams”, engineers, team”)

  4. Compare the following pairs of strings:

    1. Intersection group

    2. Intersection group + remainder1

    3. Intersection group + remainder2

The comparison from these pairs (“best matches”) is the fuzzy wuzzy token set ratio.

When to use it

This metric is helpful in cases where similar strings may have overlapping words but different construction - for example, event descriptions for different issues with the same resource.

Potential drawbacks
Jaro-winkler distance
Details Description
What it is

Useful for short strings where identical prefixes are a strong indication of correlation.

This metric uses a scale of 0-1 to indicate the similarity between two strings, where 0 is no similarity (0 matching characters between strings) and 1 is an exact match.

How it works

Jaro-Winkler similarity takes into account:

  • matching: two characters that are the same and in similar positions in the strings.

  • transpositions: matching characters that are in different sequence order in the strings.

  • prefix scale: the Jaro-Winkler distance is adjusted favorably if strings match from the beginning (a prefix is up to 4 characters).

When to use it

This is a useful metric for cases where identical prefixes are a strong indication of correlation. Jaro-Winkler distance is fairly tolerant of transpositions (eg. us-01 and us-10 would receive a high match score), but transpositions further apart in the string are less useful.

A generally-safe number to use for Jaro-Winkler similarity in moderate to long strings is 0.9; you could use ~0.85 in cases where more leniency is okay (eg. you have other, more specific logic in the Decision).

Cosine distance

Useful for comparing large blocks of text (like descriptions or entire incidents).

Details Description
What it is

Cosine distance is most commonly used to compare large blocks of text (for example, incident descriptions) and provides an easy visualization of similarity.

How it works

For each text block you're comparing, a vector is calculated to represent the count of each unique word in the block. The cosine distance of the resulting vectors is their dot product divided by the product of their magnitudes.

                It is not length of life, but depth of life.
                Depth of life does not depend on length.

Here are the word counts for these sentences:

it 1 0

is 0 1

not 1 1

length 1 1

of 2 1

life 2 1

but 1 0

depth 1 1

does 0 1

depend 0 1

on 0 1

And here are those counts represented as a vector:

                [1, 0, 1, 1, 2, 2, 1, 1, 0, 0, 0]
                [0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1]

The cosine distance of these vectors is about 0.9 (1 is the highest similarity).

When to use it

Cosine distance is most useful to compare long blocks of text, specifically when the comparison is meant to consider the text as a whole, and not differences or misspellings in individual words.

Potential drawbacks

Cosine distance is less useful for situations where small character differences in words are insignificant, eg. typos - "nagios" and "nagois" would be treated as completely different words. Also, cosine distance ignores word order in the text blocks - for example, these two sentences have a cosine similarity of 1 even though they read completely differently:

                this is the first of the sentences, it is unscrambled
                the sentences is unscrambled, the first of this it is

A detailed walkthrough of the cosine distance implementation is available here.

Hamming distance
Details Description
What it is

Works only for same-length strings, useful for shorter text with static schema.


A simpler version of "edit distance" metrics like Levenshtein distance, the Hamming distance between two strings is the number of characters in the string that don't match (in the same position). For example, in the strings below, the Hamming distance is 2:

flowers / florets
When to use it

Hamming distance requires the compared strings to be of equal length. This is a useful similarity metric for situations where the difference between two strings may be due to typos, or where you want to compare two attributes with known lengths. For example: Low Disk Space in application myapp in datacenter us01

If you wanted to be tolerant to datacenter changes, the hamming distance should be set to 4. An average use case for Hamming distance would be around 2-3.

Potential drawbacks

In the example above, if the application name changes instead of the datacenter, a correlation would also be created. As the distance grows, the usefulness of Hamming Distance plummets. For this reason, for anything remotely more complicated than being tolerant to 1-2 character substitutions (or if the string lengths will not match), a different similarity measurement should be used.

Jaccard distance
Details Description
What it is

Useful for comparing large blocks of text (like descriptions or entire incidents). Jaccard distance is sometimes referred to as the Jaccard similarity coefficient.

How it works

It's one of the most simple measures of similarity to understand - the distance, denoted as a percentage (0 being completely similar; 1 being totally dissimilar) is calculated with the following formula:

1 - [(# of characters in both sets) / (# of characters in either set) * 100]

In other words, the Jaccard distance is 1 - the number of shared characters divided by the total number of characters (shared and un-shared). A Jaccard distance of 0.1 means that 10% or fewer characters between two incidents are different.

When to use it

Jaccard distance is very easy to interpret and especially useful in cases with large data sets - for example, in comparing the similarity between two entire incidents (as opposed to one attribute).

Potential drawbacks It is less effective for small data sets or situations with missing data. Also, different permutations of the character set don't affect Jaccard distance, so care should be taken to prevent false positives (eg. abc-12 and cba-21 have a Jaccard distance of 0).

Get started with proactive detection

With proactive detection, New Relic AI can deliver real-time failure warnings when an anomaly is detected with your app data. For more information, see Proactive detection.

EU and US datacenter and New Relic AI

New Relic's Incident Intelligence service is performed solely in the United States. By using New Relic Incident Intelligence, you agree that New Relic may move your data to, and process your data in, the US region. This applies whether you store your data in New Relic's US region data center or in our EU region data center.

For more help

Recommendations for learning more: