Get started with Incident Intelligence

New Relic Applied Intelligence's Incident Intelligence provides you with an overview of all your incidents, their sources, and related events together.

diagram-applied-intelligence-workflow.png
Your sources supply the data to New Relic Applied Intelligence, which looks for potential anomalies for you to review.

Get started with Incident Intelligence

Enabling Incident Intelligence only requires four steps:

1. Configure your environment (one-time)

Incident Intelligence is a cross-account New Relic product. A lot of its power and value stems from this, since you can send in data from any New Relic account or external source for intelligent event correlation. Incident Intelligence currently supports only one shared correlation environment per master account and its sub-accounts.

When setting up an Incident Intelligence environment, a one-time admin action is required to select the associated New Relic account. The account selected should be one that your team has access to (hence selecting a master account is not always the best choice).

  • Who can set the environment? Only an administrator can set an Incident Intelligence environment, and only for the accounts they have administrator privileges for.​
  • Can more than one environment be set? An environment can be set once per master account (and its sub-accounts). If an administrator has privileges for more than one master account, setting an environment per master account (and sub-accounts) is possible.​
  • Need to change the environment's associated account? Please reach out to your New Relic Account Executive or our support team for assistance.

2. Configure sources

To get started, determine the sources for your incidents. A source refers to a data input to New Relic Applied Intelligence. You can get data from the following sources:

By integrating New Relic Applied Intelligence with your New Relic Alerts violations, you can get enhanced context and correlation for the monitors you've already configured. Adding New Relic Alerts as a source will not affect your current configuration or notifications.

To get data from Alerts:

  1. Go to one.newrelic.com > New Relic Applied Intelligence > Incident Intelligence > Sources > New Relic Alerts.
  2. Search or select the policies you want to connect to New Relic Applied Intelligence, and click Connect.

From the Sources > New Relic Alerts page, you can also add more Alerts policies or remove existing connected policies as a source.

To save your team hours in integrating individual tools that you already use with PagerDuty, you can integrate New Relic Applied Intelligence directly with your PagerDuty services. This allows New Relic Applied Intelligence to ingest, process, and enhance all of your PagerDuty incidents with a few clicks.

Connecting services to New Relic Applied Intelligence will not affect your current PagerDuty services or notifications.

To get data from PagerDuty:

  1. Make sure your PagerDuty API key has write access.
  2. Go to one.newrelic.com > New Relic Applied Intelligence > Incident Intelligence > Sources > PagerDuty.
  3. Enter your PagerDuty API key. This should be either a personal or general access API key with write access; if it's created by a user, the user should be an admin.
  4. Select the PagerDuty services you want to connect to New Relic Applied Intelligence, and click Connect.

From the Sources > PagerDuty page, you can also add more or remove existing connected services as a source.

New Relic Applied Intelligence supports Splunk Light, Splunk Cloud, and Splunk Enterprise version 6.3 and higher. By integrating with your Splunk log monitoring, New Relic Applied Intelligence:

  • Consumes your environment's log data in real time for specific searches and key term reports.
  • Correlates alerts and search reports with your other metrics and incidents.

To get data from Splunk:

  1. In your Splunk console, start a search for the relevant events.
  2. Save your search as an alert, configure your alert conditions, and choose the webhook as the delivery method.
  3. Go to one.newrelic.com > New Relic Applied Intelligence > Incident Intelligence > Sources > Splunk.
  4. Copy the collector URL, and paste it into the webhook endpoint in the Splunk console.
  5. Optional: Use Splunk tokens to enrich alert data with Splunk metadata.

To enrich alert data with your Splunk metadata, use Splunk tokens. You can leverage any search data, including search metadata and values from the first row of search results.

If you want to... Do this...
Access search data

Use the format $<fieldname>$. For example, use $app$ for the app context for the search.

Access field values

To access field values from the first result row that a search returns, use the format $result.<fieldname>$. For example, use $result.host$ for the host value and $result.sourcetype$ for the source type.

Use variables

You can leverage any of the Selected fields in the Splunk search and add any interesting fields to the Selected fields to make the data available as a variable.

The following fields automatically provide hints to the New Relic Applied Intelligence correlation engine:

  • app: will be automatically parsed as APPLICATION_NAME
  • application: will be automatically parsed as APPLICATION_NAME
  • application_name: will be automatically parsed as APPLICATION_NAME
  • cluster: will be automatically parsed as CLUSTER_NAME
  • computer: will be automatically parsed as HOST_NAME
  • Dc: will be automatically parsed as DATACENTER_NAME
  • datacenter: will be automatically parsed as DATACENTER_NAME
  • host: will be automatically parsed as HOST_NAME
  • host_name: will be automatically parsed as HOST_NAME
  • hostname: will be automatically parsed as HOST_NAME
  • transaction: will be automatically parsed as EVENT_ID
  • Transaction_id: will be automatically parsed as EVENT_ID
  • user: will be automatically parsed as USER_NAME

Alerting rules in Prometheus send alerts to an Alertmanager, which distributes them accordingly. By integrating New Relic Applied Intelligence with Prometheus Alertmanager, you can receive and correlate your Prometheus alerts with events from other sources.

To integrate with Prometheus Alertmanager:

  1. Set up your Alertmanager config file by running:

    ./alertmanager -config.file=simple.yml
  2. Go to one.newrelic.com > New Relic Applied Intelligence > Incident Intelligence > Sources > Prometheus Alertmanager.
  3. Copy the Prometheus Alertmanager URL, and paste it in the <webhook_config>/url section of your Alertmanager config file.

  4. Reload the Prometheus Alertmanager configuration by sending a SIGHUP to the process or by sending an HTTP POST request to the /-/reload endpoint.

Use the New Relic Applied Intelligence integration with Grafana's alert notifications to provide fast correlation and root cause detection for events happening across your applications and environment. The webhook notification is a simple way to send information about a state change over HTTP to a custom endpoint.

To integrate Grafana with New Relic Applied Intelligence as a new webhook notification channel:

  1. Log into your Grafana portal using Admin permissions, and choose Alerting.
  2. On the Grafana Notification Channels page, click New Channel > Webhook.
  3. Go to one.newrelic.com > New Relic Applied Intelligence > Incident Intelligence > Sources > Grafana.
  4. Copy the URL, and paste it in your new Grafana webhook.

New Relic Applied Intelligence integrates with Amazon CloudWatch to provide smarter incident management for all of your AWS services.

To integrate Amazon CloudWatch:

  1. Go to one.newrelic.com > New Relic Applied Intelligence > Incident Intelligence > Sources > AWS.
  2. Copy the URL.
  3. Create a new Amazon SNS topic.
  4. Set CloudWatch to forward all Alarms state changes to that topic:

    • In the Amazon CloudWatch UI, click Events > Event Pattern.
    • Select Service Name > CloudWatch.
    • Select Event Type > CloudWatch Alarm State Change.
    • Select Targets > SNS Topic, and select your new Amazon SNS topic.
  5. Create a new subscription:

    • In the Amazon AWS UI, click Create a Subscription.
    • Select your new Amazon SNS topic.
    • Select Protocol > choose HTTPS.
    • In Endpoint, paste the URL you previously copied from the New Relic Applied Intelligence Sources.

New Relic Applied Intelligence supports a dedicated REST API interface so you can easily integrate with additional systems. This allows instrumentation of your code or other monitoring solutions to report any kind of metric or event.

  • A metric can be a raw data point such as CPU, memory, disk utilization, or business KPI.
  • An event can be a monitoring alert, deployment event, incident, exceptions or any other change in state that you want to describe.

New Relic Applied Intelligence is totally agnostic to the event type, so you can send any type of data straight from your own systems or applications. The REST API supports secure token-based authentication and accepts JSON content as input.

For more information on authentication and the full API reference, see REST API for New Relic Applied Intelligence.

3. Configure destinations

Destinations are the data outputs to New Relic Applied Intelligence, where you can view your automatically correlated incidents. Available destinations for New Relic Applied Intelligence include PagerDuty and webhooks. For examples of destination templates, webhook formats, and JSON schema, see the Incident Intelligence destination examples.

Send data to PagerDuty

Recommendation: Create a new PagerDuty service to use as the destination for New Relic Applied Intelligence. PagerDuty services can also act as a source for New Relic Applied Intelligence. Creating one or more new services for the correlated, streamlined incident output by New Relic Applied Intelligence can help keep the differentiation clear between your original and correlated, enhanced incidents.

To configure the PagerDuty destination:

  1. Go to one.newrelic.com > New Relic Applied Intelligence > Incident Intelligence > Destinations > PagerDuty.
  2. Enter your PagerDuty API key. This should be either a personal or general access API key with write access; if it's created by a user, the user should be an admin. This can be the same API key used for the PagerDuty source (if you configured this source).
  3. Select the PagerDuty services you want to connect to New Relic Applied Intelligence, and click Connect.

When your team is ready, you can simply add on-call escalation policies for one or more PagerDuty destination services configured in New Relic Applied Intelligence. You can also transfer the policies over from your existing services, and leave those services as New Relic Applied Intelligence sources if needed.

From the Destinations > PagerDuty page, you can also:

  • Review and edit the permissions granted to New Relic Applied Intelligence for the PagerDuty services you chose. When you're ready, click Authorize.
  • Add more or delete existing connected services from the PagerDuty destination.
  • Edit permissions granted to New Relic Applied Intelligence for any service.

The following are settings for preferences and other configuration settings. For examples of destination templates, see the Incident Intelligence destination examples.

Configuration setting Description
Trigger new incidents Required. New Relic Applied Intelligence will trigger correlated parent incidents, so you can identify issues faster.
Edit incident titles Required. New Relic Applied Intelligence will automatically alter incident titles to help you orient and understand issues.
Add new integrations Required. New Relic Applied Intelligence will automatically add integrations to enable incident creation for selected services.
Add webhook extensions to sync user actions New Relic Applied Intelligence will automatically add webhook extensions to sync user actions in PagerDuty to New Relic, in real-time, and update correlated issue state.
Auto-resolve correlated incidents when parent incident is resolved When enabled, New Relic Applied Intelligence will automatically close correlated children incidents when the parent incident is resolved.
Select user that takes actions in PagerDuty on behalf of New Relic Applied Intelligence

New Relic Applied Intelligence requires a user to be able to enable the deep integration with PagerDuty, and provide additional context. You have full control of what actions we can take for you. These include:

  • Automatically add notes to incidents (required): PagerDuty incident notes are used by New Relic Applied Intelligence to enrich incidents with additional context.

  • Automatically acknowledge triggered incidents: When enabled, New Relic Applied Intelligence will acknowledge newly triggered incidents in PagerDuty to allow them to be correlated before you're being paged.

  • Use the original escalation policy: When enabled, New Relic Applied Intelligence will apply the escalation policy of the original source service to each incident.

Send data via webhook

New Relic Applied Intelligence will send the event body in JSON format by HTTPS POST. The system expects the endpoint to return a successful HTTP code (2xx).

To configure New Relic Applied Intelligence to send data via webhook:

  1. Go to one.newrelic.com > New Relic Applied Intelligence > Incident Intelligence > Destinations > Webhook.
  2. Required: Configure the unique webhook key, used in New Relic Applied Intelligence to refer to this webhook configuration and its specific settings.
  3. Required: Configure the destination endpoint where the webhook payload will be sent.
  4. Optional: Configure custom headers, which are key:value pairs of headers to be sent with the request. Example: "Authentication" "Bearer" <bearer token>
  5. Optional: Configure a custom payload template that can be used to map New Relic fields to match the destination tool's expected name and format.
  6. Optional: Configure priority mapping (critical, high, medium, or low), used to map New Relic's priorities to the priorities expected at the destination.

For examples of webhook formats and JSON schema, see the Incident Intelligence destination examples.

4. Configure pathways

Pathways tell New Relic Applied Intelligence when and where you want to receive notifications. This gives you control over the specific logic used to direct alerts to the correct team members.

  1. Go to one.newrelic.com > New Relic Applied Intelligence > Incident Intelligence > Pathways, and select Add a pathway.
  2. In the query builder box, specify or select an attribute, such as application/name. Select from the list that includes all the attributes available in PagerDuty incidents and New Relic Alert violations, or add your own attributes.
  3. Select a logical operator; for example, contains.
  4. Enter a specific value to complete the logical expression.

    • To include all issues created by the sources you configured, select Send everything. (Recommended if your team uses only one PagerDuty service to manage all incidents.)
    • To build more complex logic, use the AND/OR operators.
  5. Select one or more of the destinations you configured.

To edit or remove existing pathways, mouse over the pathway's name on the Pathways page.

Using the issue feed

After you configure sources, destinations, and pathways, Issues will begin to appear in the New Relic Applied Intelligence Issue feed. This gives you a quick overview of all of the Issues in your environment, along with some helpful information about them.

From left to right in the screenshot below, you'll see:

  • The last time the issue was updated - for example: was opened or closed, had new incidents added to it, was correlated with another issue. The issue table is sorted by last update by default.

  • An icon to indicate if the issue is correlated with other issues.

  • A human-readable title - this is created automatically by New Relic Applied Intelligence's natural language processing engine, and updated dynamically as the issue evolves.

  • The issue's state (created, active, acknowledged, or closed).

  • Its priority (determined by New Relic; this can also be dynamically updated with decision logic).

  • The path the issue has taken - data source(s) and destination(s).

  • The issue's duration - time from creation to now (if it's active) or creation to close.

  • The number of incidents included in an issue (includes incidents in the parent issue and, if applicable, incidents in correlated child issues).

This image shows the AI issue feed in New Relic One.

You can also click into any issue to see more details about it, including an analysis summary with an automatically generated symptom classification, a log of the major events that have taken place involving the issue, details about correlated issues, and links to related issues that occurred in the past.

What's the difference between an issue, incident, and event? The best way to think about these terms is as building blocks: events are raw data, transformed for consistency, from your sources; incidents are made up of one or more events; and issues are composed of one or more incidents.

In more detail:

  • Events usually indicate a state change or trigger in criteria defined by your monitoring systems (sources). An event will typically contain information about the affected entity and a description of what happened, and events are almost always triggered automatically by the monitoring system.

  • Incidents represent the observed state and evolution of failing and recovery criteria in your monitoring system - in other words, groups of events that together describe what happened in the system over time. You can think of incidents as “symptoms” of possible problems in the monitored environment; these symptoms are detected by configurations in different monitoring tools which evaluate multiple types of data streams and events.

  • Issues are groups of incidents, and represent the underlying problem that's causing the symptoms observed in the incidents. When a new incident is created, New Relic Applied Intelligence opens an issue containing this incident and evaluates the other open issues to look for correlations. In addition to incident data and correlated issue information, issues also include a progress report, which details actions and events involving the issue over time, as well as other metadata like the issue's classification and related activity.

Get started with decisions

Use Applied Intelligence decisions to further the Applied Intelligence correlation logic. Recommendations are automatically provided to you, and you can create decision logic to provide feedback and train our AI system over time.

New Relic One: AI decisions
Here is an example of a New Relic Applied Intelligence decision that can be used to create correlations between issues.

To start working with decisions:

  1. Go to one.newrelic.com > New Relic Applied Intelligence > Incident Intelligence > Decisions.
  2. Review the list of active decisions and information about them.
  3. To see the detailed rule logic that creates correlations between your issues, click the decision.
  4. To give feedback about any decision, click the like [like icon] or dislike [dislike icon] icons.
  5. Use any of the other available options to enable, disable, edit, or delete decisions.

Decisions can be created in three different ways:

  1. By default - New Relic Applied Intelligence includes a set of default decisions that are applicable to many different production environments. This enables issue correlation to work out of the box without requiring extensive configuration to get started.
  2. Suggested decisions - Over time, New Relic Applied Intelligence will learn from your data and suggest new decisions that you can enable to reduce noise in your environment. Note: In order for suggested decisions to be effective for your account, Incident Intelligence needs to ingest at least 5,000 incidents/month.
  3. By your team - you can add your own logic with the flexible, powerful decision engine. To create your own decision:
    1. Go to one.newrelic.com > New Relic Applied Intelligence > Incident Intelligence > Decisions, and click Add a decision.
    2. Use the following decision builder guidelines to build the decision.
    3. When you're done adding and fine-tuning logic, save the decision.

All new decisions are enabled by default, so check back later to see how many incidents were correlated with your rule logic.

Decision builder Guidelines
1. Tell us what data to compare

Define up to two segments of incoming data (from any of the sources you've configured) to compare based on any attribute. Choose from the dropdown, or add your own.

A broad range of comparison operators is supported, from basic “strict equals” to regular expression matching to many advanced similarity algorithms.

2. Tell us what to correlate.

Now that you've chosen the data to be compared, add rules to define when those issues should be correlated (merged). You can correlate issues based on attributes (for example, source equals aws) or on a similarity algorithm that assesses the entire incident.

3. Give it a name.

After you configure your decision logic, provide an easily-readable name and description that will be used to identify it later.

4. Optional: Configure advanced settings.

You can also change some of the other settings for the decision:

  • Time window: Set the maximum time between incidents for them to be eligible for correlation with this decision.

  • Issue priority: The default setting is to inherit priority (whichever is higher), but you can override this to add higher or lower urgency to correlated issues.

  • Frequency: Change this setting to modify the minimum number of incidents that need to meet the rule logic in order for the decision to trigger.

  • Similarity: If you're using similar to comparison operators in your rule logic, you can choose from a list of algorithms to use and set their sensitivity. The algorithm and threshold you choose will apply to all similar to operators in your rule logic. To learn more about when an algorithm should be used, mouse over its name.

For technical details of some of the correlation algorithms that are available to choose from in the New Relic Applied Intelligence Decision engine, see the following options:

Levenshtein distance
Details Description
What it is

Useful for comparing short strings with static schema and fixed length, like host names.

One of the similarity measures used in the New Relic Applied Intelligence Decision engine is Levenshtein distance, otherwise known as edit distance. Common applications of Levenshtein distance include spell checkers, computational biology, and speech recognition.

The default similarity threshold for new New Relic Applied Intelligence Decisions is an edit distance of 3 - you can change this in the Advanced mode of the Decision builder.

How it works

The Levenshtein distance between two strings is the minimum number of single-character edits to get from one string to the other. Allowed edit operations are deletion, insertion, and substitution.

Examples

number/bumble: 3 (number → bumber → bumblr → bumble)

trying/lying: 2 (trying → rying → lying)

strong/through: 4 (strong → htrong → throng → throug → through)

When to use it

Levenshtein distance is most useful for comparing relatively short strings with static schema and fixed length, such as host names - for example, the Levenshtein distance between us-a1 and us-a2 is only 1, providing strong support for correlation.

Potential drawbacks

Levenshtein distance is not normalized by default to take into account string lengths. 'a' and 'q' have a Levenshtein distance of 1, whereas "the quick brown fox jumps over the lazy dog" and "the quick brown rat jumps over the lazy dog" have a Levenshtein distance of 3.

Fuzzy score
Details Description
What it is

Useful for same-length strings where the same prefix would be a good indicator of correlation. Between two strings, a high fuzzy score indicates high similarity.

How it works

The fuzzy score algorithm works by allocating "points" for character matches between strings:

  • One point for each matching character

  • Two bonus points for subsequent matches

Examples

Example: Decisions / dcsions

d: 1

c: 1

i 1

s: 2

o: 1

n: 1

si: 2

io: 2

on: 2

ns: 2

= 15 points

When to use it

Fuzzy score is most useful for strings that have the same, and relatively short (ideally <5 character), prefixes. A minimum guaranteed score would be (length(expected prefix) * 3) - 2.

Potential drawbacks

If the first character of the first string can't be found in the second string, no points are awarded. For example, the fuzzy score for us01 and cn01 is 0. For "ru01" and the second is "us01", the score is 1, since fuzzy score finds "u" at position 1, but cannot find an "s" afterwards.

Fuzzy wuzzy ratio

Useful for comparing strings of similar length.

Details Description
What it is

The fuzzy wuzzy family of similarity measures was developed by SeatGeek to help find tickets for the same event that have different labels across multiple platforms. The fuzzy wuzzy ratio for two strings is expressed as a percentage, where a higher number indicates a more similar string. It's based on the SequenceMatcher algorithm in Python's difflib.

When to use it

Fuzzy wuzzy ratio is effective for very short (eg. hostname) or very long (eg. event description) strings, especially in comparing strings of similar length.

Potential drawbacks

This algorithm is too sensitive to be used effectively for 3-10 word strings; one of the other modifications to fuzzy wuzzy (see below) may be a better choice.

Fuzzy Wuzzy partial ratio

Useful for comparing strings of different length.

Details Description
What it is

This modification to the fuzzy wuzzy algorithm helps address the effective length limitation described previously. With fuzzy wuzzy partial ratio, the shorter string is compared to each substring of the same length within the longer string. The score of the “best matching” substring is used to determine the fuzzy wuzzy partial ratio.

Examples

For example, between the following strings:

DevOps and SRE teams

DevOps

…”DevOps” (the shorter string, length = 6) would be compared to each substring with length 6 within “DevOps and SRE teams.” Since one of those substrings (“DevOps”) is a perfect match, the fuzzy wuzzy partial ratio for these two strings will be high.

When to use it

Fuzzy wuzzy partial ratio is especially effective for the types of comparisons the basic fuzzy wuzzy algorithm fails at: 3-10 word strings where some significant substrings are likely to be overlapping.

Potential drawbacks

Where fuzzy wuzzy may be too conservative, fuzzy wuzzy partial match may be more liberal than expected with correlations - adjust the threshold in the Decision builder according to your needs.

Fuzzy wuzzy token set ratio

Useful for comparing strings where the information may not be in the same order, and of possible different lengths. Works best for sentences such as messages, descriptions, etc.

Details Description
How it works

The token set ratio algorithm follows a few steps to compare strings:

  1. Tokenize each string (eg. “DevOps and SRE teams” → “DevOps” “and” “SRE” “teams”; “SRE team and DevOps engineers” → “SRE” “teams” “and” “DevOps” “engineers”)

  2. Combine intersecting tokens into a new string, leaving the remaining tokens. (eg. intersecting: ”DevOps”, “and”, “SRE”; remainder1: “teams”; remainder2: “team”, “engineers”)

  3. Alphabetize each token group (eg. “and, DevOps, SRE”, “teams”, engineers, team”)

  4. Compare the following pairs of strings:

    1. Intersection group

    2. Intersection group + remainder1

    3. Intersection group + remainder2

The comparison from these pairs (“best matches”) is the fuzzy wuzzy token set ratio.

When to use it

This metric is helpful in cases where similar strings may have overlapping words but different construction - for example, event descriptions for different issues with the same resource.

Potential drawbacks Where fuzzy wuzzy may be too conservative, fuzzy wuzzy token set match may be more liberal than expected with correlations - adjust the threshold in the Decision builder according to your needs.
Jaro-winkler distance
Details Description
What it is

Useful for short strings where identical prefixes are a strong indication of correlation.

This metric uses a scale of 0-1 to indicate the similarity between two strings, where 0 is no similarity (0 matching characters between strings) and 1 is an exact match.

How it works

Jaro-Winkler similarity takes into account:

  • matching: two characters that are the same and in similar positions in the strings.

  • transpositions: matching characters that are in different sequence order in the strings.

  • prefix scale: the Jaro-Winkler distance is adjusted favorably if strings match from the beginning (a prefix is up to 4 characters).

When to use it

This is a useful metric for cases where identical prefixes are a strong indication of correlation. Jaro-Winkler distance is fairly tolerant of transpositions (eg. us-01 and us-10 would receive a high match score), but transpositions further apart in the string are less useful.

A generally-safe number to use for Jaro-Winkler similarity in moderate to long strings is 0.9; you could use ~0.85 in cases where more leniency is okay (eg. you have other, more specific logic in the Decision).

Cosine distance

Useful for comparing large blocks of text (like descriptions or entire incidents).

Details Description
What it is

Cosine distance is most commonly used to compare large blocks of text (for example, incident descriptions) and provides an easy visualization of similarity.

How it works

For each text block you're comparing, a vector is calculated to represent the count of each unique word in the block. The cosine distance of the resulting vectors is their dot product divided by the product of their magnitudes.

Examples
                It is not length of life, but depth of life.
                Depth of life does not depend on length.
                

Here are the word counts for these sentences:

it 1 0

is 0 1

not 1 1

length 1 1

of 2 1

life 2 1

but 1 0

depth 1 1

does 0 1

depend 0 1

on 0 1

And here are those counts represented as a vector:

                [1, 0, 1, 1, 2, 2, 1, 1, 0, 0, 0]
                [0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1]
                

The cosine distance of these vectors is about 0.9 (1 is the highest similarity).

When to use it

Cosine distance is most useful to compare long blocks of text, specifically when the comparison is meant to consider the text as a whole, and not differences or misspellings in individual words.

Potential drawbacks

Cosine distance is less useful for situations where small character differences in words are insignificant, eg. typos - "nagios" and "nagois" would be treated as completely different words. Also, cosine distance ignores word order in the text blocks - for example, these two sentences have a cosine similarity of 1 even though they read completely differently:

                this is the first of the sentences, it is unscrambled
                the sentences is unscrambled, the first of this it is
            

A detailed walkthrough of the cosine distance implementation is available here.

Hamming distance
Details Description
What it is

Works only for same-length strings, useful for shorter text with static schema.

Examples

A simpler version of "edit distance" metrics like Levenshtein distance, the Hamming distance between two strings is the number of characters in the string that don't match (in the same position). For example, in the strings below, the Hamming distance is 2:

flowers / florets
When to use it

Hamming distance requires the compared strings to be of equal length. This is a useful similarity metric for situations where the difference between two strings may be due to typos, or where you want to compare two attributes with known lengths. For example: Low Disk Space in application myapp in datacenter us01

If you wanted to be tolerant to datacenter changes, the hamming distance should be set to 4. An average use case for Hamming distance would be around 2-3.

Potential drawbacks

In the example above, if the application name changes instead of the datacenter, a correlation would also be created. As the distance grows, the usefulness of Hamming Distance plummets. For this reason, for anything remotely more complicated than being tolerant to 1-2 character substitutions (or if the string lengths will not match), a different similarity measurement should be used.

Jaccard distance
Details Description
What it is

Useful for comparing large blocks of text (like descriptions or entire incidents). Jaccard distance is sometimes referred to as the Jaccard similarity coefficient.

How it works

It's one of the most simple measures of similarity to understand - the distance, denoted as a percentage (0 being completely similar; 1 being totally dissimilar) is calculated with the following formula:

1 - [(# of characters in both sets) / (# of characters in either set) * 100]

In other words, the Jaccard distance is 1 - the number of shared characters divided by the total number of characters (shared and un-shared). A Jaccard distance of 0.1 means that 10% or fewer characters between two incidents are different.

When to use it

Jaccard distance is very easy to interpret and especially useful in cases with large data sets - for example, in comparing the similarity between two entire incidents (as opposed to one attribute).

Potential drawbacks It is less effective for small data sets or situations with missing data. Also, different permutations of the character set don't affect Jaccard distance, so care should be taken to prevent false positives (eg. abc-12 and cba-21 have a Jaccard distance of 0).

Use suggested responders

If you’re using PagerDuty as your incident notification tool, New Relic Applied Intelligence can help you get to faster issue resolution by suggesting the most relevant team members to help resolve it.

If the person on-call is stuck on the issue or needs more context to make informed troubleshooting decisions, they can check out the suggested responder and either contact that team member or search to find documentation that person may have authored about the issue. In many cases, just seeing who the suggested responder is can help with inferring what the underlying problem may be.

To get started with suggested responders, all you need to do is to enable PagerDuty as a source for New Relic Applied Intelligence. New Relic Applied Intelligence will learn from your historical PagerDuty alert data and train a powerful machine learning model to provide suggestions in real time for every new incident. You should expect to see up to 3 relevant suggestions for each new incident. In order to help the model improve, you can provide feedback (upvote/downvote) on whether the suggestions were helpful or not.

You can see the suggestions in two places:

  • The issue page, where you can also provide feedback​.

  • Directly within PagerDuty: in-case you’re using PagerDuty as a destination as well, we will post the suggestions into the payload of New Relic Applied Intelligence issue notifications. The information can be accessible to you directly within PagerDuty (both UI and API).

Limitations and considerations:

  1. In order to train the model, suggested responder uses information PD provides about individuals. We ingest incident information only, not users’ contact details.

  2. Suggested responders are available for New Relic customers who are using New Relic Applied Intelligence and using PagerDuty as their incident notification tool.

  3. Suggested responders do not account for other factors such as team member on-call availability at the time of incident.

Limits

Rate Limiting Source Data

Product Time Max allowed events Action
Free 10 minutes 20 Drop
Free 1 month 1000 Drop
Paid 1 minute 20 -

In the free offering of Incident Intelligence, if you exceed the limits above, sent events will be dropped. Please check NRIntegrationError for visibility into this.

In the paid offering of Incident Intelligence, the limits above are currently soft, and data processing can be delayed. Please contact support@newrelic.com to increase this limit.

Capabilities (Free vs. Paid)

Product Free Paid
All base features Y Y
Component Classification - Y
Signal Classification - Y
Suggested Responders - Y

Get started with proactive detection

With proactive detection, New Relic Applied Intelligence can deliver real-time failure warnings when an anomaly is detected with your app data. For more information, see Proactive detection.

EU and US datacenter and Incident Intelligence Data

New Relic's Incident Intelligence service is performed solely in the United States. By using New Relic Incident Intelligence, you agree that New Relic may move your data to, and process your data in, the US region. This applies whether you store your data in New Relic's US region data center or in our EU region data center.

If you elect to use the Suggested Responder feature and manage EU-based individuals, you may need to confirm that an appropriate data processing agreement is in place.

For more help

Recommendations for learning more: