With applied intelligence's anomaly detection, New Relic alerts your team of any unusual behavior instantly. New Relic uses applied intelligence to constantly observe your applications. We use this information to determine your application's baseline, or expected, performance. Whenever behavior deviates from the baseline, we know right away and alert your team so you can address any errors promptly and efficiently.
There are two types of anomaly detection at New Relic: custom and automatic. Learn about which anomaly detection is right for each situation your team would like to monitor and how to implement anomaly detection in your system.
The anomaly dashboard where your team can monitor any unusual behavior in your system.
How we use anomalies
At New Relic, our own developers know how important it is to monitor the health of our applications. We want our customers to have access to the data they need whenever they need it so our team needs to be alerted if there are any outliers in our system's performance. New Relic's anomaly detection uses applied intelligence to monitor three key golden signals: throughput, error rate, and latency. With anomaly detection, our developers monitor the baseline performance for these metrics.
So, let's say that one afternoon there's a spike in response time and it's taking longer than usual for our customers to access the homepage. Anomaly detection will flag this anomalous behavior because our latency metric data has deviated from its baseline. This doesn't necessarily mean there is a problem, it just indicates that AI has registered something out of the ordinary in our system and we should take a deeper look.
We monitor this unusual behavior in a few ways. First, our team uses the anomaly dashboard so we can see what changed and when.
Explore any anomaly in your system's performance to better understand what errors you're receiving and why.
We also set up notifications for anomalies to be delivered in Slack, and we set up a webhook to deliver messages when we need them.
How to set up slack or webhook notifications for anomalies in the New Relic UI.
These events are also available for querying, creating custom dashboards, and alerting. After we set up an anomaly detection configuration (a group of apps we're interested in), we can add this configuration as a source. Then the anomalies will be automatically correlated with other data sources via incident intelligence.
There are two types of anomaly detection: automatic and custom
Automatic anomalies are the most efficient way for your team to learn about unusual behavior in your APM-monitored applications. Automatic anomaly detection is a hands-off tool your team can implement to ensure that you're notified the moment behavior in your application deviates from baseline. You can use automatic anomalies to identify the source of the problem and take the appropriate steps to get your system running smoothly again.
Custom anomalies allow increased configurability for your team. Custom anomalies provide your team with the capability to alert on any NQRL condition and to adjust and optimize your thresholds. Custom anomalies also use the same advanced tuning settings as static alerting so you can ensure your team sees only the anomaly incidents important to you.
Option
Automation Level
When To Use
Coverage
Static
Entirely configurable
When you need to set a single threshold for all your data.
All entities, all signals
Anomaly (Configurable)
Semi-automated
When you want to automatically learn trends in your data but have control over the threshold
All entities, all signals
Automatic anomaly
Completely automatic
When you want a broad understanding of changes in key metrics on your applications and services with no configuration needed. Data trends and thresholds are automatically determined through our machine learning engine.
entities, golden signals
Anomaly set-up
Once you choose to monitor anomalous behavior in your system using either our custom or automatic anomaly detection, you will need to make sure that your team is notified of any unusual behavior and that you can query and understand your data. It doesn't matter if you choose custom or automatic anomaly detection, the set-up is the same.
To use New Relic's anomaly detection, ensure that you have:
Anomaly detection is enabled automatically, at no additional cost. To receive notifications or to have a configuration (group of apps) that you can add as a source for incident intelligence, you'll need to create an anomaly detection configuration.
You can create a configuration in the anomaly detection UI:
Choose a name for your configuration that helps you easily identify it from others in your account.
Select an account.
Select up to 1,000 applications. Note that certain applications with low throughput might not be good candidates for anomaly detection, as they can be more sensitive to smaller amounts of data fluctuation.
Optional: select the golden signals you'd like to monitor for anomalies.
Workflows will soon replace configurations. Learn more about our workflows here.
Use anomaly detection with Slack:
Select Slack.
Choose which Slack channel receives notifications. You can select any existing public or private channel. This prompts the workflow to add the applied intelligence Slack application to your selected channel. Or create a new channel for anomaly detection, create the channel in Slack first, then select that channel.
Tip
If you experience an error when assigning Slack channels, make sure that the New Relic AI Slack application has been added to your Slack workspace.
Save the configuration.
You can modify the applications for each configuration at any time by selecting the configuration in the configuration table.
Use anomaly detection with webhooks:
Select Webhook.
Input the following information into the form:
Provide the webhook URL.
Provide optional custom headers.
Choose to edit the custom payload, or enable using the default payload.
Save the configuration.
You can modify the applications for each configuration at any time by selecting the configuration in the configuration table.
Use anomaly detection with workflows:
Go to Workflows.
Select Advanced.
Select build a query and select “Origins” “contains” “anomalies”.
Select entitiesData.typescontainsAPM Application.
Select entit_guid to select an entity of your choice.
Select the signalType contains and the signal you would like to be notified on.
Next, select a destination channel of your choosing.
To reduce noise, we recommend sending anomalies to a low priority notification channel such as Slack or email.
In Slack, you can mute temporarily or permanently detections coming from specific applications. You can also mute temporarily the entire channel. This is useful in the case of an incident or when the channel should otherwise not be interrupted.
To mute in Slack, select Mute this app's warnings or Mute all warnings, then select the duration. Notifications will resume once the muting duration has completed.
Muting an application permanently removes it from the configuration. To add it back in, go to one.newrelic.com > Alerts & AI > Anomaly detection, and select the configuration to edit.
Each anomaly message has several key pieces of information you can use to learn more about and start troubleshooting the potential issue:
The application name and a link to more information about it in the New Relic UI.
The metric experiencing an anomaly and a link to its details in the New Relic UI.
A graph of the metric over time to provide a visual understanding of the anomaly's behavior and degree.
An Analyze button that navigates to an analysis page in applied intelligence that identifies key attributes that are unique to the anomaly, anomalies found upstream or downstream, and any other relevant signals.
Once an anomaly has returned to normal, we send a recovery notification with the option to provide feedback. Your feedback provides our development team with input to help us improve detection quality. In the case of feedback provided on throughput anomalies, an evaluation is run each hour based on feedback to fit a more suitable model. If we helped you, you can select Yes or No.
In addition to notifications for anomalies that give you information via Slack or webhook, you can view more information about the anomalies in your environment via the anomalies Feed on the alerts and applied intelligence overview page. That tab provides a list of all the recent anomalies from every configuration in the selected account, and you can select an anomaly for a detailed analysis.
In addition to viewing anomalies in the anomalies feed, anomalies are correlated to other incidents and grouped into issues. Anomalies provide context to what has changed around a critical issue. Out of the box anomalies will be correlated with incidents of the same entity type. Correlates unique to your use case can be added using decisions.
Anomalies are displayed in various New Relic activity streams and in the applied intelligence anomalies feed. You can customize what is displayed using the anomaly visibility settings (for example, hiding throughput anomalies on an activity stream but keeping them in the anomalies feed).
To find these settings: from Alerts & AI, under Anomaly detection, click Settings.
Notes on using these settings:
These settings are applied at the user level. Changes you make won’t affect others users in your organization.
Regardless of these settings, the anomalies are still reported and available for NRQL querying.
Details on these UI sections:
AI overview and anomalies tab: Use the AI overview and anomalies tab setting to hide anomalies from the AI overview and anomalies tab setting. Please note you also can use filters specific to these views as well.
Global activity stream: Use the global activity stream section to customize what anomalies are shown in the various New Relic activity streams, including the New Relic homepage, summary, and Lookout.
Anomaly types: Use the check boxes here to hide specific types of anomalies. For example, uncheck Web throughput and Non-web throughput anomalies to hide these types of anomalies from both the activity streams and the AI overview and anomalies tab. Note they are still reported and available for querying.
You can use NRQL to query and chart your anomaly detection data using the NrAiAnomaly event. For example:
FROM NrAiAnomaly SELECT *
Important
This data has previously been attached to the ProactiveDetection event. That event was deprecated on April 7, 2021. If you use ProactiveDetection in your custom charts, you should convert those queries to using NrAiAnomaly.
Here are important attributes attached to this event:
Attribute
Description
closeTime timestamp
The time when the anomaly ended. Example: 1615304100000.
configurationType string
The type of configuration monitoring the event. If at least one configuration is monitoring the entity, this is set to configuration. Otherwise, it's set to automatic.
entity.accountId number
The New Relic account ID to which the entity belongs.
entity.domain number
The domain of the entity (currently only APM but will change with future functionality).
entity.guid string
The GUID of the entity. This is used to identify and retrieve data about the entity via NerdGraph. Identical to entityGuid.
entityGuid string
The GUID of the entity. This is used to identify and retrieve data about the entity via NerdGraph. Identical to entity.guid.
entity.name string
The name of the entity whose data was determined to be anomalous. Identical to entityName. Example: Laura's coffee service.
entityName string
The name of the entity whose data was determined to be anomalous. Identical to entity.name.
entity.type string
The type of entity (currently only APPLICATION but will change with future functionality).
evaluationType string
This is always anomaly.
event string
Indicates whether it's the beginning (open) or end (close) of the anomalous data.
openTime timestamp
The time when the anomaly opened. Example: 1615303740000.
signalType string
The type of data that was analyzed. For example, error_rate or response_time.non_web.
timestamp timestamp
The time at which the event was written.
title string
Description of the anomaly. Example: Error rate was much higher than normal.
By integrating incident intelligence with your anomaly detection, you can get context and correlations. To learn about doing this in incident intelligence, see Configure sources.
You can also select Connect to incident intelligence from inside of a configuration.
Automatic anomaly detection sends the event body in JSON format via HTTPS POST. The system expects the endpoint to return a successful HTTP code (2xx). If you use webhooks to configure automatic anomaly detection, use these examples of the webhook body format and JSON schema.
Attribute
Description
category enum
The category of data that was analyzed.
Categories include web throughput, non-web throughput, web transactions, non-web transactions, and error class.