Scope alert thresholds to specific instances

New Relic Alerts allows you to set alert thresholds that trigger when they are violated by any of your Java app's instances. Scoping the alert condition to instances of your app is helpful for detecting anomalies that are occurring only in a subset of your app's instances.

These sorts of anomalies are easy to miss for apps that aggregate metrics across a large number of instances. By looking at each instance New Relic Alerts can help you more quickly identify where potential problems are originating.

Example

In this example, you set up an alert policy for a Java app with three instances. You want to receive an alert notification when the condition's Critical threshold for any instance's error rate is greater than .02% for at least five minutes.

During a five-minute time period the three instances have these error rates:

App instance 4:45 p.m. 4:50 p.m. Alert notification sent?
A 0.00% 0.00% No. This instance remained below the target threshold the entire time.
B 0.02% 0.03% Yes. The alert threshold exceeded the 0.02% threshold for this instance for at least five minutes.
C 0.10% 0.00%

No. In order for New Relic to trigger an alert for this instance, the threshold violation must remain past the defined threshold at least five minutes.

However, if you had set the threshold for at least once in five minutes, then the threshold violation must remain past the defined threshold at least one minute during the five-minute period.

Create instance-based alert

This feature is available in limited release. To create an alert policy that triggers alert notifications for violations by your app's individual instances:

  1. Follow the basic workflow process to set up a policy.
  2. When creating a condition (step 2), select APM.
  3. Select Application metric as the type of condition.
  4. To have New Relic calculate alert threshold violations individually for each of the app's selected instances, select Scope to Java application instances.
  5. Select Next, select entities, then identify one or more apps for this condition.
  6. Optional: Change the time when New Relic Alerts will close the incident automatically (default is 24 hours).
  7. Continue the rest of the policy workflow process (step 3).

To receive alert notifications based on the average of all instances for your apps, select Scope to the application instead of Scope to Java application instances.

Use "By condition" incident preference

When setting the incident preference for instance alerts, New Relic recommends that you select By condition instead of By condition and entity. Although apps are the selected entities for this alert condition, New Relic Alerts evaluates each JVM as a separate entity.

If you set your Incident preference to By condition and entity, New Relic Alerts creates a separate incident for each JVM that violated the critical threshold. When an app fails across multiple JVMs, this can lead to alert "fatigue" and frustration.

Use REST API for instance alerts

To create instance-based alert conditions with the New Relic REST API, include these items with your REST API call:

Here is an example of the API request format and JSON response.

Instance alert API call

Use this format for your API call:

    curl -X PUT 'https://api.newrelic.com/v2/alerts_entity_conditions/12345.json' \
     -H 'X-Api-Key:{YOUR_API_KEY}' -i \
     -H 'Content-Type: application/json' \
     -G -d 'entity_type=application&condition_id=234567'
    
Instance alert JSON response

New Relic returns the response as JSON.

{
  "condition": [
    {
     "id": 234567,
     "type": "apm_app_metric",
     "name": "demo",
     "enabled": true,
     "entities": [
       12345
     ],
     "metric": "apdex",
     "condition_scope": "instance",   <---<<< scope indicator
     "violation_close_timer": 24,
     "terms": [
        {
         "duration": "5",
         "operator": "above",
         "priority": "critical",
         "threshold": "1",
         "time_function": "all"
        }
      ]
    }
  ] 
}

For more help

If you need additional help, get support at support.newrelic.com.

Recommendations for learning more: