Scope alert thresholds to specific instances

New Relic Alerts allows you to set alert thresholds that trigger when they are violated by any of your Java app's instances. Scoping the alert condition to instances of your app is helpful for detecting anomalies that occur only in a subset of your app's instances.

These sorts of anomalies are easy to miss for apps that aggregate metrics across a large number of instances. By looking at each instance, you can more quickly identify where potential problems are originating.

Example

In this example, you set up an alert policy for a Java app with three instances. You want to open a violation when the condition's critical threshold for any instance's error rate is greater than .02% for at least five minutes.

During a five-minute time period the three instances have these error rates:

App instance 4:45 p.m. 4:50 p.m. Violation opened?
A 0.00% 0.00% No. This instance remained below the target threshold the entire time.
B 0.02% 0.03% Yes. The alert threshold exceeded the 0.02% threshold for this instance for at least five minutes.
C 0.10% 0.00%

No. In order for New Relic to open a violation for this instance, the threshold value must be breached for at least five consecutive minutes.

However, if you had set the threshold for at least once in five minutes, then the threshold value must be breached at least once during the five-minute period.

Create instance-based alert condition

To create an alert policy that triggers alert notifications for violations by your app's individual instances:

  1. Follow the basic workflow process to set up a policy.
  2. When creating a condition (step 2), select APM.
  3. Select Application metric as the type of condition.
  4. To have New Relic calculate alert threshold violations individually for each of the app's selected instances, select Scope to Java application instances.
  5. Select Next, select entities, then identify one or more apps for this condition.
  6. Optional: Change the time when New Relic Alerts will force-close the violation (default is 24 hours).
  7. Use By condition or By condition and entity incident preference.
  8. Continue the rest of the policy workflow process (step 3).

To open violations based on the average of all instances for your apps, select Scope to the application instead of Scope to Java application instances.

Use "By condition" incident preference

When setting the incident preference for policies containing instance-based alert conditions, we recommend that you select By condition instead of By condition and entity. Although apps are the selected entities for this alert condition, we evaluate each JVM as a separate entity.

If you set your Incident preference to By condition and entity, a separate incident will be opened for each JVM that breaches the critical threshold. When an app fails across multiple JVMs, this can lead to alert "fatigue" and frustration.

Use REST API for instance alerts

To create instance-based alert conditions with the New Relic REST API, include these items with your REST API call:

Here is an example of the API request format and JSON response.

Instance alert API call

Use this format for your API call:

    curl -X PUT 'https://api.newrelic.com/v2/alerts_entity_conditions/12345.json' \
     -H 'X-Api-Key:{YOUR_API_KEY}' -i \
     -H 'Content-Type: application/json' \
     -G -d 'entity_type=application&condition_id=234567'
    
Instance alert JSON response

New Relic returns the response as JSON.

{
  "condition": [
    {
     "id": 234567,
     "type": "apm_app_metric",
     "name": "demo",
     "enabled": true,
     "entities": [
       12345
     ],
     "metric": "apdex",
     "condition_scope": "instance",   <---<<< scope indicator
     "violation_close_timer": 24,
     "terms": [
        {
         "duration": "5",
         "operator": "above",
         "priority": "critical",
         "threshold": "1",
         "time_function": "all"
        }
      ]
    }
  ] 
}

For more help

If you need additional help, get support at support.newrelic.com.

Recommendations for learning more: