REST API calls for New Relic Infrastructure alerts

You can manage individual Infrastructure and Integrations alerting conditions directly within the New Relic Infrastructure UI. You can also use Infrastructure's REST API to add (PUT), update (POST), delete, and list (GET) alerting conditions.

If your account hosts data in the EU data center, ensure you are using the proper API endpoints for EU region accounts.

Infrastructure REST API Advantages over using the UI
Consistency
  • Create the same set of conditions for every new tagged cluster without needing to individually set up identical conditions each time.
  • Manage multiple conditions quickly, without needing to update them individually in the UI.
Flexibility
  • Create conditions for an arbitrary group of hosts.
  • Disable or delete conditions for hosts taken offline anytime.
  • Create a condition with exclusionary filtering (for instance, environment NOT LIKE x). For more on this, see this post on exclusion filtering.
  • For AWS Cloud Integrations, select attributes that haven't been sent up by AWS yet.
  • Create compound alert conditions (by using the where_clause to specify limits on a secondary or tertiary metric).
  • Exceed the 500-facet limitation on NRQL alert conditions.
Reliability
  • Audit when a condition was last updated.

Requirements

In order to use the Infrastructure REST API, you need:

REST API calls for Infrastructure alerts are not available in New Relic's API Explorer.

The Infrastructure REST API has a limit of 3,700 alert conditions, including both active and disabled conditions. The API, whether used directly or via the UI, will reject all requests to add any additional alert conditions beyond the 3,700 alert condition limit.

Basic Infrastructure API calls

Here are some basic cURL commands and their responses for Infrastructure alert conditions. Depending on the type of condition, the DATA information you provide in the call will vary for POST (add) and PUT (update) calls. Definitions of each attribute used in the data blocks can be found in the Definitions section.

GET Infrastructure conditions

You can either GET a list of Infrastructure conditions or GET a specific Infrastructure condition. Here are a few tips for listing Infrastructure conditions.

  • For pagination, use limit (records per page) and offset (how many records to skip) parameters. Default is 50 records per page, and offset starts at 0 (skip no records).
  • To scope the results to a specific policy, use policy_id.

If you want to use the GET output response as a template for your PUT or POST input, be sure to remove the created_at_epoch_millis, updated_at_epoch_millis and id information.

GET a list of Infrastructure conditions

curl -v -X GET --header "X-Api-Key: {admin_api_key}" \
  "https://infra-api.newrelic.com/v2/alerts/conditions?policy_id=111111"

Response showing 2 of the 3 conditions for the example policy (formatted for readability and truncated):

HTTP/1.1 200 OK
Content-Length: 622
Content-Type: application/json

{
   "data":[
      {
         "type":"infra_process_running",
         "name":"Java is running",
         "enabled":true,
         "where_clause":"(`hostname` LIKE '%cassandra%')",
         "id":13890,
         "created_at_epoch_millis":1490996713872,
         "updated_at_epoch_millis":1490996713872,
         "policy_id":111111,
         "comparison":"equal",
         "critical_threshold":{
            "value":0,
            "duration_minutes":6
         },
         "process_where_clause":"(`commandName` = 'java')"
      },
      {
         "created_at_epoch_millis": 1501704525462,
         "critical_threshold": {
             "duration_minutes": 5
         },
         "enabled": true,
         "filter": {
            "and": [
                {
                     "like": {
                         "fullHostname": "Production_1"
                     }
                 }
             ]
         },
         "id": 448036,
         "name": "PROD - Host Machine's Agent Not Responding ....",
         "policy_id": 98485,
         "type": "infra_host_not_reporting",
         "updated_at_epoch_millis": 1504879191220
      }
     . . . 
   ],
   "meta":{
      "limit":50,
      "offset":0,
      "total":3
   },
   "links":{
      "next":"infra-api.newrelic.com/v2/alerts/conditions?limit=1\u0026offset=50\u0026policy_id=111111"
   }
}

To get a list of the 10 Infrastructure conditions beyond the 50 limit:

curl -v -X GET --header "X-Api-Key: {admin_api_key}" \
  "https://infra-api.newrelic.com/v2/alerts/conditions?policy_id=111111&offset=50&list=10"

GET a specific Infrastructure condition

To get information about a single Infrastructure condition:

curl -v -X GET --header "X-Api-Key: {admin_api_key}" \
 "https://infra-api.newrelic.com/v2/alerts/conditions/{condition-id}"

Response (formatted for readability):

HTTP/1.1 200 OK
Content-Length: 246
Content-Type: application/json

{
   "data":{
      "type":"infra_host_not_reporting",
      "name":"demo condition",
      "enabled":false,
      "id":13887,
      "created_at_epoch_millis":1490981583580,
      "updated_at_epoch_millis":1490981583580,
      "policy_id":23635,
      "critical_threshold":{
         "duration_minutes":100
      }
   }
}
Create (POST) an Infrastructure condition

Do not include an "id": when adding a new condition (POST). It will be generated when the condition is created.

To add an Infrastructure condition, use this basic cURL command:

curl -X POST 'https://infra-api.newrelic.com/v2/alerts/conditions' \
     -H 'X-Api-Key:{admin_api_key}' -i \
     -H 'Content-Type: application/json' \
     -d \
'{
   "data":{DATA object details
      }
   }
}'

Include details in the DATA object (-d \ section) for the type of Infrastructure condition you are adding:

Update (PUT) an Infrastructure condition

You only need to include the fields that need to be changed when updating an Infrastructure condition. The API keeps the existing values for any missing fields.

If you want to change the condition type, do not use PUT. Instead, delete the existing condition, then add (POST) a new condition with the new condition type and all fields.

To update an Infrastructure condition, use this basic cURL command. To indicate which condition is to be updated, be sure to include the "id": .

curl -X PUT 'https://infra-api.newrelic.com/v2/alerts/conditions/{condition-id}' \
     -H 'X-Api-Key:{admin_api_key}' -i \
     -H 'Content-Type: application/json' \
     -d \
'{
   "data":{DATA object details
      }
   }
}'

Include details in the DATA object (-d \ section) for the type of Infrastructure condition you are updating:

Remove (DELETE) an Infrastructure condition

To delete an Infrastructure condition, use this basic cURL command:

curl -v -X DELETE --header "X-Api-Key: {admin_api_key}" \
  "https://infra-api.newrelic.com/v2/alerts/conditions/{condition_id}"

Types of conditions

Process running conditions API data

A process running condition alerts you when the number of processes is above, below, or equal to the threshold you define. To add (POST) or update (PUT) a process running condition, use your Admin user's API key, and refer to the definitions to customize your values in the API call.

For example:

curl -X POST 'https://infra-api.newrelic.com/v2/alerts/conditions' \
     -H 'X-Api-Key:{admin_api_key}' -i \
     -H 'Content-Type: application/json' \
     -d \
'{
   "data":{
      "type":"infra_process_running",
      "name":"Java is running",
      "enabled":true,
      "where_clause":"(hostname LIKE '\''%cassandra%'\'')",
      "policy_id":{policy_id},
      "comparison":"equal",
      "critical_threshold":{
         "value":0,
         "duration_minutes":6
      },
      "process_where_clause":"(commandName = '\''java'\'')"
   }
}'

Note the extra single quotes escaping the single quote around the where_clause and process_where_clause

Metric conditions API data

A metric condition alerts you when the metric of your choice is above, below, or equal to the threshold you define. This includes:

To add (POST) or update (PUT) a metric condition, use your Admin user's API key, and refer to the definitions to customize your values in the API call.

If you are adding or updating a cloud integration alert condition:

  • For the event_type field, enter the event type generated by your selected cloud integration service (for example, ComputeSample for the AWS EC2 integration).

  • If you are setting up an alert condition on a cloud integration service that requires a provider value (for example, AWS RDS uses DatastoreSample with a provider value of RdsDbInstance or RdsDbCluster), you will need to add the "integration_provider" field and use the value that is appropriate for the service your alert condition is targeting (for example, "integration_provider":"RdsDbInstance").

  • For the select_value field, build the metric name by using the following syntax, where provider is a standard prefix string:

    provider.metric.aggregation_type
    • metric: Use the metric name as described in the New Relic documentation for your integration.
    • aggregation_type: Use Sum, Average, Minimum, or Maximum. Refer to the original documentation by the integration's cloud provider to see which statistic aggregations are available for each metric.

For example:

curl -X POST 'https://infra-api.newrelic.com/v2/alerts/conditions' \
     -H 'X-Api-Key:{admin_api_key}' -i \
     -H 'Content-Type: application/json' \
     -d \
'{
   "data":{
      "type":"infra_metric",
      "name":"Disk Space Condition",
      "enabled":true,
      "where_clause":"(hostname LIKE '\''%cassandra%'\'')",
      "policy_id":{policy_id},
      "event_type":"StorageSample",
      "select_value":"diskFreePercent",
      "comparison":"below",
      "critical_threshold":{
         "value":10,
         "duration_minutes":1,
         "time_function":"any"
      },
      "warning_threshold":{
         "value":30,
         "duration_minutes":2,
         "time_function":"any"
      }
   }
}'

Note the extra single quotes escaping the single quote around the where_clause

Host not reporting condition

A host not reporting condition alerts you when a host stops reporting. To add (POST) or update (PUT) a host not reporting condition, use your Admin user's API key, and refer to the definitions to customize your values in the API call.

For example:

curl -X POST 'https://infra-api.newrelic.com/v2/alerts/conditions' \
     -H 'X-Api-Key:{admin_api_key}' -i \
     -H 'Content-Type: application/json' \
     -d \
'{
   "data":{
      "type":"infra_host_not_reporting",
      "name":"Cassandra Host Reporting Condition",
      "enabled":true,
      "where_clause":"(hostname LIKE '\''%cassandra%'\'')",
      "policy_id":{policy_id},
      "critical_threshold":{
         "duration_minutes":12
      }
   }
}'

Note the extra single quotes escaping the single quote around the where_clause

Definitions

When formatting your cURL commands, use these values as needed. These are listed in alphabetical order, not the order they appear in your API calls.

Field Definition

comparison (enum)

Condition type: infra_metric, infra_process_running

The value used to define the threshold; for example, "["above", "below", "equal"].

critical_threshold and warning_threshold

Condition type: all

This object identifies the threshold value before opening a violation.

  • The critical_threshold is required.
  • The warning_threshold is optional and may only be used with infra_metric conditions.

The keys of this object depend on the condition type.

Condition type: infra_metric format:

"critical_threshold":{
   "value":<number>,
   "duration_minutes":<integer>,
   "time_function":"any" or "all"
},

Condition type: infra_process_running format:

"critical_threshold":{
   "value":<integer>,
   "duration_minutes":<integer>,
},

Condition type: infra_host_not_reporting format:

"critical_threshold":{
   "duration_minutes":<integer>,
},
value

The numeric value that must be breached for the condition to open a violation

duration_minutes

The number of minutes the value must be passed or met for the condition to open a violation

time_function

Indicates if the condition needs to be sustained for a certain period of time to create a violation, or if it only needs to break the threshold once within a certain period of time. If you're setting up a for at least x minutes threshold, use all; for an at least once in x minutes threshold, use any.

enabled (boolean)

Condition type: all

Whether the condition is turned on or off; true or false.

event_type (string)

Condition type: infra_metric

The metric event; for example, system metrics, process metrics, storage metrics, or network metrics. This automatically populates for Infrastructure Integrations; for example, StorageSample.

filter (string)

Condition type: all

If the condition was made in the UI, filter appears instead of where_clause; for example:

{and: [{is: {ec2InstanceType: "m3.medium"}}]}

Recommendation: Use where_clause when creating a new condition.

id (integer)

Condition type: all

The condition ID located in the URL.

  • GET: This value appears in the GET response.
  • PUT: Include this value in the DATA section.
  • POST: Do not include this in the DATA section.
  • DELETE: Include this value in the -X DELETE call.

integration_provider (string)

Condition type: infra_metric

For alerts on integrations, use integration_provider instead of event_type. To see valid values: From the API link, select Integrations.

name (string)

Condition type: all

The Infrastructure alerting condition's name; for example:

"[test] process running"

policy_id (integer)

Condition type: all

The unique ID for the alert policy's account ID associated with the condition; for example, 1234567890. This is not the policy's global ID.

process_where_clause (string)

Condition type: infra_process_running

Any filters applied to processes; for example:

"commandName = '\''java'\''"

select_value (string)

Condition type: infra_metric

The attribute name to identify the type of metric condition; for example, "network", "process", "system", or "storage". This automatically populates for Infrastructure Integrations; for example, diskFreePercent.

type (enum)

Condition type: all

The type of Infrastructure alert condition: "infra_process_running", "infra_metric", or "infra_host_not_reporting".

where_clause (string)

Condition type: all

If applicable, this identifies any Infrastructure host filters used; for example:

"(`hostname` LIKE '\''%cassandra%'\'')",

For more help

Recommendations for learning more: