REST API calls for infrastructure alerts

Use the infrastructure REST API to add, update, delete, and list alerting conditions. You can also manage individual alerting conditions using the infrastructure monitoring UI.

REST API calls for infrastructure alerts are not available in the API Explorer.

Why use the API Examples
Consistency
  • Define the same set of conditions for every cluster without having to set up identical conditions in the Infrastructure monitoring UI each time.
  • Manage multiple conditions quickly, without having to update them one by one using the UI.
Flexibility
  • Create conditions for an arbitrary group of hosts.
  • Disable or delete conditions for hosts taken offline anytime.
  • Create a condition with exclusionary filtering (for instance, environment NOT LIKE x). For more on this, see this post on exclusion filtering.
  • For AWS Cloud integrations, select attributes that haven't been sent up by AWS yet.
  • Create compound alert conditions by using the where_clause, which allows you to specify the limits on a secondary or tertiary metric.
  • Exceed the 500-facet limitation on NRQL alert conditions.
Reliability
  • Audit when a condition was last updated.

Requirements

In order to use the Infrastructure REST API, you need:

If your account hosts data in the EU data center, make sure you are using the proper API endpoints for EU region accounts.

Using infrastructure API calls

Here are some basic cURL commands and their responses for Infrastructure alert conditions. Depending on the type of condition, the DATA information you provide in the call will vary for POST (add) and PUT (update) calls.

Definitions of each attribute used in the data blocks can be found in the Definitions section.

GET infrastructure conditions

You can either GET a list of infrastructure conditions or GET a specific infrastructure condition. Here are a few tips for listing infrastructure conditions.

  • For pagination, use limit (records per page) and offset (how many records to skip) parameters. Default is 50 records per page, and offset starts at 0 (skip no records).
  • To scope the results to a specific policy, use policy_id.

If you want to use the GET response as a template for your PUT or POST input, be sure to remove the created_at_epoch_millis, updated_at_epoch_millis and id information.

GET a list of infrastructure conditions

curl -v -X GET --header "X-Api-Key: admin_api_key" "https://infra-api.newrelic.com/v2/alerts/conditions?policy_id=111111"
Example GET a list of conditions

Response showing 2 of the 3 conditions for the example policy (formatted for readability and truncated):

HTTP/1.1 200 OK
Content-Length: 622
Content-Type: application/json

{
   "data":[
      {
         "type":"infra_process_running",
         "name":"Java is running",
         "enabled":true,
         "where_clause":"(`hostname` LIKE '%cassandra%')",
         "id":13890,
         "created_at_epoch_millis":1490996713872,
         "updated_at_epoch_millis":1490996713872,
         "policy_id":111111,
         "comparison":"equal",
         "critical_threshold":{
            "value":0,
            "duration_minutes":6
         },
         "process_where_clause":"(`commandName` = 'java')"
      },
      {
         "created_at_epoch_millis": 1501704525462,
         "critical_threshold": {
             "duration_minutes": 5
         },
         "enabled": true,
         "filter": {
            "and": [
                {
                     "like": {
                         "fullHostname": "Production_1"
                     }
                 }
             ]
         },
         "id": 448036,
         "name": "PROD - Host Machine's Agent Not Responding ....",
         "policy_id": 98485,
         "type": "infra_host_not_reporting",
         "updated_at_epoch_millis": 1504879191220
      }
     . . . 
   ],
   "meta":{
      "limit":50,
      "offset":0,
      "total":3
   },
   "links":{
      "next":"infra-api.newrelic.com/v2/alerts/conditions?limit=1\u0026offset=50\u0026policy_id=111111"
   }
}

To get a list of the 10 Infrastructure conditions beyond the 50 limit:

curl -v -X GET --header "X-Api-Key: admin_api_key" "https://infra-api.newrelic.com/v2/alerts/conditions?policy_id=111111&offset=50&list=10"

GET a specific infrastructure condition

To get information about a single Infrastructure condition:

curl -v -X GET --header "X-Api-Key: admin_api_key" "https://infra-api.newrelic.com/v2/alerts/conditions/condition-id"
Example GET a specific condition

Response (formatted for readability):

HTTP/1.1 200 OK
Content-Length: 246
Content-Type: application/json

{
   "data":{
      "type":"infra_host_not_reporting",
      "name":"demo condition",
      "enabled":false,
      "id":13887,
      "created_at_epoch_millis":1490981583580,
      "updated_at_epoch_millis":1490981583580,
      "policy_id":23635,
      "critical_threshold":{
         "duration_minutes":100
      }
   }
}

Create (POST) an infrastructure condition

Do not include an "id": when adding a new condition (POST). It will be generated when the condition is created.

To add an infrastructure condition, use this basic cURL command:

curl -X POST 'https://infra-api.newrelic.com/v2/alerts/conditions' -H 'X-Api-Key:admin_api_key' -i -H 'Content-Type: application/json' -d '{"data":{DATA object details}}'

Include details in the DATA object (-d \ section) for the type of infrastructure condition you are adding:

Update (PUT) an infrastructure condition

You only need to include the fields that need to be changed when updating an infrastructure condition. The API keeps the existing values for any missing fields.

If you want to change the condition type, do not use PUT. Instead, delete the existing condition, then add (POST) a new condition with the new condition type and all fields.

To update an infrastructure condition, use this basic cURL command. To indicate which condition is to be updated, be sure to include the "id": .

Example update (PUT) a condition
curl -X PUT 'https://infra-api.newrelic.com/v2/alerts/conditions/condition-id' -H 'X-Api-Key:admin_api_key' -i -H 'Content-Type: application/json' -d '{"data":{DATA object details}}'

Include details in the DATA object (-d \ section) for the type of infrastructure condition you are updating:

Remove (DELETE) an infrastructure condition

To delete an infrastructure condition, use this basic cURL command:

curl -v -X DELETE --header "X-Api-Key: admin_api_key" "https://infra-api.newrelic.com/v2/alerts/conditions/condition_id"

Types of conditions

Process running conditions API data

A process running condition alerts you when the number of processes is above, below, or equal to the threshold you define. To add (POST) or update (PUT) a process running condition, use your Admin user's API key, and refer to the definitions to customize your values in the API call.

Example condition types

For example:

curl -X POST 'https://infra-api.newrelic.com/v2/alerts/conditions' -H 'X-Api-Key:admin_api_key' -i -H 'Content-Type: application/json' -d
'{
   "data":{
      "type":"infra_process_running",
      "name":"Java is running",
      "enabled":true,
      "where_clause":"(hostname LIKE '\''%cassandra%'\'')",
      "policy_id":policy_id,
      "comparison":"equal",
      "critical_threshold":{
         "value":0,
         "duration_minutes":6
      },
      "process_where_clause":"(commandName = '\''java'\'')"
   }
}'

Note the extra single quotes escaping the single quote around the where_clause and process_where_clause

Metric conditions API data

A metric condition alerts you when the metric of your choice is above, below, or equal to the threshold you define. This includes:

To add (POST) or update (PUT) a metric condition, use your Admin user's API key, and refer to the definitions to customize your values in the API call.

If you are adding or updating a cloud integration alert condition:

  • For the event_type field, enter the event type generated by your selected cloud integration service (for example, ComputeSample for the AWS EC2 integration).

  • If you are setting up an alert condition on a cloud integration service that requires a provider value (for example, AWS RDS uses DatastoreSample with a provider value of RdsDbInstance or RdsDbCluster), you will need to add the "integration_provider" field and use the value that is appropriate for the service your alert condition is targeting (for example, "integration_provider":"RdsDbInstance").
  • For the select_value field, build the metric name by using the following syntax, where provider is a standard prefix string:

    provider.metric.aggregation_type
    • metric: Use the metric name as described in the New Relic documentation for your integration.
    • aggregation_type: Use Sum, Average, Minimum, or Maximum. Refer to the original documentation by the integration's cloud provider to see which statistic aggregations are available for each metric.
Example

For example:

curl -X POST 'https://infra-api.newrelic.com/v2/alerts/conditions' -H 'X-Api-Key:admin_api_key' -i -H 'Content-Type: application/json' -d
'{
   "data":{
      "type":"infra_metric",
      "name":"Disk Space Condition",
      "enabled":true,
      "where_clause":"(hostname LIKE '\''%cassandra%'\'')",
      "policy_id":policy_id,
      "event_type":"StorageSample",
      "select_value":"diskFreePercent",
      "comparison":"below",
      "critical_threshold":{
         "value":10,
         "duration_minutes":1,
         "time_function":"any"
      },
      "warning_threshold":{
         "value":30,
         "duration_minutes":2,
         "time_function":"any"
      }
   }
}'

Note the extra single quotes escaping the single quote around the where_clause

Host not reporting condition

A host not reporting condition alerts you when a host stops reporting. To add (POST) or update (PUT) a host not reporting condition, use your Admin user's API key, and refer to the definitions to customize your values in the API call.

Example

For example:

curl -X POST 'https://infra-api.newrelic.com/v2/alerts/conditions' -H 'X-Api-Key:admin_api_key' -i -H 'Content-Type: application/json' -d
'{
   "data":{
      "type":"infra_host_not_reporting",
      "name":"Cassandra Host Reporting Condition",
      "enabled":true,
      "where_clause":"(hostname LIKE '\''%cassandra%'\'')",
      "policy_id":policy_id,
      "critical_threshold":{
         "duration_minutes":12
      }
   }
}'

Note the extra single quotes escaping the single quote around the where_clause

Definitions

When formatting your cURL commands, use these values as needed. These are listed in alphabetical order, not the order they appear in your API calls.

Field Definition

comparison (enum)

Condition type: infra_metric, infra_process_running

The value used to define the threshold; for example, "["above", "below", "equal"].

critical_threshold and warning_threshold

Condition type: all

This object identifies the threshold value before opening a violation.

  • The critical_threshold is required.
  • The warning_threshold is optional and may only be used with infra_metric conditions.

The keys of this object depend on the condition type.

Condition type: infra_metric format:

"critical_threshold":{
   "value":<number>,
   "duration_minutes":<integer>,
   "time_function":"any" or "all"
},

Condition type: infra_process_running format:

"critical_threshold":{
   "value":<integer>,
   "duration_minutes":<integer>,
},

Condition type: infra_host_not_reporting format:

"critical_threshold":{
   "duration_minutes":<integer>,
},
value

The numeric value that must be breached for the condition to open a violation

duration_minutes

The number of minutes the value must be passed or met for the condition to open a violation

time_function

Indicates if the condition needs to be sustained for a certain period of time to create a violation, or if it only needs to break the threshold once within a certain period of time. If you're setting up a for at least x minutes threshold, use all; for an at least once in x minutes threshold, use any.

enabled (boolean)

Condition type: all

Whether the condition is turned on or off; true or false.

event_type (string)

Condition type: infra_metric

The metric event; for example, system metrics, process metrics, storage metrics, or network metrics. This automatically populates for infrastructure integrations; for example, StorageSample or SystemSample.

filter (string)

Condition type: all

If the condition was made in the UI, filter appears instead of where_clause; for example:

{and: [{is: {ec2InstanceType: "m3.medium"}}]}

Recommendation: Use where_clause when creating a new condition.

id (integer)

Condition type: all

The condition ID located in the URL.

  • GET: This value appears in the GET response.
  • PUT: Include this value in the DATA section.
  • POST: Do not include this in the DATA section.
  • DELETE: Include this value in the -X DELETE call.

integration_provider (string)

Condition type: infra_metric

For alerts on integrations, use integration_provider instead of event_type. To see valid values: From the New Relic documentation for your cloud service, check the Find and use data section.

Example: In the AWS RDS monitoring integration documentation, you can see that the DatastoreSample event type can be used with an integration_provider value of either RdsDbInstance for DB instances, or RdsDbCluster for Aurora DB clusters.

name (string)

Condition type: all

The infrastructure alerting condition's name; for example:

"[test] process running"

policy_id (integer)

Condition type: all

The unique ID for the alert policy's account ID associated with the condition; for example, 1234567890. This is not the policy's global ID.

process_where_clause (string)

Condition type: infra_process_running

Any filters applied to processes, specifically in process running alert conditions. This parameter is mandatory for those types of alert conditions. For example:

"commandName = '\''java'\''"

runbook_url (string)

Condition type: all

The runbook URL to display in notifications.

select_value (string)

Condition type: infra_metric

The attribute name to identify the metric being targeted; for example, "cpuPercent", "diskFreePercent", "memoryResidentSizeBytes", or "memoryFreeBytes/memoryTotalBytes*100". This automatically populates for Infrastructure Integrations; for example, diskFreePercent.

type (enum)

Condition type: all

The type of infrastructure alert condition: "infra_process_running", "infra_metric", or "infra_host_not_reporting".

violation_close_timer (integer)

Condition type: all

The Violation time limit setting, expressed as hours. Possible values are 0, 1, 2, 4, 8,12, 24, 48, 72. This determines how much time will pass before a violation is automatically closed. Setting the time limit to 0 prevents a violation from being force-closed.

For new conditions, if a value is not provided, the following default values are used:

  • Host Not Responding (HNR) conditions: 0 (disabled)
  • All other conditions: 24
When updating existing conditions, if a value is provided, it overrides the existing value, but does not affect already opened violations.

where_clause (string)

Condition type: all

If applicable, this identifies any infrastructure host filters used; for example:

"(`hostname` LIKE '\''%cassandra%'\'')",

For more help

If you need more help, check out these support and learning resources: