On-host integration executable file: JSON specifications

A New Relic infrastructure agent on-host integration will consist of at least two files: an executable file and at least one configuration file. The executable file generates JSON data that is consumed by the infrastructure agent and sent to New Relic. We refer to the JSON object as the SDK integration protocol.

Executable file requirements

The executable can be any file that runs from a command-line interface; for example:

A shell script
A scripting language script
A compiled binary

The only requirement of your executable file is that it exports JSON data, in a single line format, that meets the specifications in this document.

Recommendation: Use Go to create integrations; it's the language we use to create on-host integrations and the integration building tools. However, you can create an integration in any language.

File placement

The executable file goes in this directory:

Linux:
```
/var/db/newrelic-infra/custom-integrations
```

Windows:

C:\Program Files\New Relic\newrelic-infra\newrelic-integrations

Integration protocol v4: Example JSON output

The following section explains the new JSON schema (integration protocol v4). The SDK v4 only supports this new protocol version. These are the most important changes:

A new integration object at the top level.
The entity and metrics objects have been modified.

See the v3 to v4 migration guide for more information.

{
  "protocol_version":"4",                      # protocol version number
  "integration":{                              # this data will be added to all metrics and events as attributes,
                                               # and also sent as inventory
    "name":"integration name",
    "version":"integration version"
  },
  "data":[                                    # List of objects containing entities, metrics, events and inventory
    {
      "entity":{                              # this object is optional. If it's not provided, then the Entity will get 
                                              # the same entity ID as the agent that executes the integration. 
        "name":"redis:192.168.100.200:1234",  # unique entity name per customer account
        "type":"RedisInstance",               # entity's category
        "displayName":"my redis instance",    # human readable name
        "metadata":{}                         # can hold general metadata or tags. Both are key-value pairs that will 
                                              # be also added as attributes to all metrics and events
      },
      "metrics":[                             # list of metrics using the dimensional metric format
        {
          "name":"redis.metric1",
          "type":"count",                     # gauge, count, summary, cumulative-count, rate or cumulative-rate
          "value":93, 
          "attributes":{}                     # set of key-value pairs that define the dimensions of the metric
        }
      ],
      "inventory":{...},                      # Inventory remains the same
      "events":[...]                          # Events remain the same
    }
  ]
}

Integration protocol v3: Example JSON output

The JSON includes:

A header, with basic integration data (name, version)
A data list, which includes one or more entities reporting data (metric, inventory, and/or event data)

This diagram shows this structure:

New Relic Integrations SDK data structure diagram

Here is an example JSON output (formatted with line breaks for readability). Definitions and specifications follow this example:

{
  "name": "my.company.integration",
  "protocol_version": "3",
  "integration_version": "x.y.z",  
  "data": [
    {
      "entity": {
        "name": "my_garage",
        "type": "building",
        "id_attributes": [
          {
            "key": "environment",
            "value": "production"
          }, 
          { 
             "key": "node",
             "value": "master"
          }
        ]
      },
      "metrics": [
        {
          "temperature": 25.3,
          "humidity": 0.45,
          "displayName": "my_garage",
          "entityName": "building:my_garage",
          "event_type": "BuildingStatus"
        }
      ],
      "inventory": {
        "out_door": {
          "status": "open"
        }
      },
      "events": []
    },
    {
      "entity": {
        "name": "my_family_car",
        "type": "car",
        "id_attributes": [ 
          {
            "key": "environment",
            "value": "production"
          },
          {
            "key": "node",
            "value": "master"
          } 
        ]
      },
      "metrics": [
        {
          "speed": 95,
          "fuel": 768,
          "displayName": "my_family_car",
          "entityName": "car:my_family_car",
          "event_type": "VehicleStatus"
        }
      ],
      "inventory": {
        "motor": {
          "brand": "renault",
          "cc": 1800
        }
      },
      "events": [
        {
          "category": "gear",
          "summary": "gear has been changed"
        }
      ],
      "add_hostname": true
    }
  ]
}

JSON: General specifications

Here are general specifications for the JSON output:

JSON: Header

Here's an example of the first part of an on-host integration's JSON output:

"name":"com.myorg.nginx",
"protocol_version":"3",
"integration_version":"1.0.0",
"data": [ {entities}...]

A minimal payload would be a JSON object with only the header fields. Recommendation: If there is no data to collect, use the program return code and log messages written to stderr.

JSON header fields	Description
`name`	Required. Must be identical to the `name` field in the configuration file. Recommendation: Use reverse domain names to generate unique integration names.
`protocol_version`	Required. The version number of the exchange protocol between the integration and the agent that the integration executable is using. The current version is 3. This protocol requires infrastructure agent 1.2.25 or higher. Protocol 2 requires infrastructure agent 1.0.859 or higher. Protocol 1 is compatible with all agents. For more information, see SDK changes.
`integration_version`	Optional. The integration version. Used to track the integration version running on each host. An integration can have more than one executable. Therefore this is not simply the executable's version.
`data`	Required for reporting data. A list containing the data reported from one or more entities.

JSON: Entities

Inside the data list of the JSON output are one or more entities. The entity entry fields include:

Entity JSON fields	Description
`entity`	Required. Entity data or properties.
`metrics`	Optional. Entity related metric list.
`inventory`	Optional. Entity related inventory items.
`events`	Optional. Entity related event list.
`add_hostname`	Optional. Boolean. If `true`, the entity metrics will be decorated with the `hostname`.

Inside the data list of the JSON output are one or more entities and their data. The entity entry has two fields:

Entity data JSON fields	Description
`name`	Required. The identifier/name of the entity. Recommendation: Use reverse domain names to generate unique integration names.
`type`	Required. The kind of entity. It will be used by the infrastructure agent as a namespace to compose a unique identifier in conjunction with the `name`.
`id_attributes`	Optional. A list of key-value attributes that provide uniqueness to an entity. They are attached to the name in the form of `key=value` to ease readability, provide extra information, and improve entity name uniqueness. Identifier attributes are useful when the entity name is not enough to work as a unique identifier, or when it doesn't provide enough meaningful information. For example: `[ { "key": "service", "value": "mysql" }, { "key": "role", "value": "master" }, ... ]`

Entity data JSON fields

Description

name

Required. The identifier/name of the entity.

Recommendation: Use reverse domain names to generate unique integration names.

type

Required. The kind of entity. It will be used by the infrastructure agent as a namespace to compose a unique identifier in conjunction with the name.

id_attributes

Optional. A list of key-value attributes that provide uniqueness to an entity. They are attached to the name in the form of key=value to ease readability, provide extra information, and improve entity name uniqueness.

Identifier attributes are useful when the entity name is not enough to work as a unique identifier, or when it doesn't provide enough meaningful information. For example:

[
  {
    "key": "service", 
    "value": "mysql"
  },
  { 
    "key": "role", 
    "value": "master" 
  }, ...
]

Loopback address replacement on entity names

As of infrastructure agent version 1.2.25 or higher, protocol v3 improves remote entities uniqueness by adding local address replacement on entity names at agent level.

When several remote entities have their name based on an endpoint (either ip or hostname), and this name contains loopback addresses, there are two problems:

This localhost value does not provide valuable info without more context.
The name could collide with other service being named with a local address.

This happens when:

Endpoints names are like localhost:port.
Ports tend to be the same for a given service; for example, 3306 for Mysql.

On incoming protocol v3 data, the Infrastructure agent replaces loopback addresses on the entity name (and key) with the first available item of the following list:

Cloud provider instance ID, retrieved by the agent if applicable
Display name, set via the display_name agent config option
Hostname, as retrieved by the agent

For example, if an integration using protocol v3 returns an entity with the name localhost:3306, and the agent is running on bare metal (doesn’t have cloud provider instance ID), the display_name has not been set, and the hostname is prod-mysql-01, then the agent will replace the localhost and produce the entity name prod-mysql-01:3306.

The infrastructure agent enables loopback address replacement automatically for v3 integration protocol. You can also enable this for v2 via the agent configuration flag replace_v2_loopback_entity_names. In this case all the integrations being run by the agent using v2 will have their names replaced whenever they carry a local address.

JSON: Metric, inventory, and event data

Data values follow the executable file header. You can record three data types:

Metrics
Events
Inventory

Important

From the perspective of New Relic dashboards, infrastructure metrics and events are both classified as event data.

Infrastructure metric data typically is used for simple numeric data; for example:

Count of MySQL requests in a queue per second

Count of active connections to a specific system per minute

Besides associated metadata, a metric is essentially just a metric name and a numeric value. To learn more about this data, see Event data.

Here's an example of an entity's metric data JSON:

[ 
    {
         "event_type":"MyorgNginxSample",
         "net.connectionsActive": 54,  # metric data (a key/value pair)
         "net.requestsPerSecond": 21,  # metric data (a key/value pair)
         "net.reading": 23,   # metric data (a key/value pair)
    }
]

JSON metric data field	Description
`event_type`	Required. `event_type` defines where the metrics will be stored. Each set of metrics is stored as a sample inside the specified event type. Each integration must store its data in its own event type. If you are generating multiple types of samples from the same integration, use different event types for each. Recommendation: To ensure the event types used by your integration are unique, prefix the event type with your company name or acronym. For example, if your custom integration captures Cassandra node metrics and Cassandra column family metrics as different samples, store them in different event types, such as `MyOrgCassandraSample` and `MyOrgCassandraColumnFamilySample`. If the event type does not exist, it will be created automatically when New Relic receives data from your integration and make it available in the UI.
One or more metric data key/value pairs	Required (at least one). A metric measurement containing a name (key) and its value. Make sure these generally conform to the entity type's specification for maximum compatibility with infrastructure features. Recommendation: Prefix your metric with a category to help when navigating through metrics in the New Relic UI. New Relic integrations currently use: `net`: Number of connections, web server requests, bytes transmitted over the network, etc.; for example, `net.connectionsActive`. `query`: Metrics directly related to database queries; for example, `query.comInsertPerSecond`. `db`: Internal database metrics; for example, `db.openTables`. Use multilevel prefixes for additional grouping when it makes sense; for example, `db.innodb.bufferPoolPagesFree`. Use the `innerCamelCase` naming format; for example: `net.requestsPerSecond`. Use a metric name as close to the original one as possible while respecting the other specifications. For example: Original name: `Qcache_hits` Metric name: `db.qCacheHits`
Measurement unit	Recommendation: Specify the measurement unit using a unit suffix if it's not already included in the original metric name, as in the following examples: Percentages: Use `Percent`; for example: `cpuUtilPercent`. Rates: Use a format such as `PerSecond`. Seconds is the standard rate measurement, but you can also use other units, such as `PerMinute` or `PerDay`. Byte measurements: Use `Bytes`. Recommendation: If a metric is captured in a different unit, such as `Megabytes`, convert it to `Bytes`. For example: `db.allMemtablesOffHeapSizeBytes`. Time measurements: Use `Milliseconds`. Recommendation: If a metric is captured in a different unit, such as `Seconds`, convert it to `Milliseconds`. For example: `query.readLatency50thPercentileMilliseconds`
Value	Use a string or a number (integer or float). Strings can be used as associated metadata, allowing data to be filtered in the New Relic UI. A boolean would need to be expressed as either a string ("true", "false") or an integer (1, 0). Do not use complex types of values, such as arrays or hashes.

Infrastructure event data represents arbitrary, one-off messages for key activities on a system; for example:

Starting up a specific service

Creating a new table

You can view this data in the Infrastructure Events page and Infrastructure events heatmap. You can also query the InfrastructureEvent event type in New Relic.

Here's an example of an integration's event data JSON payload, which follows the header JSON, and field definitions.

[
   {
      "summary":"More than 10 request errors logged in the last 5 minutes",
      "category": "notifications"
   }
]

JSON event field	Description
`summary`	Required. The message to be sent. Use a simple string.
`category`	Optional. String value of one of the existing categories used in infrastructure monitoring, or a new category. The default value is `notifications`. Examples of categories: `applications` `automation` `configuration` `metadata` `notifications` `os` `packages` `services` `sessions` `system` `users`

JSON event field

Description

summary

Required. The message to be sent. Use a simple string.

category

Optional. String value of one of the existing categories used in infrastructure monitoring, or a new category. The default value is notifications. Examples of categories:

applications
automation
configuration
metadata
notifications
os
packages
services
sessions
system
users

Infrastructure inventory data captures live state system information; for example:

Configuration data
System versions installed
Other system metadata
You can view this data on the Inventory page and Infrastructure events heatmap. You can also query data related to inventory changes.
The inventory data type is a hash of one or more JSON sub-objects containing:
A unique inventory id key (required): The inventory item's identifier. This is used in combination with the integration's prefix to create a path to the inventory item's data. Like paths combine across entities and show possible variance. This ID points to a hash.
A hash of key/value pairs, one per inventory attribute. At least one is required.
Keys should be strings.
Values may either be a scalar type (string or number) or another hash object of key/values. New Relic supports hierarchy, but the final value nodes must be a scalar.
Here's an example of an integration's inventory data JSON:
```
{
    "events/worker_connections": {
        "value": 1024
                },
    "http/gzip" : {
        "value": "on"
                }
}
```

On-host integration executable file: JSON specifications

Executable file requirements

File placement

Integration protocol v4: Example JSON output

Integration protocol v3: Example JSON output

JSON: General specifications

General output and JSON formatting

Errors and logging

Exit/close of executable

JSON: Header

JSON: Entities

Loopback address replacement on entity names

JSON: Metric, inventory, and event data

Important

Metric data

Event data

Inventory data

On-host integration executable file: JSON specifications

Executable file requirements .css-21sua1{background:none;border:none;width:0;padding:0;}

File placement

Integration protocol v4: Example JSON output

Integration protocol v3: Example JSON output

JSON: General specifications

Errors and logging

Exit/close of executable

JSON: Header

JSON: Entities

Loopback address replacement on entity names

JSON: Metric, inventory, and event data

Important

Metric data

Event data

Inventory data

Executable file requirements