Integration configuration file specifications (agent v1.8.0+)

Integrations are external programs that are executed by the Infrastructure agent. Each integration returns, via standard output, custom metrics, events or inventory from their monitored services, enabling to monitor a range of On-Host Services. Users can monitor their custom services by creating programs that print an accepted JSON format.

Since Infrastructure agent 1.8.0, a new integration engine is available, featuring:

  • Simpler setup of integrations: less files, less mandatory properties.
  • Flexible configuration via command-line arguments, environment variables or external files.
  • Ability to group different integrations in the same file.
  • Hot reload. Adding a new integration or changing its configuration does not require restarting the agent.
  • Timeouts: if an integration does not respond before a user-specified time, the integration process is killed and restarted.
  • Backwards compatibility with the previous integrations' engine.

Quick reference table

This section provides a brief summary of the properties provided to configure an integration. For a comprehensive description of all the properties, with sample values, please read the following sections or click the property name to view more detailed information.

From the properties below, only the name property is mandatory to specify. The remaining properties are optional.

Property name Description
name Name of the integration. If the exec field is not set it will also be the name of the integration executable
exec Full path to the integration executable, plus arguments. It may be a single-line string or a string array. If left unspecified, the exec field will default to the name field.
env YAML map containing the environment variables to be passed to the integration, where key is the environment variable name, and value is the variable value
config Configuration that will be written as an external file and the path that will be passed to the integration with the CONFIG_PATH environment variable or the ${config.path} variable placeholder
config_template_path Any external file whose path will be passed to the integration with the CONFIG_PATH environment variable or the ${config.path} variable placeholder. Its usage allows applying discovery and secrets binding to any external configuration
integration_user Name of the user to run the integration as
interval Time between consecutive executions of the integration. It must be a number followed by a time unit (s, m or h), without spaces
inventory_source Allows overriding the category and term of the inventory source
labels Map with labels that will decorate the metrics, events and inventory of the integration
timeout A number followed by a time unit (ms, s, m or h) indicating that an integration that has responded within this time period must be killed and restarted
working_dir Working directory for the integration binary. If left unspecified,

Integrations configuration comprehensive guide

You can setup your integrations as a YAML file under the following directory:

  • Linux: /etc/newrelic-infra/integrations.d
  • Windows: C:\Program Files\New Relic\newrelic-infra\integrations.d

An integrations YAML must have an integrations top-level section containing a YAML array, where each entry represents an integration and its configuration. Please read the rest of this document to know how to configure your integration instances.

For each integration entry, only the name property is mandatory. The other properties are optional.

Example:

integrations:
  # New Relic integration that does not require any configuration
  - name: nri-docker
  # New Relic integration that gets its configuration from the environment
  - name: nri-mysql
    env:
      PORT: 3306
      USERNAME: newrelic
      PASSWORD: n3wr3l1cp4wd # to hide this field, read the secrets management page
  # Any free-form integration executed via a user-provided command
  - name: my-own-integration
    exec: python /opt/integrations/my-script.py --host=127.0.0.1

You can have as many configuration YAML files as you want, and freely group their your integration instances.

Each configuration YAML file can also contain discovery and variables top-level sections.

IMPORTANT: Since version 1.8.0, the Agent detects changes in the configuration files (creation, modification or deletion). Configuration changes will be effective without having to restart the agent. Be careful to only save the configuration file when you have defined the correct version. Any versions saved with intermediate changes can be loaded by the agent causing it stop executing integrations until a correct version is saved.

The remainder of this document describes integration configuration properties grouped by their functionality:

Selecting an integration to run

There are two properties to select which integration will run: name and exec.

name is the only mandatory property for any integration entry. The remaining properties specified in this document are optional.

Example:

integrations:
  - name: nri-docker
  - name: my-integration
    exec: /usr/local/bin/my-integration --metrics --inventory

name

The mandatory name property can work in two ways:

  • If the exec property is set, the name property will only provide an identifier for the integration instance. This identifier is used in log messages and to provide a default inventory category/source in the form integration/<name> (e.g. integration/nri-redis). This inventory path can be overridden with the inventory_source configuration option.
  • If the exec property is not set, the Agent will look for (and execute) an executable with the name value in any of the following folders:

    • Linux:
    • /var/db/newrelic-infra/custom-integrations/newrelic-integrations/bin
    • /var/db/newrelic-infra/custom-integrations/newrelic-integrations
    • Windows
    • C:\Program Files\New Relic\newrelic-infra\newrelic-integrations\bin
    • C:\Program Files\New Relic\newrelic-infra\newrelic-integrations

    If there is no executable with this name in the above folders the Agent will log an error and the integration will not be executed.

    In Windows you must not append the .exe extension to the name. The Agent will do this for you (e.g. name: nri-mysql would look for nri-mysql.exe in the above folders).

exec

The exec optional property allows writing down the path, command and command-line arguments of the integration to execute. When none of the path folders nor arguments have spaces, it can be written in a single-line string:

- name: my-integration
  exec: /usr/bin/python /opt/integrations/my-script.py --host=127.0.0.1

If any of the path/arguments have spaces that are part of a single element, you can use a YAML array notation:

- name: my-integration
  exec:
    - C:\Program Files\My Integration\integration.exe
    - --host
    - 127.0.0.1
    - --port
    - 8080

The default working directory is the root directory of the agent configuration. It can be overridden with the working_dir property.

Passing configuration to the integration executable

Often integration executables need to receive a configuration to work properly (e.g. hostname and port of the monitored system, user credentials, etc...).

The Infrastructure Agent allows configuring the integration commands by three means (which you can combine):

  • Command-line arguments, passed in the exec property (see the previous section).
  • Environment variables, using the env section below.
  • Configuration files (whose path needs to be passed through environment variables or command-line arguments, as shown later in this section).

Example:

integrations:
  - name: my-integration
    exec: /opt/path/bin/script --host 127.0.0.1 --port 8081
  - name: nri-mysql
    env:
      STATUS_URL: http://10.0.0.3/server-status?auto
      REMOTE_MONITORING: true

env

The env property allows you to set environment variables that will be passed to the executable. It is a key-value map with the required variables.

Example:

integrations:
  - name: nri-postgresql
    env:
      DATABASE: postgres
      PORT: 6432
      COLLECTION_LIST: '["postgres"]'
      COLLECT_DB_LOCK_METRICS: false
      VERBOSE: 1

If you expect your integration to receive the configuration from the host's environment rather than specifying it explicitly in the configuration file, you need to set the required variables in the Infrastructure Agent passthrough_environment global configuration property

config

Some integration commands may get their configuration from an external file. If your integration requires a configuration file, nothing prevents you from directly passing its path directly as a command-line argument or an environment variable. For example:

integrations:
  - name: nri-flex
    env:
      CONFIG_FILE: /etc/nri-flex/configs/http-service.yaml
  - name: other-integration
    exec: /opt/integration/integration -f /opt/integration/config.properties

The above example assumes that the http-service.yaml and config.properties files exist. We can see that the nri-flex integration is expecting the http-service.yaml complete path via the CONFIG_FILE environment variable and the other-integration expects the complete config.properties path after the -f command-line flag.

In the above example, it is necessary for the integration installer/configurator that the configuration files exist in the provided path and that the Agent and Integrations have read permissions on them.

If you prefer to keep your configuration file with the rest of the integration configuration, you can use the config section in the integration entry, which can contain a valid YAML object or just a multi-line string:

integrations:
  - name: nri-flex
    env:
      CONFIG_FILE: ${config.path}
    config:
      name: csvFileExample
      apis: 
        - name: csvFile
          file: /Users/hello/test.csv
  - name: other-integration
    exec: /opt/integration/integration -f ${config.path}
    config: |
      example.cfg.verbose=true
      example.cfg.logger=/var/logs/integration.log
      example.cfg.hostname=localhost
      example.cfg.port=9025

In the above examples, every time the nri-flex integration is executed, the Agent will create a temporary file with the following contents:

name: csvFileExample
apis: 
  - name: csvFile
    file: /Users/hello/test.csv

The above YAML is only a configuration example for the nri-flex integration. The Agent ignores its contents; instead it creates a temporary file and replaces the ${config.path} variable placeholder with its path. When the integration completes execution the temporary file is removed.

Analogously the Agent will create another temporary file before executing the other-integration integration:

example.cfg.verbose=true
example.cfg.logger=/var/logs/integration.log
example.cfg.hostname=localhost
example.cfg.port=9025

It will also replace the -f ${config.path} command-line placeholder with the temporary path of the written file.

By convention, if you do not place the ${config.path} variable in any command-line argument or environment variable value, the Agent will pass the path of the configuration file via the CONFIG_PATH environment variable:

# assuming that nri-example command is prepared to receive the configuration
# file via the CONFIG_PATH environment variable
integrations:
  - name: nri-example
    config:
      name: csvFileExample
      apis: 
        - name: csvFile
          file: /Users/hello/test.csv

Using config with secrets management and discovery mechanisms

The main motivator for using a config section instead of hardcoding the full path of an external file is that you can insert ${variable} placeholders to apply Discovery and Secrets management.

Example :

variables:
  my_credentials:
    vault:
      http:
        url: http://my.vault.host/v1/newengine/data/secret
        headers:
          X-Vault-Token: my-vault-token
discovery:
  docker:
    match:
      label.service: foo
integrations:
  - name: foo-monitor
    exec: /opt/foo/bin/monitor --config=${config.path}
    config: |
      foo.host=${discovery.ip} 
      foo.port=${discovery.port}
      foo.user=${my_credentials.user}
      foo.password=${my_credentials.password}

(for a better understanding of the variables and discovery sections, please visit the Discovery and Secrets management documentation).

The above example relies on the following premises:

  • There is a Vault service that allows retrieving a JSON object formed by the user and password fields.
  • There may be a variable number of Docker containers labeled with service=foo, which are accessible from the Agent host via a discoverable public IP and Port.
  • The user has configured the foo-monitor integration to monitor all the service=foo labeled containers, which share a common user and password. Each instance of the foo-monitor integration requires executing the /opt/foo/bin/monitor executable, passing the text configuration inside the config section via the --config=<path> command-line argument.

As example of workflow, imagine that the Vault invocation returns the following JSON:

{"user":"monitorer","password":"5up3r53cr3t!"}

, and, at the moment of executing the foo-monitor integration, there are three running containers labeled with service=foo:

  1. ip: 10.0.0.3, port: 8080
  2. ip: 10.0.0.3, port: 8081
  3. ip: 10.0.0.3, port: 8082

The Agent then creates the following three temporary files, using the contents of the config property as a template, but replacing the ${placeholders} by the acquired variables and discovered items (files' path is invented for the sake of simplicity):

  • First match (/tmp/123_discovered):

    foo.host=10.0.0.3 
    foo.port=8080
    foo.user=monitorer
    foo.password=5up3r53cr3t!
    
  • Second match (/tmp/456_discovered):

    foo.host=10.0.0.3 
    foo.port=8081
    foo.user=monitorer
    foo.password=5up3r53cr3t!
    
  • Third match (/tmp/789_discovered)

    foo.host=10.0.0.3 
    foo.port=8082
    foo.user=monitorer
    foo.password=5up3r53cr3t!
    

After the config variable placeholders have been replaced and the temporary files have been created, the /opt/foo/bin/monitor executable is executed three times (one per matched container), replacing the ${config.path} command-line placeholder with the temporary file corresponding to each discovered configuration:

  • First match: /opt/foo/bin/monitor --config=/tmp/123_discovered
  • Second match: /opt/foo/bin/monitor --config=/tmp/456_discovered
  • Third match: /opt/foo/bin/monitor --config=/tmp/789_discovered

To ensure security and to minimize the chance of leaking secrets to disk, the agent will:

  • Create the files owned by the Agent user, e.g. root or nri-agent, depending on the user you have configured to run the Agent.
  • Set read permissions only for the owner.
  • Remove the created files when the integration instance finishes its execution.

config_template_path

If you want to use the aforementioned Secrets Management and Discovery in the configuration files that you are passing to the integration executable, but you prefer to keep them as an individual file, you can use the config_template_path: <path> option. It will work exactly as in the config section:

  1. The Agent applies Secrets Management and Discovery to the file contents
  2. The Agent creates different temporary files that are passed to the integration via the ${config.path} placeholder (or the CONFIG_PATH environment variable)

Example:

discovery:
  docker:
    match:
      name: /^redis/ 
integrations:
  - name: nri-flex
    env:
      CONFIG_FILE: ${config.path}
    config_template_path: /etc/flex-configs/redis.yml

In the above example, the redis.yml external file can contain container discovery variable placeholders, like ${discovery.ip} or ${discovery.port}.

Configure how the Agent executes your integrations

The properties of this section modify the way the Agent executes and interacts with the integrations, or the way the agent decorates the integrations' data.

integration_user

The integration commands run as the same user as the Agent (usually root or nri-agent). If due to permission restrictions an integration needs to run as another user, its name must be specified in the integration_user property.

Example:

integrations:
  - name: dbus-inventory
    exec: python my-dbus-script.py
    integration_user: dbus

interval (default: 30s)

The interval option sets the time between consecutive executions of an integration. The accepted format is an integer immediately followed by a time unit (s for seconds, m for minutes, h for hours).

The minimum accepted value is 15s. Any value lower than 15s will be automatically set to 15s.

Example:

integrations:
  - name: nri-nginx
    env:
      STATUS_URL: http://127.0.0.1/status
      STATUS_MODULE: discover
    interval: 20s

inventory_source (default: integration/ + value of the name property)

Any inventory item must be catalogued under a category/source taxonomy. By default, each integration inventory is stored as integration/ + name value (e.g. integration/nri-apache, integration/nri-mysql).

The inventory_source property allows you to override the default taxonomy of inventory data.

Example:

integrations:
  - name: nri-nginx
  - name: nri-apache
    exec:
      - /var/db/newrelic-infra/newrelic-integrations/bin/nri-apache
      - --inventory
    inventory_source: config/apache

In the above example, the nri-nginx inventory, if any, would be visible in the New Relic User Interface under the integration/nri-nginx source. The nri-apache inventory would be visible under config/apache.

labels

labels is a key-value map that allows extra metadata to be provided for the integration. The Agent will use those labels to decorate the metrics, events and inventory that it receives from a given integration instance.

Example:

integrations:
  - name: nri-apache
    inventory_source: config/apache
    labels:
      env: production
      role: load_balancer

In the above example, the Agent will decorate all the metrics and events from the nri-apache instance with the following fields:

  • label.env: production
  • label.role: load_balancer

Also, the following entries will be added to the integration inventory:

  • config/apache/labels/env: production
  • config/apache/labels/role: load_balancer

timeout (default: 120s)

If an integration has not returned any metric (or a heartbeat message as described below) before the time specified in the timeout value, the Agent will kill the integration process and restart it after the corresponding interval. The accepted format is an integer number immediately followed (without spaces) by a time unit (ms for milliseconds, s for seconds, m for minutes, h for hours).

If a zero (or negative) timeout value is provided, the integration can run forever without being killed by a timeout expiration.

For long-running integrations (integrations that keep running, periodically returning metrics/events/inventory), each time the integration submits a metrics/events/inventory payload, the timeout deadline is restarted. That means that long-running integrations must return a valid JSON payload in an interval that is lower than timeout.

Returning an empty JSON ({}) is interpreted as a heart-beat message that restarts the timeout, preventing the long-running integration from being killed, even if they don't have information to report.

The minimum accepted value is 100ms. Any provided value lower than 100ms will be automatically set to 100ms.

Example:

integrations:
  - name: nri-jmx
    arguments:
      JMX_HOST: jmx-host.localnet
      JMX_PORT: 7096
      COLLECTION_FILES: "/etc/newrelic-infra/integrations.d/jvm-metrics.yml"
    timeout: 30s

working_dir (default: Infrastructure Agent root directory)

working_dir sets the working directory of the command. If empty or unspecified, the Agent runs the command in the Infrastructure Agent's current directory.

Example:

integrations:
  - name: my-integration
    exec: /opt/integration/bin/integration
    working_dir: /opt/integration/scratch-zone

Use your old integrations

Before Agent v1.8.0, the integrations configuration was divided in two files:

  • A ...-definition.yml file, containing a map of tagged commands and arguments and an interval in seconds and a value for the inventory source.
  • The ...-config.yml file, containing references to the definition command and some configuration arguments.

This file structure and format is still supported in the agent, but you may want to migrate your old integrations to the new configuration files and take advantage of the features added from Agent version 1.8.0.

The following YAML shows a legacy integration configuration that is still supported:

integration_name: com.newrelic.apache

instances:
  - name: apache-server-metrics
    command: metrics
    arguments:
      status_url: http://127.0.0.1/server-status?auto
      remote_monitoring: true
    labels:
      env: production
      role: load_balancer
  - name: apache-server-inventory
    command: inventory
    arguments:
      remote_monitoring: true
    labels:
      env: production
      role: load_balancer

If you want to migrate your legacy integration configuration to the new integrations engine, you just need to perform two steps:

  • Rename the instances top-level section to integrations.
  • Remove the integration_name top-level section and add it to each integration entry. You are no longer required to keep a separate file for each integration type and you can group your legacy integration entries in the same file as other integrations.

The ported version of the previous apache example would be:

integrations:
  - name: apache-server-metrics
    integration_name: com.newrelic.apache
    command: metrics
    arguments:
      status_url: http://127.0.0.1/server-status?auto
      remote_monitoring: true
    labels:
      env: production
      role: load_balancer
  - name: apache-server-inventory
    integration_name: com.newrelic.apache
    command: inventory
    arguments:
      remote_monitoring: true
    labels:
      env: production
      role: load_balancer

Please note that because the legacy engine does not support hot reloading, you will need to restart the agent to remove the old integrations configuration (otherwise the old instances will coexist with the new ones).

Recommendations for learning more: