On-host integrations: Newer configuration format

New Relic Infrastructure on-host integrations can use one of two types of configuration formats.

In December 2019, Infrastructure agent version 1.8.0 began supporting a newer configuration format that makes use of a single configuration file (instead of two separate files), and provides other improvements. This document will explain how this newer format works.

The older standard configuration format is supported by current Infrastructure agents.

For an introduction to configuration, see Config overview.

Configuration structure

An on-host integration's configuration YAML must have an integrations top-level section containing a YAML array, where each entry represents an integration and its configuration.

For each integration entry, only the name property is mandatory. The other properties are optional.

Here's an example configuration featuring two integrations: our built-in Docker integration, which requires no configuration, and our MySQL integration:

integrations:
  # New Relic integration that does not require any configuration
  - name: nri-docker
  # New Relic integration that gets its configuration from the environment
  - name: nri-mysql
    env:
      PORT: 3306
      USERNAME: newrelic
      PASSWORD: 123456789 # to hide this field, read the secrets management documentation
  # Any free-form integration executed via a user-provided command
  - name: my-own-integration
    exec: python /opt/integrations/my-script.py --host=127.0.0.1

You can have as many configuration YAML files as you want and can group your integration instances.

We recommend linting the YAML configuration files before using them to avoid formatting issues.

Each configuration YAML file can also contain discovery and variables top-level sections.

This configuration format does not require an agent restart. When saved, changes are detected and implemented immediately. This means that saving intermediate configuration changes may cause the integration to stop working.

List of configuration properties

This is a list of the general properties used to configure an integration. For more details about using these properties, including example values, see the documentation following the table.

Config Description
name Name of the integration. This is the only mandatory configuration property across all on-host integrations. If the exec field is not set it will also be the name of the integration executable.
exec Full path to the integration executable, plus arguments. It may be a single-line string or a string array. If left unspecified, the exec field defaults to the name field.
env YAML map containing the environment variables to be passed to the integration, where key is the environment variable name, and value is the variable value.
config Configuration that is written as an external file and the path that is passed to the integration with the CONFIG_PATH environment variable or the ${config.path} variable placeholder.
config_template_path Any external file whose path is passed to the integration with the CONFIG_PATH environment variable or the ${config.path} variable placeholder. Its usage allows applying discovery and secrets binding to any external configuration.
integration_user Name of the user who runs the integration.
interval Time between consecutive executions of the integration. It must be a number followed by a time unit (s, m or h), without spaces.
inventory_source Allows overriding the category and term of the inventory source.
labels Map with labels that decorate the data (metrics, events, inventory) reported by the integration.
timeout A number followed by a time unit (ms, s, m or h). An integration that hasn't responded in this time period is killed and restarted.
working_dir Working directory for the integration binary.

The remainder of this document describes config properties grouped by their functionality:

Select an integration to run

There are two properties to select which integration will run: name and exec.

The only mandatory property across all on-host integrations is name. The remaining properties specified in this document are optional.

Example:

integrations:
  - name: nri-docker
  - name: my-integration
    exec: /usr/local/bin/my-integration --metrics --inventory
name

The mandatory name property can work in two ways:

  • If the exec property is set: The name property only provides an identifier for the integration instance. This identifier is used in log messages and to provide a default inventory category/source in the form integration/<name> (for example, integration/nri-redis). This inventory path can be overridden with the inventory_source configuration option.

  • If the exec property is not set: The agent looks for (and executes) an executable with the name value in any of the following folders:

    • Linux:
      • /var/db/newrelic-infra/newrelic-integrations/bin
      • /var/db/newrelic-infra/newrelic-integrations
      • /var/db/newrelic-infra/custom-integrations/bin
      • /var/db/newrelic-infra/custom-integrations
    • Windows
      • C:\Program Files\New Relic\newrelic-infra\newrelic-integrations\bin
      • C:\Program Files\New Relic\newrelic-infra\newrelic-integrations

If there is no executable with this name in the above folders the agent logs an error and the integration is not executed.

In Windows, do not append the .exe extension to the name. The agent does this for you (for example, name: nri-mysql would look for nri-mysql.exe in the above folders).

exec

The exec optional property specifies the path, command, and command-line arguments of the integration to execute. When none of the path folders or arguments have spaces, it can be written in a single-line string:

- name: my-integration
  exec: /usr/bin/python /opt/integrations/my-script.py --host=127.0.0.1

If any of the path/arguments have spaces that are part of a single element, you can use a YAML array notation:

- name: my-integration
  exec:
    - C:\Program Files\My Integration\integration.exe
    - --host
    - 127.0.0.1
    - --port
    - 8080

The default working directory is the root directory of the agent configuration. It can be overridden with the working_dir property.

Pass configuration to the integration executable

Often integration executables need to receive a configuration to work properly (for example, hostname and port of the monitored system, user credentials, etc.).

The Infrastructure agent allows you to configure the integration commands in three ways (which you can combine):

  • Command-line arguments, passed in the exec property.
  • Environment variables, using the env property.
  • Configuration files, whose path needs to be passed through environment variables or command-line arguments (see the config) property.

Example:

integrations:
  - name: my-integration
    exec: /opt/path/bin/script --host 127.0.0.1 --port 8081
  - name: nri-mysql
    env:
      STATUS_URL: http://10.0.0.3/server-status?auto
      REMOTE_MONITORING: true
env

The env property allows you to set environment variables that are passed to the executable. It is a key-value map with the required variables.

Example:

integrations:
  - name: nri-postgresql
    env:
      DATABASE: postgres
      PORT: 6432
      COLLECTION_LIST: '["postgres"]'
      COLLECT_DB_LOCK_METRICS: false
      VERBOSE: 1

If you expect your integration to receive the configuration from the host's environment rather than specifying it explicitly in the configuration file, you need to set the required variables in the Infrastructure agent passthrough_environment global configuration property

config

This section describes various ways to pass configuration information to an integration.

Pass configuration file directly

Some integration commands may get their configuration from an external file. If your integration requires a configuration file, nothing prevents you from directly passing its path directly as a command-line argument or an environment variable. Here's an example using configuration of our Flex integration:

integrations:
  - name: nri-flex
    env:
      CONFIG_FILE: /etc/nri-flex/configs/http-service.yaml
  - name: other-integration
    exec: /opt/integration/integration -f /opt/integration/config.properties

The above example assumes that the http-service.yaml and config.properties files exist. We can see that the nri-flex integration is expecting the http-service.yaml complete path via the CONFIG_FILE environment variable and the other-integration expects the complete config.properties path after the -f command-line flag.

In the above example, it's necessary for the integration installer/configurator that the configuration files exist in the provided path and that the agent and integrations have read permissions on them.

Pass configuration through config section

If you prefer to keep your configuration file with the rest of the integration configuration, you can use the config section in the integration entry, which can contain a valid YAML object or just a multi-line string:

integrations:
  - name: nri-flex
    env:
      CONFIG_FILE: ${config.path}
    config:
      name: csvFileExample
      apis: 
        - name: csvFile
          file: /Users/hello/test.csv
  - name: other-integration
    exec: /opt/integration/integration -f ${config.path}
    config: |
      example.cfg.verbose=true
      example.cfg.logger=/var/logs/integration.log
      example.cfg.hostname=localhost
      example.cfg.port=9025

In the above examples, every time the nri-flex integration is executed, the agent creates a temporary file with the following contents:

name: csvFileExample
apis: 
  - name: csvFile
    file: /Users/hello/test.csv

The above YAML is only a configuration example for the nri-flex integration. The agent ignores its contents; instead, it creates a temporary file and replaces the ${config.path} variable placeholder with its path. When the integration completes execution the temporary file is removed.

Also, the agent creates another temporary file before executing the other-integration integration:

example.cfg.verbose=true
example.cfg.logger=/var/logs/integration.log
example.cfg.hostid=localhost
example.cfg.port=9025

It replaces the -f ${config.path} command-line placeholder with the temporary path of the written file.

By convention, if you do not place the ${config.path} variable in any command-line argument or environment variable value, the agent passes the path of the configuration file via the CONFIG_PATH environment variable:

# assuming that nri-example command is prepared to receive the configuration
# file via the CONFIG_PATH environment variable
integrations:
  - name: nri-example
    config:
      name: csvFileExample
      apis: 
        - name: csvFile
          file: /Users/hello/test.csv

Pass secrets and discovery through config section

The main benefit for using a config section instead of hardcoding the full path of an external file is that you can insert ${variable} placeholders to apply our auto-discovery feature and secrets management.

Here's an example followed by some explanations:

variables:
  my_credentials:
    vault:
      http:
        url: http://my.vault.host/v1/newengine/data/secret
        headers:
          X-Vault-Token: my-vault-token
discovery:
  docker:
    match:
      label.service: foo
integrations:
  - name: foo-monitor
    exec: /opt/foo/bin/monitor --config=${config.path}
    config: |
      foo.host=${discovery.ip} 
      foo.port=${discovery.port}
      foo.user=${my_credentials.user}
      foo.password=${my_credentials.password}

(For more details about the variables and discovery sections, please visit the discovery and secrets management documentation).

The above example relies on the following premises:

  • There is a Vault service that allows retrieving a JSON object formed by the user and password fields.
  • There may be a variable number of Docker containers labeled with service=foo, which are accessible from the agent host via a discoverable public IP and Port.
  • The user has configured the foo-monitor integration to monitor all the service=foo labeled containers, which share a common user and password. Each instance of the foo-monitor integration requires executing the /opt/foo/bin/monitor executable, passing the text configuration inside the config section via the --config=<path> command-line argument.

As example of workflow, imagine that the Vault invocation returns the following JSON:

{"user":"monitorer","password":"5up3r53cr3t!"}

At the moment of executing the foo-monitor integration, there are three running containers labeled with service=foo:

  1. ip: 10.0.0.3, port: 8080
  2. ip: 10.0.0.3, port: 8081
  3. ip: 10.0.0.3, port: 8082

The agent then creates the following three temporary files, using the contents of the config property as a template, but replacing the ${placeholders} by the acquired variables and discovered items (files' path is invented for the sake of simplicity):

  • First match (/tmp/123_discovered):

    foo.host=10.0.0.3 
    foo.port=8080
    foo.user=monitorer
    foo.password=5up3r53cr3t!
    
  • Second match (/tmp/456_discovered):

    foo.host=10.0.0.3 
    foo.port=8081
    foo.user=monitorer
    foo.password=5up3r53cr3t!
    
  • Third match (/tmp/789_discovered)

    foo.host=10.0.0.3 
    foo.port=8082
    foo.user=monitorer
    foo.password=5up3r53cr3t!
    

After the config variable placeholders have been replaced and the temporary files have been created, the /opt/foo/bin/monitor executable is executed three times (one per matched container), replacing the ${config.path} command-line placeholder with the temporary file corresponding to each discovered configuration:

  • First match: /opt/foo/bin/monitor --config=/tmp/123_discovered
  • Second match: /opt/foo/bin/monitor --config=/tmp/456_discovered
  • Third match: /opt/foo/bin/monitor --config=/tmp/789_discovered

To ensure security and to minimize the chance of leaking secrets to disk, the agent:

  • Creates the files owned by the agent user, for example, root or nri-agent, depending on the user you have configured to run the agent.
  • Sets read permissions only for the owner.
  • Removes the created files when the integration instance finishes its execution.
config_template_path

If you want to use the secrets management and discovery in the configuration files that you're passing to the integration executable, but you prefer to keep them as an individual file, you can use the config_template_path: <path> option. It works exactly as in the config section:

  1. The agent applies secrets management and discovery to the file contents.
  2. The agent creates different temporary files that are passed to the integration via the ${config.path} placeholder (or the CONFIG_PATH environment variable).

Example:

discovery:
    docker:
      match:
        name: /^redis/ 
  integrations:
    - name: nri-flex
      env:
        CONFIG_FILE: ${config.path}
      config_template_path: /etc/flex-configs/redis.yml
  

In the above example, the redis.yml external file can contain container discovery variable placeholders, like ${discovery.ip} or ${discovery.port}.

Configure how the agent executes your integrations

The properties of this section modify the way the Infrastructure agent executes and interacts with the integrations, or the way the agent decorates the integrations' data.

integration_user

The integration commands run as the same user as the agent (usually root or nri-agent). If due to permission restrictions an integration needs to run as another user, its name must be specified in the integration_user property.

Example:

integrations:
  - name: dbus-inventory
    exec: python my-dbus-script.py
    integration_user: dbus
interval

The interval option sets the time between consecutive executions of an integration. The accepted format is an integer immediately followed by a time unit (s for seconds, m for minutes, h for hours).

The default is 30s, and the minimum accepted value is 15s. Any value lower than 15s is automatically set to 15s.

Example:

integrations:
  - name: nri-nginx
    env:
      STATUS_URL: http://127.0.0.1/status
      STATUS_MODULE: discover
    interval: 20s
inventory_source

Any inventory item must be catalogued under a category/source taxonomy. By default, each integration inventory is stored as integration/ + name value (for example, integration/nri-apache, integration/nri-mysql).

The inventory_source property allows you to override the default taxonomy of inventory data.

Example:

integrations:
  - name: nri-nginx
  - name: nri-apache
    exec:
      - /var/db/newrelic-infra/newrelic-integrations/bin/nri-apache
      - --inventory
    inventory_source: config/apache

In the above example, the nri-nginx inventory, if any, would be visible in the New Relic UI under the integration/nri-nginx source. The nri-apache inventory would be visible under config/apache.

labels

labels is a key-value map that allows extra metadata to be provided for the integration. The agent uses those labels to decorate the metrics, events, and inventory that it receives from a given integration instance.

Example:

integrations:
  - name: nri-apache
    inventory_source: config/apache
    labels:
      env: production
      role: load_balancer

In the above example, the agent decorates all the metrics and events from the nri-apache instance with the following fields:

  • label.env: production
  • label.role: load_balancer

Also, the following entries are added to the integration inventory:

  • config/apache/labels/env: production
  • config/apache/labels/role: load_balancer
timeout

If an integration has not returned any metric (or a heartbeat message as described below) before the time specified in the timeout value, the agent kills the integration process and restarts it after the corresponding interval. The accepted format is an integer number immediately followed (without spaces) by a time unit (ms for milliseconds, s for seconds, m for minutes, h for hours).

If a zero (or negative) timeout value is provided, the integration can run forever without being killed by a timeout expiration.

For long-running integrations (integrations that keep running, periodically returning metrics/events/inventory), each time the integration submits a metrics/events/inventory payload, the timeout deadline is restarted. That means that long-running integrations must return a valid JSON payload in an interval that is lower than timeout.

Returning an empty JSON ({}) is interpreted as a heart-beat message that restarts the timeout, preventing the long-running integration from being killed, even if they don't have information to report.

The default is 120s, and the minimum accepted value is 100ms. Any value lower than 100ms is automatically set to 100ms.

Example:

integrations:
  - name: nri-jmx
    arguments:
      JMX_HOST: jmx-host.localnet
      JMX_PORT: 7096
      COLLECTION_FILES: "/etc/newrelic-infra/integrations.d/jvm-metrics.yml"
    timeout: 30s
working_dir

working_dir sets the working directory of the command. If empty or unspecified, the agent runs the command in the Infrastructure agent's current directory.

The default is the Infrastructure agent root directory.

Example:

integrations:
  - name: my-integration
    exec: /opt/integration/bin/integration
    working_dir: /opt/integration/scratch-zone

Update older integration configuration

In December 2019, the Infrastructure agent version 1.8.0 began using a different configuration format. For details, see Config format differences.

The main difference between these formats is that the older configuration format uses two separate configuration files (a INTEGRATION_NAME-definition.yml file and a INTEGRATION_NAME-config.yml file) and the newer version uses a single configuration file.

Here are some of the features added by the newer configuration functionality:

  • Flexible configuration via command-line arguments, environment variables, or external files.
  • Ability to group different integrations in the same file.
  • Hot reload: adding a new integration or changing its configuration does not require restarting the agent.
  • Timeouts: if an integration doesn't respond before a user-specified time, the integration process is killed and restarted.

Not all on-host integrations come with the newer configuration format, but you can update the configuration to the new format for all on-host integrations to take advantage of the new features.

The following YAML shows an example Apache integration configuration using the older configuration format. Note that this configuration will still work with newer agents, but we recommend updating your integrations to take full advantage of features.

integration_name: com.newrelic.apache

instances:
  - name: apache-server-metrics
    command: metrics
    arguments:
      status_url: http://127.0.0.1/server-status?auto
      remote_monitoring: true
    labels:
      env: production
      role: load_balancer
  - name: apache-server-inventory
    command: inventory
    arguments:
      remote_monitoring: true
    labels:
      env: production
      role: load_balancer

To update an older integration configuration to the new format, you must perform two steps:

  • Rename the instances top-level section to integrations.
  • Remove the integration_name top-level section and add it to each integration entry. You are no longer required to keep a separate file for each integration type and you can group your legacy integration entries in the same file as other integrations.

The new version of the Apache integration config:

integrations:
  - name: apache-server-metrics
    integration_name: com.newrelic.apache
    command: metrics
    arguments:
      status_url: http://127.0.0.1/server-status?auto
      remote_monitoring: true
    labels:
      env: production
      role: load_balancer
  - name: apache-server-inventory
    integration_name: com.newrelic.apache
    command: inventory
    arguments:
      remote_monitoring: true
    labels:
      env: production
      role: load_balancer

Please note that because the older configuration format doesn't support hot reloading. You will need to restart the Infrastructure agent to remove the old integrations configuration (otherwise the old instances will coexist with the new ones).

For more help