Add, rename, or copy Prometheus attributes

The New Relic Prometheus OpenMetrics integration provides controls to transform the Prometheus metrics for Docker and Kubernetes before sending them to New Relic. After you define the transformations in the integration config file, they are performed for all endpoints.

Hierarchy

The nri-prometheus-latest.yaml manifest file includes the nri-prometheus-cfg config map showing an example configuration. The transformations are executed in the following order:

1. Ignore metrics.
2. Add or include attributes.
3. Rename attributes.
4. Copy attributes.

Example configuration

To use these options, set up the scraper container configuration file (config.yaml in the current directory):

docker run -d --restart unless-stopped \
    --name nri-prometheus \
    -e CLUSTER_NAME="YOUR_CLUSTER_NAME"
    -e LICENSE_KEY="YOUR_LICENSE_KEY" \
    -v "$(pwd)/config.yaml:/config.yaml" \
    newrelic/nri-prometheus:latest --configfile=/config.yaml

Here is an example configuration file containing all of these examples:

transformations:
 - description: "Transformation for MySQL exporter"
   add_attributes:
     - metric_prefix: "mysql_"
       attributes:
         owningTeam: "database-team"
   rename_attributes:
     - metric_prefix: "mysql_"
       attributes:
         table: "tableName"
         under_score: "CamelCase"
   copy_attributes:
     - from_metric: "mysql_version_info"
       to_metrics:
         - "mysql_"
       attributes:
         - "innodb_version"
         - "version"
   ignore_metrics:
     - prefixes:
       - "go_"
       - "process_"

Add attributes

This transformation allows you to include a set of statically defined attributes to a set of target metrics.

Add attributes

Configuration:

To include the owningTeam attribute to all metrics starting with mysql_:

add_attributes:
  - prefix: "mysql_"
    attributes: 
      owningTeam: "database-team"

To include the datacenter attribute to all the metrics:

add_attributes:
  - prefix: ""
    attributes: 
      datacenter: "europe"

Input:

mysql_info_schema_table_rows{schema="sys",table="host_summary"} 123 another_metric{table="first"} 800

Output:

mysql_info_schema_table_rows{schema="sys",table="host_summary","owningTeam":"database-team","datacenter":"europe"} 123 another_metric{table="first","datacenter":"europe"} 800

Rename attributes

Not all Prometheus endpoints have consistent naming. You can rename the attributes as needed.

Rename attributes

Configuration:

To rename the table attribute to tableName for metrics that start with mysql_:

rename_attributes:
  - metric_prefix: "mysql_" 
    attributes:
      table: "tableName"

Input:

mysql_info_schema_table_rows{schema="sys",table="host_summary"} 123 another_metric{table="first"} 800

Output:

mysql_info_schema_table_rows{schema="sys",tableName="host_summary"} 123 another_metric{table="first"} 800

Copy attributes

Some Prometheus endpoints provide an _info or _static metric containing metadata about the service, such as the version. It can be helpful to have this attribute on all metrics for that service. This transformation allows you to copy attributes from a source metric to a set of target metrics.

Copy attributes

Configuration:

To copy the innodb_version and version attributes from the mysql_version_info metric to all metrics starting with mysql_:

copy_attributes:
  - from_metric: "mysql_version_info"
    to_metrics:
      - "mysql_" 
    attributes: 
      - "innodb_version"
      - "version"

Input:

# HELP mysql_version_info MySQL version and distribution. mysql_version_info{innodb_version="5.7.14",version="5.7.14",version_comment="MySQL Community Server (GPL)"} 1

# HELP mysql_global_variables_slave_transaction_retries Generic gauge metric from SHOW GLOBAL VARIABLES. mysql_global_variables_slave_transaction_retries 10

Output:

mysql_global_variables_slave_transaction_retries{innodb_version="5.7.14",version="5.7.14"} 10

For more help