New Relic Prometheus OpenMetrics integration (Docker)

The New Relic Prometheus OpenMetrics integration for Docker scrapes Prometheus endpoints and sends the data to New Relic. A static list of endpoints can be provided through an environment variable. The collected metrics can then be visualized in the Insights UI.

New Relic has contributed the Prometheus integration to the open source community under an Apache 2.0 license.


This integration supports Prometheus protocol version 2. The integration was tested using Docker 1.9, 1.11 and 1.13 on kops, GKE and minikube, respectively. The following limits apply:

  • 50 prometheus endpoints
  • 50 attributes per metric
  • 50k unique timeseries (A timeseries is a single, unique combination of a metric name and any tags/attributes.)
  • 100k data points per minute (contact New Relic for a higher limit)
Prometheus OpenMetrics - Docker
Example workflow: Here is an example of the workflow using the New Relic Prometheus OpenMetrics integration for Docker.

Install the integration

To install the New Relic Prometheus OpenMetrics integration in a Docker environment:

  1. Start the integration in the background. Add your cluster name to identify the cluster used to group the data (required), New Relic license key (required), and the endpoints to scrape (required). Add the http://localhost:8080/metrics endpoint to collect metrics about the integration itself.

    $ docker run -d --restart unless-stopped \
        --name nri-prometheus \
        -e VERBOSE=true \
        -v "$(pwd)/config.yaml:/config.yaml" \
  2. Confirm the container is running properly::

    $ docker ps -f "name=nri-prometheus"
  3. To confirm that the integration has been configured correctly, wait a few minutes, then go to, and run this NRQL query to see if data has been reported:

    FROM MetricRaw SELECT count(*) WHERE clusterName = 'YOUR_CLUSTER_NAME' since 1 hour ago

Configuration options

License key

We recommend to configure the license key as an environment variable. This provides a more secure environment, as this environment variable can be loaded from a Kubernetes secret. The environment variable is named LICENSE_KEY and it is required.

To find your license key: Go to Account dropdown > Account settings. The license key appears in the Account information section on the right side of the Summary page. For more information, see License key.

General configuration

The nri-prometheus-latest.yaml manifest file includes the nri-prometheus-cfg config map showing an example configuration. It can be used to configure the following parameters:

Key name Description



The name of the Kubernetes cluster. This value will be included as the "clusterName" attribute for all the metrics.


Stringified boolean.

  • true (default): Logs debugging information.
  • false: Only logs error messages.
targets Configuration of static endpoints to be scraped by the integration. It contains a list of objects. For more information about this structure, see Target configuration.


String. The Integration will check if the Kubernetes pod and service are annotated or have a label with this value to decide if it has to be scraped.

Default: ""


How often should the scraper run. Increasing this value will lower memory usage. Decreasing it has the opposite effect.

The impact on memory usage is due to distributing the target fetching over the scrape interval to avoid querying (and buffering) all the data at once.

Default is 30s. Valid values include 1s, 15s, 30s, 1m, 5m, etc.


The HTTP client timeout when fetching data from endpoints.

Default is 30s. Valid values include 1s, 15s, 30s, 1m, 5m, etc.


Whether or not Kubernetes nodes need to be labeled to be scraped.

Default is false.

percentiles Histogram support is based on New Relic's guidelines for higher level metrics abstractions.
To better support visualization of this data, percentiles are calculated based on the histogram metrics and sent to New Relic.
Defaults are: 50, 95 and 99.

Proxy to use by the integration when submitting metrics. It should be in the format [scheme]://[domain]:[port].

This proxy won't be used when fetching metrics from the targets.

By default is empty, meaning that no proxy will be used.


Certificate to add to the root CA that the emitter will use when verifying server certificates.

If left empty, TLS uses the host's root CA set


Wheter the emitter should skip TLS verification when submitting data.

Defaults to false.

Target configuration

If you want the target key in the configuration file to contain one or more objects, use the following structure in the YAML list:

Key name Description


A friendly description for the following URLs in this target.

urls A list of strings with the URLs to be scraped.



Authentication configuration used to send requests to all the configured. It supports TLS and Mutual TLS. For more information, see Mutual TLS authentication.

View and query your metrics

The collected metrics can be queried using NRQL in the Insights UI. All metrics are stored in the MetricRaw type.

By default, the following attributes will be added to all the metrics:

  • clusterName: The name of the cluster provided in the scraper configuration.
  • scrapedEndpoint: The URL of the endpoint that is being scraped.
  • integrationName: The name of this integration (nri-prometheus).
  • integrationVersion: The version of the integration; for example, 0.2.0.
  • metricName: The name of the metric itself.
  • nrMetricType: The type of the New Relic Metric type (Gauges).
  • promMetricType: The metric type of the Prometheus metric.

When you build queries, be aware that there is no linking between the metrics, entities, and attributes. Use the following NRQL queries to find out which metrics are available and which attributes are present on these metrics:

Get metric names

To get all metric names:

FROM Metric SELECT uniques(nrMetricName)

To get metric names for a specific endpoint:

FROM Metric SELECT uniques(nrMetricName) WHERE scrapedEndpoint='<ep>'
Get the attributes for a metric

To get all attributes for the selected metric:

FROM Metric SELECT keyset() WHERE nrMetricName='<mn>'
Get the values for an attribute

The autocomplete will show all values of the attribute, regardless of the endpoint. To determine the attribute values for a specific endpoint:

FROM MetricRaw SELECT uniques(<attribute>) WHERE metricName='<mn>' AND podName='<ep>'

Histograms and summaries

Histogram and summaries support was added in version 1.2.0 of The New Relic's Prometheus OpenMetrics integration, It is based on New Relic's guidelines for higher level metrics abstractions.

For summary types, quantiles are transformed into percentiles. A metric <basename>{quantile="0.3"} will be sent to New Relic as <basename>.percentiles and will have the dimention {percentile="30"}.

In terms of histograms, a bucket <basename>_bucket{le="42"} will be sent as <basename>_buckets and will have the dimention {histogram.bucket.upperBound="42"}.

To better support visualization of histograms, percentiles are calculated based on the histogram metrics and sent to New Relic. The calculated percentiles can be configured using the percentiles configuration option.

Build the query

Using the metric name and attributes retrieved, you can now query your data. For more information about facets, timeseries, and time selection, see the NRQL documentation.

Get raw metric values

To get raw metric values:

FROM Metric SELECT <metricname> WHERE <attribute>='<value>'.
Get a graph of the metric

To get a graph of the metric (possible aggregators are average, min, max, sum):

FROM Metric SELECT <aggregator>(<metricname>) WHERE <attribute>='<value>' TIMESERIES
Example: Average memory free

This example assumes you are scraping node exporters. To view average memory free for all scraped endpoints in a cluster:

FROM Metric SELECT average(node_memory_MemFree_bytes) WHERE clusterName='my-cluster'
Example: Connected Redis clients per endpoint

This example assumes you are scraping Redis exporters. To view the number of connected Redis clients per endpoint in a cluster:

FROM Metric SELECT latest(redis_connected_clients) WHERE clusterName='my-cluster' FACET scrapedEndpoint TIMESERIES
Query counter metrics

Currently the integration calculates the deltas for counter metrics. This is why queries on counter metrics will show the deltas of the counter instead of the absolute value of the counter.


Here are some troubleshooting tips when using the Prometheus OpenMetrics integration for Docker.

Basic troubleshooting

If you are having problems with the integration:

  1. Check if the Prometheus OpenMetrics integration is running:

    $ docker ps -f "name=nri-prometheus"
  2. Check the Status field for the container.

    $ docker inspect nri-prometheus
  3. For more detailed information, use Docker inspect.
No data appears in Insights

If no data appears in the the Insights UI:

  1. Run this NRQL query:

    $ docker logs nri-prometheus | grep "error emitting metrics"
  2. Check whether the log contains this:

    metrics api responded with status code 403

    If yes, check the LICENSE_KEY in your Docker command.

Get scraper logs

To get the scraper logs of the Prometheus OpenMetrics integration:

$ docker logs nri-prometheus
$ docker logs nri-prometheus 2>&1 | grep -v "level=debug"
Get scraper metrics

If you configure the http://localhost:8080/metrics endpoint, the integration will also scrape its own metrics. The following metrics can give some insight into what is happening inside the scraper:

  • nr_stats_targets_total: The total number of scraped endpoints.
  • nr_stats_targets_failed_fetch: The number of endpoints where scraping is failing.
  • nr_stats_metrics_total: The total number of scraped metrics.
  • nr_stats_metrics_total_timeseries: The total number of scraped time series.
View metric data sent to Metric API

To see the exact data that is being sent to the Metric API, set the EMITTERS environment variable to "api,stdout".

Update the integration

To update the integration, remove the Docker container, then follow standard installation procedures to start a new Docker container.

The integration logs its version when it starts up. To determine the running version:

$ docker logs nri-prometheus 2>&1 | grep "Integration version"

Example output:

time="2019-02-26T09:21:21Z" level=info msg="Starting New Relic's Prometheus OpenMetrics Integration version 1.0.0"

Metric transformations

The New Relic Prometheus OpenMetrics integration provides a few controls to transform the Prometheus metrics before sending them to New Relic. You can define these transformations in the integration config file. The transformation are performed for all endpoints.

The nri-prometheus-latest.yaml manifest file includes the nri-prometheus-cfg config map showing an example configuration. The transformations are executed in the following order:

  1. Ignore metrics.
  2. Add attributes.
  3. Rename attributes.
  4. Copy attributes.

Rename attributes

Not all Prometheus endpoints have consistent naming. You can rename the attributes as needed. For example:

Example: Configuration

To rename the table attribute to tableName and the under_score attribute to CamelCase for metrics that start with mysql_:

  - metric_prefix: "mysql_" 
      table: "tableName"
      under_score: "CamelCase"

Example: Input

mysql_info_schema_table_rows{schema="sys",table="host_summary"} 123 another_metric{table="first"} 800

Example: Output

mysql_info_schema_table_rows{schema="sys",tableName="host_summary"} 123 another_metric{table="first"} 800

Example: Copy attributes

Some Prometheus endpoints provide an _info or _static metric containing metadata about the service, such as the version. It can be helpful to have this attribute on all metrics for that service. This transformation allows you to copy attributes from a source metric to a set of target metrics.

You can only copy attributes between metrics in the same endpoint.

Example: Configuration

To copy the innodb_version and version attributes from the mysql_version_info metric to all metrics starting with mysql_:

  - from_metric: "mysql_version_info"
      - "mysql_" 
      - "innodb_version"
      - "version"

Example: Input

# HELP mysql_version_info MySQL version and distribution. mysql_version_info{innodb_version="5.7.14",version="5.7.14",version_comment="MySQL Community Server (GPL)"} 1
# HELP mysql_global_variables_slave_transaction_retries Generic gauge metric from SHOW GLOBAL VARIABLES. mysql_global_variables_slave_transaction_retries 10

Example: Output

mysql_global_variables_slave_transaction_retries{innodb_version="5.7.14",version="5.7.14"} 10

Ignore metrics

Some endpoints contain metric that are generating a lot of noise or are just confusing. To ignore these metrics, use following transformation.

Example: Configuration

To drop all metrics that start with go_ or process_:

  - prefixes:
    - "go_"
    - "process_"

Example: Input

go_goroutines 13
process_virtual_memory_bytes 2.062336e+07
mysql_global_status_commands_total{command="ha_close"} 0
mysql_global_status_commands_total{command="ha_open"} 0

This is taken from the MySQL exporter. Besides the MySQL metrics, it also exposes metrics about the exporter that might not be of interest to you.

Example: Output

mysql_global_status_commands_total{command="ha_close"} 0 
mysql_global_status_commands_total{command="ha_open"} 0 

Transformation configuration

This is the full configuration file containing all of these examples:

 - description: "Transformation for MySQL exporter"
     - metric_prefix: "mysql_"
         table: "tableName"
         under_score: "CamelCase"
     - from_metric: "mysql_version_info"
         - "mysql_"
         - "innodb_version"
         - "version"
     - prefixes:
       - "go_"
       - "process_"

Set up the scraper container to use the configuration file (config.yaml in the current directory):

$ docker run -d --restart unless-stopped \
    --name nri-prometheus \
    -v "$(pwd)/config.yaml:/config.yaml" \
    -e VERBOSE=true \
    newrelic/nri-prometheus:latest --configfile=/config.yaml

Mutual TLS authentication

The New Relic Prometheus integration provides the ability to configure mutual TLS authentication for the endpoints that require this feature. This can be configured in the integration's config file.

Recommendation: Put the CA bundle and the key and cert files in a secret, and mount them in the Prometheus metrics integration's container.

MTLS authentication is limited to a static list of URLs. To configure endpoints that require MTLS authentication, follow this example:

- description: "Secure etcd example"
  urls: ["", ""]
    ca_file_path: "/etc/etcd/etcd-client-ca.crt"
    cert_file_path: "/etc/etcd/etcd-client.crt"
    key_file_path: "/etc/etcd/etcd-client.key"


To uninstall the integration, execute the following command:

$ docker rm -f nri-prometheus

For more help

Recommendations for learning more:

  • Browse New Relic's Explorers Hub for community discussions about New Relic's APIs.
  • Use your preferred search engine to find other New Relic resources.