New Relic Prometheus OpenMetrics integration (Docker)

The New Relic Prometheus OpenMetrics integration for Docker scrapes Prometheus endpoints and sends the data to New Relic. A static list of endpoints can be provided. The collected metrics can then be visualized in the Insights UI.

New Relic has contributed the Prometheus integration to the open source community under an Apache 2.0 license.

Requirements

This integration supports Prometheus protocol version 2. The integration was tested using Docker 1.9, 1.11 and 1.13 on kops, GKE and minikube, respectively. The following limits apply:

  • 50 attributes per metric
  • 50k unique timeseries (A timeseries is a single, unique combination of a metric name and any tags/attributes.)
  • 100k data points per minute (contact New Relic for a higher limit)
Prometheus OpenMetrics - Docker
Example workflow: Here is an example of the workflow using the New Relic Prometheus OpenMetrics integration for Docker.

Install the integration

To install the New Relic Prometheus OpenMetrics integration in a Docker environment:

  1. Create a configuration file config.yaml. We provide one you can use in the example configuration file section of this document, or you can look at the nri-prometheus-latest.yaml manifest file which includes the nri-prometheus-cfg config map showing an example configuration. Remember to change the cluster_name option.

  2. Optional: Specify which metrics you want to include. This can be controlled precisely according to the prefixes of the metrics and the labels. See the ignore metrics and include specific metrics in this document for more details.

  3. Start the integration in the background. Add your New Relic license key (required), and the endpoints to scrape (required). Add the http://localhost:8080/metrics endpoint to collect metrics about the integration itself.

    docker run -d --restart unless-stopped \
        --name nri-prometheus \
        -e LICENSE_KEY="YOUR_LICENSE_KEY" \
        -v "$(pwd)/config.yaml:/config.yaml" \
        newrelic/nri-prometheus:1.1
  4. Confirm the container is running properly::

    docker ps -f "name=nri-prometheus"
  5. To confirm that the integration has been configured correctly, wait a few minutes, then go to insights.newrelic.com, and run this NRQL query to see if data has been reported:

    FROM Metric SELECT count(*) WHERE clusterName = 'YOUR_CLUSTER_NAME' since 1 hour ago

Configuration options

License key

We recommend to configure the license key as an environment variable. This provides a more secure environment, as this environment variable can be loaded from a Kubernetes secret. The environment variable is named LICENSE_KEY and it is required.

To find your license key: Go to Account dropdown > Account settings. The license key appears in the Account information section on the right side of the Summary page. For more information, see License key.

General configuration

The nri-prometheus-latest.yaml manifest file includes the nri-prometheus-cfg config map showing an example configuration. It can be used to configure the following parameters:

Key name Description

cluster_name

Required.

The name of the Kubernetes cluster. This value will be included as the "clusterName" attribute for all the metrics.

verbose

Stringified boolean.

  • true (default): Logs debugging information.
  • false: Only logs error messages.
targets Configuration of static endpoints to be scraped by the integration. It contains a list of objects. For more information about this structure, see Target configuration.

scrape_duration

How often should the scraper run. Increasing this value will lower memory usage. Decreasing it has the opposite effect.

The impact on memory usage is due to distributing the target fetching over the scrape interval to avoid querying (and buffering) all the data at once.

Default is 30s. Valid values include 1s, 15s, 30s, 1m, 5m, etc.

scrape_timeout

The HTTP client timeout when fetching data from endpoints.

Default is 5s. Valid values include 1s, 15s, 30s, 1m, 5m, etc.

percentiles Histogram support is based on New Relic's guidelines for higher level metrics abstractions.
To better support visualization of this data, percentiles are calculated based on the histogram metrics and sent to New Relic.
Defaults are: 50, 95 and 99.
emitter_proxy

Proxy to use by the integration when submitting metrics. It should be in the format [scheme]://[domain]:[port].

This proxy won't be used when fetching metrics from the targets.

By default is empty, meaning that no proxy will be used.

emitter_ca_file

Certificate to add to the root CA that the emitter will use when verifying server certificates.

If left empty, TLS uses the host's root CA set

emitter_insecure_skip_verify

Wheter the emitter should skip TLS verification when submitting data.

Defaults to false.

Target configuration

If you want the target key in the configuration file to contain one or more objects, use the following structure in the YAML list:

Key name Description

description

A friendly description for the following URLs in this target.

urls A list of strings with the URLs to be scraped.

tls_config

Optional.

Authentication configuration used to send requests to all the configured. It supports TLS and Mutual TLS. For more information, see Mutual TLS authentication.

Example configuration file

The following is an example configuration file, you can save it and modify it to fit your needs. Read the sections for mutual TLS authentication and metric transformations for detailed information about their corresponding configuration options.

# The name of your cluster. It's important to match other New Relic products to relate the data.
cluster_name: "my-cluster-name"

# How often the integration should run. Defaults to 30s.
# scrape_duration: "30s"

# The HTTP client timeout when fetching data from endpoints. Defaults to 5s.
# scrape_timeout: "5s"

# Wether the integration should run in verbose mode or not. Defaults to false.
verbose: false

# Wether the integration should skip TLS verification or not. Defaults to false.
insecure_skip_verify: false

# The label used to identify k8s scrapable targets. Defaults to "prometheus.io/scrape".
scrape_enabled_label: "prometheus.io/scrape"

# Whether k8s nodes need to be labelled to be scraped or not. Defaults to true.
require_scrape_enabled_label_for_nodes: true

# targets:
# - description: Secure etcd example
# urls: ["https://192.168.3.1:2379", "https://192.168.3.2:2379", "https://192.168.3.3:2379"]
# tls_config:
# ca_file_path: "/etc/etcd/etcd-client-ca.crt"
# cert_file_path: "/etc/etcd/etcd-client.crt"
# key_file_path: "/etc/etcd/etcd-client.key"

# Proxy to be used by the emitters when submitting metrics. It should be
# in the format [scheme]://[domain]:[port].
# The emitter is the component in charge of sending the scraped metrics.
# This proxy won't be used when scraping metrics from the targets.
# By default it's empty, meaning that no proxy will be used.
# emitter_proxy: "http://localhost:8888"

# Certificate to add to the root CA that the emitter will use when
# verifying server certificates.
# If left empty, TLS uses the host's root CA set.
# emitter_ca_file: "/path/to/cert/server.pem"

# Whether the emitter should skip TLS verification when submitting data.
# Defaults to false.
# emitter_insecure_skip_verify: false

# Histogram support is based on New Relic's guidelines for higher
# level metrics abstractions https://github.com/newrelic/newrelic-exporter-specs/blob/master/Guidelines.md.
# To better support visualization of this data, percentiles are calculated
# based on the histogram metrics and sent to New Relic.
# By default, the following percentiles are calculated: 50, 95 and 99.
#
# percentiles:
# - 50
# - 95
# - 99

# transformations: 
#   - description: "Transformation for MySQL exporter" 
#     rename_attributes: 
#       - metric_prefix: "mysql_" 
#         attributes: 
#           table: "tableName" 
#     copy_attributes: 
#       - from_metric: "mysql_version_info" 
#         to_metrics: - "mysql_" 
#         attributes: 
#           - "innodb_version" 
#           - "version" 
#     ignore_metrics: 
#       - except: 
#         - "mysql_"

To run the integration with the previous configuration file, copy the content and save it to a file called config.yaml. Then, from within the same directory, run the the command:

docker run -d --restart unless-stopped \
    --name nri-prometheus \
    -e CLUSTER_NAME="YOUR_CLUSTER_NAME"
    -e LICENSE_KEY="YOUR_LICENSE_KEY" \
    -v "$(pwd)/config.yaml:/config.yaml" \
    newrelic/nri-prometheus:latest --configfile=/config.yaml

Configuration reload

Currently this integration does not automatically reload configuration when changes are done in the configuration file. To reload the configuration, restart the container running the integration:

docker restart nri-prometheus

View and query your metrics

You can query the collected metrics using NRQL in the Insights UI. All metrics are stored in the Metric type.

By default, the following attributes will be added to all the metrics:

  • clusterName: The name of the cluster provided in the scraper configuration.
  • scrapedEndpoint: The URL of the endpoint that is being scraped.
  • integrationName: The name of this integration (nri-prometheus).
  • integrationVersion: The version of the integration; for example, 0.2.0.
  • metricName: The name of the metric itself.
  • nrMetricType: The type of the New Relic Metric type (Gauges).
  • promMetricType: The metric type of the Prometheus metric.

When you build queries, be aware that there is no linking between the metrics, entities, and attributes. Use the following NRQL queries to find out which metrics are available and which attributes are present on these metrics:

Get metric names

To get all metric names:

FROM Metric SELECT uniques(nrMetricName)

To get metric names for a specific endpoint:

FROM Metric SELECT uniques(nrMetricName) WHERE scrapedEndpoint='<ep>'
Get the attributes for a metric

To get all attributes for the selected metric:

FROM Metric SELECT keyset() WHERE nrMetricName='<mn>'
Get the values for an attribute

The autocomplete will show all values of the attribute, regardless of the endpoint. To determine the attribute values for a specific endpoint:

FROM Metric SELECT uniques(<attribute>) WHERE metricName='<mn>' AND podName='<ep>'

Histograms and summaries

Histogram and summaries support was added in version 1.2.0 of The New Relic's Prometheus OpenMetrics integration, It is based on New Relic's guidelines for higher level metrics abstractions.

For summary types, quantiles are transformed into percentiles. A metric <basename>{quantile="0.3"} will be sent to New Relic as <basename>.percentiles and will have the dimention {percentile="30"}.

In terms of histograms, a bucket <basename>_bucket{le="42"} will be sent as <basename>_buckets and will have the dimention {histogram.bucket.upperBound="42"}.

To better support visualization of histograms, percentiles are calculated based on the histogram metrics and sent to New Relic. The calculated percentiles can be configured using the percentiles configuration option.

Build the query

Using the metric name and attributes retrieved, you can now query your data. For more information about facets, timeseries, and time selection, see the NRQL documentation.

Get raw metric values

To get raw metric values:

FROM Metric SELECT <metricname> WHERE <attribute>='<value>'.
Get a graph of the metric

To get a graph of the metric (possible aggregators are average, min, max, sum):

FROM Metric SELECT <aggregator>(<metricname>) WHERE <attribute>='<value>' TIMESERIES
Example: Average memory free

This example assumes you are scraping node exporters. To view average memory free for all scraped endpoints in a cluster:

FROM Metric SELECT average(node_memory_MemFree_bytes) WHERE clusterName='my-cluster'
Example: Connected Redis clients per endpoint

This example assumes you are scraping Redis exporters. To view the number of connected Redis clients per endpoint in a cluster:

FROM Metric SELECT latest(redis_connected_clients) WHERE clusterName='my-cluster' FACET scrapedEndpoint TIMESERIES
Query counter metrics

Currently the integration calculates the deltas for counter metrics. This is why queries on counter metrics will show the deltas of the counter instead of the absolute value of the counter.

Troubleshooting

Here are some troubleshooting tips when using the Prometheus OpenMetrics integration for Docker.

Basic troubleshooting

If you are having problems with the integration:

  1. Check if the Prometheus OpenMetrics integration is running:

    docker ps -f "name=nri-prometheus"
  2. Check the Status field for the container.

    docker inspect nri-prometheus
  3. For more detailed information, use Docker inspect.
No data appears in Insights

If no data appears in the the Insights UI:

  1. Run this NRQL query:

    docker logs nri-prometheus | grep "error emitting metrics"
    
  2. Check whether the log contains this:

    metrics api responded with status code 403

    If yes, check the LICENSE_KEY in your Docker command.

Get scraper logs

To get the scraper logs of the Prometheus OpenMetrics integration:

docker logs nri-prometheus
docker logs nri-prometheus 2>&1 | grep -v "level=debug"
Get scraper metrics

If you configure the http://localhost:8080/metrics endpoint, the integration will also scrape its own metrics. The following metrics can give some insight into what is happening inside the scraper:

  • nr_stats_targets_total: The total number of scraped endpoints.
  • nr_stats_targets_failed_fetch: The number of endpoints where scraping is failing.
  • nr_stats_metrics_total: The total number of scraped metrics.
  • nr_stats_metrics_total_timeseries: The total number of scraped time series.
View metric data sent to Metric API

To see the exact data that is being sent to the Metric API, set the EMITTERS environment variable to "api,stdout".

Update the integration

To update the integration, remove the Docker container, then follow standard installation procedures to start a new Docker container.

The integration logs its version when it starts up. To determine the running version:

docker logs nri-prometheus 2>&1 | grep "Integration version"

Example output:

time="2019-02-26T09:21:21Z" level=info msg="Starting New Relic's Prometheus OpenMetrics Integration version 1.0.0"

Metric transformations

The New Relic Prometheus OpenMetrics integration provides a few controls to transform the Prometheus metrics before sending them to New Relic. You can define these transformations in the integration config file. The transformation are performed for all endpoints.

The nri-prometheus-latest.yaml manifest file includes the nri-prometheus-cfg config map showing an example configuration. The transformations are executed in the following order:

  1. Ignore metrics.
  2. Add attributes.
  3. Rename attributes.
  4. Copy attributes.

Rename attributes

Not all Prometheus endpoints have consistent naming. You can rename the attributes as needed. For example:

Example: Configuration

To rename the table attribute to tableName and the under_score attribute to CamelCase for metrics that start with mysql_:

rename_attributes:
  - metric_prefix: "mysql_" 
    attributes:
      table: "tableName"
      under_score: "CamelCase"

Example: Input

mysql_info_schema_table_rows{schema="sys",table="host_summary"} 123 another_metric{table="first"} 800

Example: Output

mysql_info_schema_table_rows{schema="sys",tableName="host_summary"} 123 another_metric{table="first"} 800

Example: Copy attributes

Some Prometheus endpoints provide an _info or _static metric containing metadata about the service, such as the version. It can be helpful to have this attribute on all metrics for that service. This transformation allows you to copy attributes from a source metric to a set of target metrics.

You can only copy attributes between metrics in the same endpoint.

Example: Configuration

To copy the innodb_version and version attributes from the mysql_version_info metric to all metrics starting with mysql_:

copy_attributes:
  - from_metric: "mysql_version_info"
    to_metrics:
      - "mysql_" 
    attributes: 
      - "innodb_version"
      - "version"

Example: Input

# HELP mysql_version_info MySQL version and distribution. mysql_version_info{innodb_version="5.7.14",version="5.7.14",version_comment="MySQL Community Server (GPL)"} 1
# HELP mysql_global_variables_slave_transaction_retries Generic gauge metric from SHOW GLOBAL VARIABLES. mysql_global_variables_slave_transaction_retries 10

Example: Output

mysql_global_variables_slave_transaction_retries{innodb_version="5.7.14",version="5.7.14"} 10

Ignore metrics

You should try to avoid sending data that's not relevant to your monitoring needs. If you set the integration to scrape all the available targets and to sent all the data that's exposed from those targets, you might end up reaching the platform limits (which you can find in the requirements section) or increasing your bill charges.

Our recommended approach is to use the integration filtering capabilities to control the amount of data sent. Explore your data and refine your filters to scrape only relevant targets and send useful metrics.

To ignore unwanted metrics, use following transformation.

Example: Configuration

To drop all metrics that start with go_ or process_:

ignore_metrics:
  - prefixes:
    - "go_"
    - "process_"

Example: Input

go_goroutines 13
process_virtual_memory_bytes 2.062336e+07
mysql_global_status_commands_total{command="ha_close"} 0
mysql_global_status_commands_total{command="ha_open"} 0

This is taken from the MySQL exporter. Besides the MySQL metrics, it also exposes metrics about the exporter that might not be of interest to you.

Example: Output

mysql_global_status_commands_total{command="ha_close"} 0 
mysql_global_status_commands_total{command="ha_open"} 0 

Include only specific metrics

If you only want to include specific metrics, you can use the except list under the ignore_metrics section. As the name implies, this will ignore all the metrics except the ones that contain the with the given prefixes.

Example: Configuration

To drop all metrics except kube_hpa_:

ignore_metrics:
  - except:
    - kube_hpa_

Transformation configuration

This is the full configuration file containing all of these examples:

transformations:
 - description: "Transformation for MySQL exporter"
   rename_attributes:
     - metric_prefix: "mysql_"
       attributes:
         table: "tableName"
         under_score: "CamelCase"
   copy_attributes:
     - from_metric: "mysql_version_info"
       to_metrics:
         - "mysql_"
       attributes:
         - "innodb_version"
         - "version"
   ignore_metrics:
     - prefixes:
       - "go_"
       - "process_"

Set up the scraper container to use the configuration file (config.yaml in the current directory):

docker run -d --restart unless-stopped \
    --name nri-prometheus \
    -e CLUSTER_NAME="YOUR_CLUSTER_NAME"
    -e LICENSE_KEY="YOUR_LICENSE_KEY" \
    -v "$(pwd)/config.yaml:/config.yaml" \
    newrelic/nri-prometheus:latest --configfile=/config.yaml

Mutual TLS authentication

The New Relic Prometheus integration provides the ability to configure mutual TLS authentication for the endpoints that require this feature. This can be configured in the integration's config file.

Recommendation: Put the CA bundle and the key and cert files in a secret, and mount them in the Prometheus metrics integration's container.

MTLS authentication is limited to a static list of URLs. To configure endpoints that require MTLS authentication, follow this example:

targets:
- description: "Secure etcd example"
  urls: ["https://192.168.3.1:2379", "https://192.168.3.2:2379"]
  tls_config:
    ca_file_path: "/etc/etcd/etcd-client-ca.crt"
    cert_file_path: "/etc/etcd/etcd-client.crt"
    key_file_path: "/etc/etcd/etcd-client.key"
transformations:
  ...

Uninstall

To uninstall the integration, execute the following command:

docker rm -f nri-prometheus

For more help

Recommendations for learning more:

  • Browse New Relic's Explorers Hub for community discussions about New Relic's APIs.
  • Use your preferred search engine to find other New Relic resources.