Configure Prometheus OpenMetrics integrations

Unless otherwise noted, configuration options for your Prometheus OpenMetrics integration with New Relic apply to both Docker and Kubernetes environments. At a minimum, the following configuration values are required:

Recommendation: Configure your New Relic license key as an environment variable named LICENSE_KEY. This provides a more secure environment, as New Relic can load your environment variable from a mutual TLS authentication secret.

Configure nri-prometheus-latest.yaml

The nri-prometheus-latest.yaml manifest file includes the nri-prometheus-cfg map showing an example configuration. Use the manifest file to configure the following parameters.

Example configuration file

The following is an example configuration file that you can save and modify to fit your needs. For more information, see the documentation about mutual TLS authentication and translating PromQL to NRQL.

# The name of your cluster. It's important to match other New Relic products to relate the data.
cluster_name: "my-cluster-name"

# How often the integration should run. Defaults to 30s.
# scrape_duration: "30s"

# The HTTP client timeout when fetching data from endpoints. Defaults to 5s.
# scrape_timeout: "5s"

# Whether the integration should run in verbose mode or not. Defaults to false.
verbose: false

# Whether the integration should skip TLS verification or not. Defaults to false.
insecure_skip_verify: false

# The label used to identify scrapeable targets. Defaults to "prometheus.io/scrape".
scrape_enabled_label: "prometheus.io/scrape"

# Whether nodes need to be labeled to be scraped or not. Defaults to true.
require_scrape_enabled_label_for_nodes: true

# targets:
# - description: "Secure etcd example"
#   urls: ["https://123.456.7.1:2379", "https://123.456.7.2:2379"]
#   tls_config:
#     ca_file_path: "/etc/etcd/etcd-client-ca.crt"
#     cert_file_path: "/etc/etcd/etcd-client.crt"
#     key_file_path: "/etc/etcd/etcd-client.key"

# Proxy to be used by the emitters when submitting metrics. It should be
# in the format [scheme]://[domain]:[port].
# The emitter is the component in charge of sending the scraped metrics.
# This proxy won't be used when scraping metrics from the targets.
# By default it's empty, meaning that no proxy will be used.
# emitter_proxy: "http://localhost:8888"

# Certificate to add to the root CA that the emitter will use when
# verifying server certificates.
# If left empty, TLS uses the host's root CA set.
# emitter_ca_file: "/path/to/cert/server.pem"

# Whether the emitter should skip TLS verification when submitting data.
# Defaults to false.
# emitter_insecure_skip_verify: false

# Set to true in order to disable autodiscovery in the k8s cluster. 
# It can be useful when running the Pod with a service account
# having limited privileges. Defaults to false.
# disable_autodiscovery: false

# Histogram support is based on New Relic's guidelines for higher
# level metrics abstractions https://github.com/newrelic/newrelic-exporter-specs/blob/master/Guidelines.md.
# To better support visualization of this data, percentiles are calculated
# based on the histogram metrics and sent to New Relic.
# By default, the following percentiles are calculated: 50, 95 and 99.
#
# percentiles:
# - 50
# - 95
# - 99

# transformations: 
#   - description: "Transformation for MySQL exporter" 
#     rename_attributes: 
#       - metric_prefix: "mysql_" 
#         attributes: 
#           table: "tableName" 
#     copy_attributes: 
#       - from_metric: "mysql_version_info" 
#         to_metrics: - "mysql_" 
#         attributes: 
#           - "innodb_version" 
#           - "version" 
#     ignore_metrics: 
#       - except: 
#         - "mysql_"
Key names and definitions

Here are some key names and definitions for your Prometheus OpenMetrics config file.

Key name Description

cluster_name

Required.

The name of the cluster. This value will be included as the clusterName attribute for all metrics.

verbose

Stringified boolean.

  • true (default): Logs debugging information.
  • false: Only logs error messages.

targets

Configuration of static endpoints to be scraped by the integration. It contains a list of objects. For more information about this structure, see the documentation about target configuration.

scrape_enabled_label

img-integration-k8s@2x.png Kubernetes

String. The integration will check if the Kubernetes pod and service are annotated or have a label with this value to decide if it has to be scraped.

This is particularly useful when you want to limit the amount of data by ignoring metrics or including specific metrics that are sent to New Relic. Since by default we use the same label Prometheus uses to discover targets that can be scraped, most exporters that you install automatically set this label.

To keep a fine-grained control on the targets you want the integration to scrape, you can set this option to some other value (such as newrelic/scrape) and then add the annotation or label newrelic/scrape: "true" to your Kubernetes objects. If both are set, annotations take precedence over labels.

Default: "prometheus.io/scrape"

scrape_duration

How often should the scraper run.

  • To lower memory usage, increase this value.
  • To raise memory usage, decrease this value.

The impact on memory usage is due to distributing target fetching over the scrape interval to avoid querying (and buffering) all the data at once.

Default is 30s. Valid values include 1s, 15s, 30s, 1m, 5m, etc.

scrape_timeout

The HTTP client timeout when fetching data from endpoints.

Default: 5s. Valid values include 1s, 15s, 30s, 1m, 5m, etc.

require_scrape_enabled_label_for_nodes

img-integration-k8s@2x.png Kubernetes

Whether or not Kubernetes nodes need labels to be scraped.

Default: true.

percentiles

Histogram support is based on New Relic's guidelines for higher level metrics abstractions.

To better support visualization of this data, percentiles are calculated based on the histogram metrics and sent to New Relic. Valid values include 50, 95, and 99.

emitter_proxy

Proxy used by the integration when submitting metrics:

[scheme]://[domain]:[port]

This proxy won't be used when fetching metrics from the targets.

By default this is empty, and no proxy will be used.

emitter_ca_file

Certificate to add to the root CA that the emitter will use when verifying server certificates. If left empty, TLS uses the host's root CA set.

emitter_insecure_skip_verify

Whether the emitter should skip TLS verification when submitting data. Default: false.

disable_autodiscovery

Set to true in order to disable autodiscovery in the k8s cluster. It can be useful when running the Pod with a service account having limited privileges. Default: false.

Configure objects in target key

If you want the target key in the configuration file to contain one or more objects, use the following structure in the YAML list:

Key name Description

description

A description for the URLs in this target.

urls

A list of strings with the URLs to be scraped.

tls_config

Authentication configuration used to send requests. It supports TLS and Mutual TLS. For more information, see the documentation about mutual TLS authentication.

img-integration-k8s@2x.png Kubernetes port and endpoint path

New Relic's Prometheus OpenMetrics integration automatically discovers which targets to scrape. To specify the port and endpoint path to be used when constructing the target, you can use the prometheus.io/port and prometheus.io/path annotations or label in your Kubernetes pods and services. Annotations take precedence over labels.

  • If prometheus.io/port is not present, the integration will try to scrape each port or ContainerPort defined for the service.
  • If prometheus.io/path is not present, the integration will default to /metrics.
  • If a service is not running on the default /my-metrics-path path, add a label to the pod prometheus.io/path=my-metrics-path. If the path to the metrics endpoint is more complex and cannot be a valid label value (for example, foo/bar), use annotations instead.

img-integration-k8s@2x.png Example: Labels for Kubernetes port and path

In this example, you have a deployment in your cluster, and the pods expose Prometheus metrics on port 8080 and in the path my-metrics.

In the PodSpec metadata of the deployment manifest, set the labels prometheus.io/port: "8080" and prometheus.io/path: "my-metrics". When the integration tries to retrieve the metrics from your pods, it will send a request to http://<pod-ip>:8080/my-metrics.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
        prometheus.io/scrape: "true" 
        prometheus.io/port: "8080" 
        prometheus.io/path: "my-metrics"

Reload the configuration

The Prometheus OpenMetrics integration does not automatically reload the configuration when you make changes to the configuration file.

Docker icon Docker:

To reload the configuration, restart the container running the integration:

docker restart nri-prometheus

img-integration-k8s@2x.png Kubernetes:

To reload the configuration, restart the integration. Recommendation: Scale the deployment down to zero replicas, and then scale it back to one replica:

kubectl scale deployment nri-prometheus --replicas=0
kubectl scale deployment nri-prometheus --replicas=1

Docker: Run previous config file

Docker icon Docker: To run the integration with the previous configuration file:

  1. Copy the content and save it to a config.yaml file.
  2. From within the same directory, run the command:

    docker run -d --restart unless-stopped \
        --name nri-prometheus \
        -e CLUSTER_NAME="YOUR_CLUSTER_NAME" \
        -e LICENSE_KEY="YOUR_LICENSE_KEY" \
        -v "$(pwd)/config.yaml:/config.yaml" \
        newrelic/nri-prometheus:latest --configfile=/config.yaml

For more help

If you need more help, check out these support and learning resources: