If you are using our Prometheus remote write integration in a high-availability (HA) configuration, you need to make sure your Prometheus servers aren't sending multiple copies of the same metrics to New Relic. This document describes how you can configure your remote write integration so that New Relic does not keep duplicated metrics.
Tip
For information on standard Prometheus remote write integration without using a high-availability configuration, see Set up your Prometheus remote write integration.
External labels
New Relic requires two external labels to deduplicate data from replicas in a high-availability configuration:
Label name | Description | Example value |
|---|---|---|
| A label whose value identifies the name of a high-availability cluster or group of Prometheus servers. |
|
| A label whose value identifies the unique replica sending this data. |
|
The remaining sections explain how labels work with Prometheus Operator and standalone Prometheus.
Prometheus Operator
These external labels are added by default if you use Prometheus Operator version 0.19.0 (or higher). This applies whether you use Prometheus Operator directly or via the helm chart.
The operator sets the value of the prometheus label (the one identifying a cluster) as <prometheus deployment namespace>/<prometheus deployment name>. For example, if your namespace for the Prometheus deployment is monitoring and the name of the deployment is prometheus-cluster1, the value is monitoring/prometheus-cluster1.
The operator sets the value of the prometheus_replica label as the name of the pod for each replica. This follows the format replica-<replica number>, where the number is the ordinal of that replica (for example, the first replica is named replica-1).
Tip
If you still see duplicate copies of replica data, make sure you do not have replicaExternalLabelName or prometheusExternalLabelName in your Prometheus spec or chart configuration because these overrides change the label name.
Standalone Prometheus
When deploying a Prometheus server directly, you need to add the external labels to the configuration file. Here are two different example configurations for replicas within the same high-availability cluster:
Replica 1 (prometheus.yml)
global: external_labels: prometheus: monitoring-cluster prometheus_replica: replica-1Replica 2 (prometheus.yml)
global: external_labels: prometheus: monitoring-cluster prometheus_replica: replica-2