Prometheus High Availability (HA)

If you are using our Prometheus remote write integration in a high-availability (HA) configuration, you need to make sure your Prometheus servers aren't sending multiple copies of the same metrics to New Relic. This document describes how you can configure your remote write integration so that New Relic does not keep duplicated metrics.

For information on standard Prometheus remote write integration without using a high-availability configuration, see Set up your Prometheus remote write integration.

External labels

New Relic requires two external labels to deduplicate data from replicas in a high-availability configuration:

Label name Description Example value
prometheus A label whose value identifies the name of a high-availability cluster or group of Prometheus servers. monitoring-cluster
prometheus_replica A label whose value identifies the unique replica sending this data. replica-1

The remaining sections explain how labels work with Prometheus Operator and standalone Prometheus.

Prometheus Operator

These external labels are added by default if you use Prometheus Operator version 0.19.0 (or higher). This applies whether you use Prometheus Operator directly or via the helm chart.

The operator sets the value of the prometheus label (the one identifying a cluster) as <prometheus deployment namespace>/<prometheus deployment name>. For example, if your namespace for the Prometheus deployment is monitoring and the name of the deployment is prometheus-cluster1, the value is monitoring/prometheus-cluster1.

The operator sets the value of the prometheus_replica label as the name of the pod for each replica. This follows the format replica-<replica number>, where the number is the ordinal of that replica (for example, the first replica is named replica-1).

If you still see duplicate copies of replica data, make sure you do not have replicaExternalLabelName or prometheusExternalLabelName in your Prometheus spec or chart configuration because these overrides change the label name.

Standalone Prometheus

When deploying a Prometheus server directly, you need to add the external labels to the configuration file. Here are two different example configurations for replicas within the same high-availability cluster:

Replica 1 (prometheus.yml)

global:
	external_labels:
		prometheus: monitoring-cluster
		prometheus_replica: replica-1

Replica 2 (prometheus.yml)

global:
	external_labels:
		prometheus: monitoring-cluster
		prometheus_replica: replica-2

For more help

If you need more help, check out these support and learning resources: