You can get Prometheus data flowing in New Relic with just a few simple steps. This page covers basic setup for the remote write integration, as well as a few common troubleshooting topics. For information on integrating Prometheus servers in a high availability (HA) configuration, see our Prometheus high availability documentation.
To use Prometheus integrations and infrastructure monitoring, as well as the rest of our observability platform, join the New Relic family! Sign up to create your free account in only a few seconds. Then ingest up to 100GB of data for free each month. Forever.
Set up the integration
Go to the Prometheus remote write setup launcher in New Relic One, then complete these steps.
-
Enter a name for the Prometheus server to be connected and your
remote_write
URL. Important: The name you enter for the server will create an attribute on your data. It will also be the name that identifies which Prometheus server is sending data to New Relic. -
Add a new
remote_write
URL to your Prometheus YML file. Add this information underglobal_config
in the file, at the same indentation level as theglobal
section.Use the following syntax:
remote_write: - url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=YOUR_DATA_SOURCE_NAME bearer_token:YOUR_LICENSE_KEY
OR
remote_write: - url: https://metric-api.newrelic.com/prometheus/v1/write?X-License-Key=YOUR_LICENSE_KEY&prometheus_server=YOUR_DATA_SOURCE_NAME
European Union accounts: If you're connecting from the EU, use the following URL:
https://metric-api.eu.newrelic.com/prometheus/v1/write
Kubernetes and Helm remote write integrations: Add the remote write URL to your Helm
values.yaml
file. ReplaceremoteWrite: []
with two lines similar to the following example. Be sure to use your remote write URL and use indentation that matches the rest of the file:remoteWrite: - url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=YOUR_DATA_SOURCE_NAME bearerToken:YOUR_LICENSE_KEY
-
Restart your Prometheus server.
-
View your data in the New Relic UI. For example, use the remote write dashboard we automatically create when you set up your integration.
Mapping of Prometheus metric types
The Prometheus remote write protocol does not include metric type information or other helpful metric metadata when sending metrics to New Relic. Because the remote write protocol doesn't include this information, New Relic infers the metric type based on Prometheus naming conventions. Metrics not following these naming conventions may not be mapped correctly.
New Relic maps Prometheus metrics types into New Relic metric types based on Prometheus metric naming conventions as follows:
metricName_bucket
is stored as a New Relic count metric type.metricName_count
is stored as a New Relic count metric type.metricName_total
is stored as a New Relic count metric type.metricName_sum
is stored as a New Relic summary metric type.
Everything else is stored as a New Relic gauge metric type.
Override metric type mappings
If you have metrics that don't follow Prometheus naming conventions, you can configure remote-write to tag the metric with a newrelic_metric_type
label that indicates the metric type. This label is stripped when received by New Relic.
Example: You have a counter metric named my_counter
, which does not have our naming convention suffix of _bucket
, _count
or _total
. In this situation, your metric would be identified as a gauge rather than a counter. To correct this, add the following relabel configuration to your prometheus.yml
:
- url: https://metric-api.newrelic.com/prometheus/v1/write?X-License-Key=... write_relabel_configs: - source_labels: [__name__] regex: ^my_counter$ target_label: newrelic_metric_type replacement: "counter" action: replace
This rule matches any metric with the name my_counter
and adds a newrelic_metric_type
label that identifies it as a counter. You can use the following (case sensitive) values as the replacement value:
- counter
- gauge
- summary
When a newrelic_metric_type
label is present on a metric received and set to one of the valid values, New Relic will assign the indicated type to the metric (and strip the label) before downstream consumption in the data pipeline. If you have multiple metrics that don't follow the above naming conventions, you can add multiple rules with each rule matching different source labels.
Customize remote write behavior
You can customize the following parameters if you are writing to more than one account in New Relic or are connecting more than one Prometheus data source to the same account in New Relic. For more information, see the docs on remote write tuning.
- X-License Key
-
Your account's license key is not an API key. The license key is used for authentication and to identify which account to write data into. If you are configuring Prometheus to write into different New Relic accounts or sub-accounts, use a different key on each remote write URL.
prometheus_server
URL parameter-
The
prometheus_server
parameter is a label or attribute used to add to stats that are written to NRDB. Use this same label when configuring your Grafana data source to limit results to just those from a particularprometheus_server
. - Optimize throughput and memory consumption
-
Remote write increases the total memory consumption of your Prometheus servers.
If you're experiencing issues we recommend the following:
- Increase
max_samples_per_send
for higher throughput workloads, along a proportional increase incapacity
. - If memory consumption is still a problem, try limiting the number of
max_shards
per server.
- Increase
Troubleshoot error messages
If you receive an integration error message from New Relic or error messages in your Prometheus server logs after restarting your Prometheus server, review our remote write troubleshooting documentation. This includes fixing common errors, such as missing or incorrect characters, bad requests, request entity too large, and rate limit errors.
Remove the integration
When you remove the Prometheus remote write integration, this stops new data from flowing, but it will not purge or remove any historical data. To remove the integration, remove the configuration code snippet from your Prometheus YML file, then restart the server.