--- title: Set up your Prometheus remote write integration source: https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/install-configure-remote-write/set-your-prometheus-remote-write-integration --- You can get Prometheus data flowing in New Relic with just a few simple steps. Once you integrate, your data will be visible in query-based dashboards (and other query results), often within about five minutes. This page covers basic setup for the remote write integration, as well as a few common troubleshooting topics. For information on integrating Prometheus servers in a high availability (HA) configuration, see our [Prometheus high availability](https://docs.newrelic.com/docs/prometheus-high-availability-ha) documentation. ### (Optional) Prometheus Operator configuration If you're using the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator), you'll need to create a secret with the New Relic license key for the account to which you want to report data. Make sure the API key is of the type `Ingest - License`. ```shell kubectl -n YOUR_PROM_NAMESPACE create secret generic nr-license-key --from-literal=value=YOUR_LICENSE_KEY ``` Next, add the following to your Prometheus CRD (`kind:Prometheus`) in the corresponding field from the [Helm chart](https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml): ```yaml prometheus: prometheusSpec: remoteWrite: - url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=YOUR_CLUSTER_NAME authorization: credentials: key: value name: nr-license-key ``` ## Set up the integration [#setup] Go to the [Prometheus remote write setup launcher in the UI](https://one.newrelic.com/marketplace/install-data-source?state=c8c296ea-4be7-405a-eb38-53215c68d0bb), and complete these steps to add Prometheus data. [Add Prometheus data](https://one.newrelic.com/marketplace/install-data-source?state=c8c296ea-4be7-405a-eb38-53215c68d0bb) 1. Enter a name for the Prometheus server to be connected and your `remote_write` URL. > #### ⚠️ IMPORTANT > > The name you enter for the server creates an attribute on your data. This is also the name that identifies which Prometheus server is sending data to New Relic. 2. Add a new `remote_write` URL to your Prometheus YML file. Add this information under `global_config` in the file, at the same indentation level as the `global` section. Use the following syntax: Prometheus v2.26 and higher ```yaml remote_write: - url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=YOUR_DATA_SOURCE_NAME authorization: credentials: YOUR_LICENSE_KEY ``` Prometheus lower than v2.26 ```yaml remote_write: - url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=YOUR_DATA_SOURCE_NAME bearer_token: YOUR_LICENSE_KEY ``` OR Any Prometheus version ```yaml remote_write: - url: https://metric-api.newrelic.com/prometheus/v1/write?X-License-Key=YOUR_LICENSE_KEY&prometheus_server=YOUR_DATA_SOURCE_NAME ``` _This approach passes credentials in the URL. We don't recommend using it unless one of these other approaches doesn't work in your environment._ EU or JP accounts: If you're connecting from the EU or JP data center, use the appropriate URL: - EU: `https://metric-api.eu.newrelic.com/prometheus/v1/write` - JP: `https://metric-api.jp.nr-data.net/prometheus/v1/write` Kubernetes and Helm remote write integrations: Add the remote write URL to your Helm [`values.yaml`](https://github.com/helm/charts/blob/c40486ab2eba0391119b7cc1509d6958fd21c31d/stable/prometheus/values.yaml#L631) file. Replace `remoteWrite: []` with two lines similar to the following example. Be sure to use your remote write URL and use indentation that matches the rest of the file: ```yaml remoteWrite: - url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=YOUR_DATA_SOURCE_NAME bearer_token: YOUR_LICENSE_KEY ``` 3. Restart your Prometheus server. 4. View your data in the New Relic UI. For example, use the remote write [dashboard](https://docs.newrelic.com/docs/query-your-data/explore-query-data/dashboards/introduction-dashboards) we automatically create when you set up your integration. ## Map Prometheus and New Relic metric types [#mapping] The Prometheus remote write protocol doesn't include [metric type](https://docs.newrelic.com/docs/telemetry-data-platform/ingest-manage-data/understand-data/metric-data-type#metric-types) information or other helpful metric metadata when sending metrics to New Relic, so we infer the metric type based on Prometheus naming conventions. Metrics not following these naming conventions may not be mapped correctly. We map Prometheus metrics types into New Relic metric types based on Prometheus metric naming conventions as follows: - `metricName_bucket` is stored as a New Relic count metric type. - `metricName_count` is stored as a New Relic count metric type. - `metricName_total` is stored as a New Relic count metric type. - `metricName_sum` is stored as a New Relic summary metric type. Everything else is stored as a New Relic gauge metric type. ## Override metric type mappings [#override-mapping] If you have metrics that don't follow Prometheus naming conventions, you can configure remote write to tag the metric with a `newrelic_metric_type` label that indicates the metric type. This label is stripped when received by New Relic. **Example:** You have a counter metric named `my_counter`, which does not have our naming convention suffix of `_bucket`, `_count` or `_total`. In this situation, your metric would be identified as a gauge rather than a counter. To correct this, add the following relabel configuration to your `prometheus.yml`: ```yaml - url: https://metric-api.newrelic.com/prometheus/v1/write?X-License-Key=... write_relabel_configs: - source_labels: [__name__] regex: ^my_counter$ target_label: newrelic_metric_type replacement: "counter" action: replace ``` This rule matches any metric with the name `my_counter` and adds a `newrelic_metric_type` label that identifies it as a counter. You can use the following (case sensitive!) values as the replacement value: - `counter` - `gauge` - `summary` When a `newrelic_metric_type` label is present on a metric received and set to one of the valid values, New Relic will assign the indicated type to the metric (and strip the label) before downstream consumption in the data pipeline. If you have multiple metrics that don't follow the above naming conventions, you can add multiple rules with each rule matching different source labels. ## Set allow or deny lists for sent metrics [#allow-deny] If you need greater control over the data you send to New Relic, you can send a subset of your metrics. To do this, configure `remote-write` with the `write_relabel_configs` parameter with a subparameter `action` value of `keep` or `deny`. In this example, you'll only send the metrics that match the regular expression. Unmatched metrics won't be sent. Alternatively, you can use `action: drop` to drop all of the metrics that match the regular expression. ```yaml - url: https://metric-api.newrelic.com/prometheus/v1/write?X-License-Key=... write_relabel_configs: - source_labels: [__name__] regex: "coredns_(.*)|etcd_(.*)" action: keep ``` This Kubernetes example uses this Helm chart's [`values.yaml`](https://github.com/helm/charts/blob/c40486ab2eba0391119b7cc1509d6958fd21c31d/stable/prometheus/values.yaml#L631) file. If you're using a different Helm chart, please check its `remoteWrite` documentation (for example, some Helm files use camelcase `writeRelabelConfigs` instead). ```yaml remoteWrite: - url: https://metric-api.newrelic.com/prometheus/v1/write?X-License-Key=... write_relabel_configs: - source_labels: [__name__] regex: "coredns_(.*)|etcd_(.*)" action: keep ``` ## Customize remote write behavior [#customize] You can customize the following parameters if you are writing to more than one account in New Relic or are connecting more than one Prometheus data source to the same account in New Relic. For more information, see the [docs on remote write tuning](https://prometheus.io/docs/practices/remote_write/). **X-License Key** Your license key is not an API key. The license key is used for authentication and to identify which account to write data into. If you are configuring Prometheus to write into different New Relic accounts, use a different key on each remote write URL. **`prometheus_server` URL parameter** The `prometheus_server` parameter is a label or attribute used to add to stats that are written to [NRDB](https://newrelic.com/resources/ebooks/inside-nrdb-flexible-unified-database). Use this same label when [configuring your Grafana data source](https://docs.newrelic.com/docs/configure-prometheus-data-source-grafana) to limit results to just those from a particular `prometheus_server`. **Optimize throughput and memory consumption** Remote write [increases the total memory consumption](https://prometheus.io/docs/practices/remote_write/#memory-usage) of your Prometheus servers. If you're experiencing issues we recommend the following: - Increase [`max_samples_per_send`](https://prometheus.io/docs/practices/remote_write/#max_samples_per_send) for higher throughput workloads, along a proportional increase in [`capacity`](https://prometheus.io/docs/practices/remote_write/#capacity). - If memory consumption is still a problem, try limiting the number of [`max_shards`](https://prometheus.io/docs/practices/remote_write/#max_shards) per server. ## Troubleshoot error messages [#error-messages] If you receive an integration error message from New Relic or error messages in your Prometheus server logs after restarting your Prometheus server, review our [remote write troubleshooting documentation](https://docs.newrelic.com/docs/integrations/prometheus-integrations/install-configure-remote-write/remote-write-errors-error-messages). This includes fixing common errors, such as missing or incorrect characters, bad requests, request entity too large, and rate limit errors. ## Remove the integration [#remove-integration] When you remove the Prometheus remote write integration, this stops new data from flowing, but it will not purge or remove any historical data. To remove the integration, remove the configuration code snippet from your Prometheus YML file, then restart the server.