Set up your Prometheus remote write integration

You can get Prometheus data flowing in New Relic with just a few simple steps. If you don't already have an account, you can sign up for New Relic right now.

This page covers basic setup for the remote write integration, as well as a few common troubleshooting topics. For information on high availability integration, see the Prometheus High Availability resource.

Set up the integration

From the Prometheus remote write setup page, complete these steps.

  1. Enter a name for the Prometheus server you’re connecting to and your remote_write URL.

    The name you enter for the server will create an attribute on your data, and will be the name used to identify which Prometheus server you're connecting to.

  2. Add a new remote_write URL to your Prometheus YML file. A good place to add this information is under global_config in the file, at the same indentation level as the global section. The end result should look like this:
    remote_write: 
    - url: https://metric-api.newrelic.com/prometheus/v1/write?X-License-Key=<YOUR_LICENSE_KEY>&prometheus_server=<YOUR_DATA_SOURCE_NAME>  
    

    ​​​​If you are completing the remote write integration using Kubernetes (with or without Helm), you should instead add the remote write URL to your YAML operator file. The result should look something like this example.

  3. Restart your Prometheus server.
  4. View your data.

Customize remote write behavior

You may be interested in customizing the following parameters if you are writing to more than one account in New Relic or are connecting more than one Prometheus data source to the same account in New Relic.

X-License Key

Your account's license key is not an API key. The license key is used for authentication and to identify which account to write data into.

Details:

  • If you are configuring Prometheus to write into different New Relic accounts or sub-accounts you should use a different key on each remote write URL.
prometheus_server URL parameter

The prometheus_server parameter is a label or attribute used to add to stats that are written to NRDB. You should use this same label when configuring your Grafana data source to limit results to just those from a particular prometheus_server.

Troubleshooting the integration

If you receive an integration error message from New Relic or error messages in your Prometheus server logs after restarting your Prometheus server, there are several actions you can take to troubleshoot and get data flowing properly. Below are a few tips regarding common issues and error messages.

Common errors and issues:

  • Missing or incorrect characters in the remote write URL in the config file (for example the endpoint, license key, or prometheus_server name) or incorrect placement of the information in the file will result in the Prometheus server not starting, remote write not working properly, or errors appearing in Prometheus server logs.
  • 400: bad request error. If no data appears with a bad request error, check your configuration file to confirm that the placement of the remote write information is correct, and that there are no missing or incorrect characters.
  • 413: request entity too large. This means you have sent a request in which one or more fields, or the entire payload, has exceeded our limits.
  • 429: rate limit error. This means you have hit a rate limit on the amount of data being sent at one time (for example cardinality or data points per minute). You can troubleshoot by reducing the amount of Prometheus or general metric data you are sending, or by requesting a rate-limit increase.

You can investigate error messages in New Relic by doing either or both of the following:

  • Run a query on the error message using NRQL then look at the Message field in UI to see a description of what went wrong. For example:
    SELECT * FROM NrIntegrationError WHERE newRelicFeature = 'Metrics'
    
  • Investigate individual errors in time to see when and where they occur and any simultaneously occurring issues, and perform targeted troubleshooting based on what you find out. For example:
    SELECT count(*) FROM NrIntegrationError WHERE newRelicFeature = 'Metrics' TIMESERIES
  • If you’ve validated that you can send data successfully but are unable to query it, you may be running into other kinds of limits, like the inspected count limit. This may manifest itself as an error message during the integration process that says: Unable to retrieve data for Prometheus data source <name>.

Remove the integration

To remove the integration, remove the configuration code snippet from your Prometheus YML file and restart the server.

When you remove the integration, this stops new data from flowing, but will not purge or remove any historical data.

For more help

If you need more help, check out these support and learning resources: