Set up your Prometheus remote write integration

You can get Prometheus data flowing in New Relic with just a few simple steps. If you don't already have an account, you can sign up for New Relic right now.

Set up the integration

From the Prometheus remote write setup page, complete these steps.

1. Enter a name for the Prometheus server you’re connecting to and your remote_write URL.

   The name you enter for the server will create an attribute on your data, and will be the name used to identify which Prometheus server you're connecting to.

2. Add a new remote_write URL to your Prometheus YML file. A good place to add this information is under global_config in the file, at the same indentation level as the global section. The end result should look like this:

   remote_write: 
   - url: https://metric-api.newrelic.com/prometheus/v1/write?X-License-Key=<YOUR_LICENSE_KEY>&prometheus_server=<YOUR_DATA_SOURCE_NAME>  
   

   If you are completing the remote write integration using Kubernetes and Helm, you should instead add the remote write URL to your Helm values.yaml file. Replace remoteWrite: [] with two lines similar to the example below. Be sure to use your remote write URL and use indentation that matches the rest of the file:

   remoteWrite: 
   - url: https://metric-api.newrelic.com/prometheus/v1/write?X-License-Key=<YOUR_LICENSE_KEY>&prometheus_server=<YOUR_DATA_SOURCE_NAME>  
   

3. Restart your Prometheus server.
4. View your data.

Customize remote write behavior

You may be interested in customizing the following parameters if you are writing to more than one account in New Relic or are connecting more than one Prometheus data source to the same account in New Relic.

X-License Key

Your account's license key is not an API key. The license key is used for authentication and to identify which account to write data into.

Details:

• If you are configuring Prometheus to write into different New Relic accounts or sub-accounts you should use a different key on each remote write URL.

prometheus_server URL parameter

The prometheus_server parameter is a label or attribute used to add to stats that are written to NRDB. You should use this same label when configuring your Grafana data source to limit results to just those from a particular prometheus_server.

Troubleshooting the integration

If you receive an integration error message from New Relic or error messages in your Prometheus server logs after restarting your Prometheus server, there are several actions you can take to troubleshoot and get data flowing properly. Below are a few tips regarding common issues and error messages.

Common errors and issues:

• Missing or incorrect characters in the remote write URL in the config file (for example the endpoint, license key, or prometheus_server name) or incorrect placement of the information in the file will result in the Prometheus server not starting, remote write not working properly, or errors appearing in Prometheus server logs.
• 400: bad request error. If no data appears with a bad request error, check your configuration file to confirm that the placement of the remote write information is correct, and that there are no missing or incorrect characters.
• 413: request entity too large. This means you have sent a request in which one or more fields, or the entire payload, has exceeded our limits.
• 429: rate limit error. This means you have hit a rate limit on the amount of data being sent at one time (for example cardinality or data points per minute). You can troubleshoot by reducing the amount of Prometheus or general metric data you are sending, or by requesting a rate-limit increase.

You can investigate error messages in New Relic by doing either or both of the following:

• Run a query on the error message using NRQL then look at the Message field in UI to see a description of what went wrong. For example:

  SELECT * FROM NrIntegrationError WHERE newRelicFeature = 'Metrics'
  

• Investigate individual errors in time to see when and where they occur and any simultaneously occurring issues, and perform targeted troubleshooting based on what you find out. For example:

  SELECT count(*) FROM NrIntegrationError WHERE newRelicFeature = 'Metrics' TIMESERIES

• If you’ve validated that you can send data successfully but are unable to query it, you may be running into other kinds of limits, like the inspected count limit. This may manifest itself as an error message during the integration process that says: Unable to retrieve data for Prometheus data source <name>.

Remove the integration

To remove the integration, remove the configuration code snippet from your Prometheus YML file and restart the server.

For more help