You can get Prometheus data flowing in New Relic with just a few simple steps. If you don't already have an account, you can sign up for New Relic right now.
This page covers basic setup for the remote write integration, as well as a few common troubleshooting topics. For information on high availability integration, see the Prometheus High Availability resource.
Set up the integration
From the Prometheus remote write setup page, complete these steps.
- Enter a name for the Prometheus server you’re connecting to and your
The name you enter for the server will create an attribute on your data, and will be the name used to identify which Prometheus server you're connecting to.
- Add a new
remote_writeURL to your Prometheus YML file. A good place to add this information is under
global_configin the file, at the same indentation level as the
globalsection. The end result should look like this:
remote_write: - url: https://metric-api.newrelic.com/prometheus/v1/write?X-License-Key=<YOUR_LICENSE_KEY>&prometheus_server=<YOUR_DATA_SOURCE_NAME>
If you are completing the remote write integration using Kubernetes and Helm, you should instead add the remote write URL to your Helm
remoteWrite: with two lines similar to the example below. Be sure to use your remote write URL and use indentation that matches the rest of the file:
remoteWrite: - url: https://metric-api.newrelic.com/prometheus/v1/write?X-License-Key=<YOUR_LICENSE_KEY>&prometheus_server=<YOUR_DATA_SOURCE_NAME>
- Restart your Prometheus server.
- View your data.
Customize remote write behavior
You may be interested in customizing the following parameters if you are writing to more than one account in New Relic or are connecting more than one Prometheus data source to the same account in New Relic.
- X-License Key
Your account's license key is not an API key. The license key is used for authentication and to identify which account to write data into.
- If you are configuring Prometheus to write into different New Relic accounts or sub-accounts you should use a different key on each remote write URL.
prometheus_serverparameter is a label or attribute used to add to stats that are written to NRDB. You should use this same label when configuring your Grafana data source to limit results to just those from a particular
Troubleshooting the integration
If you receive an integration error message from New Relic or error messages in your Prometheus server logs after restarting your Prometheus server, there are several actions you can take to troubleshoot and get data flowing properly. Below are a few tips regarding common issues and error messages.
Common errors and issues:
- Missing or incorrect characters in the remote write URL in the config file (for example the endpoint, license key, or
prometheus_servername) or incorrect placement of the information in the file will result in the Prometheus server not starting, remote write not working properly, or errors appearing in Prometheus server logs.
- 400: bad request error. If no data appears with a bad request error, check your configuration file to confirm that the placement of the remote write information is correct, and that there are no missing or incorrect characters.
- 413: request entity too large. This means you have sent a request in which one or more fields, or the entire payload, has exceeded our limits.
- 429: rate limit error. This means you have hit a rate limit on the amount of data being sent at one time (for example cardinality or data points per minute). You can troubleshoot by reducing the amount of Prometheus or general metric data you are sending, or by requesting a rate-limit increase.
You can investigate error messages in New Relic by doing either or both of the following:
- Run a query on the error message using NRQL then look at the Message field in UI to see a description of what went wrong. For example:
SELECT * FROM NrIntegrationError WHERE newRelicFeature = 'Metrics'
- Investigate individual errors in time to see when and where they occur and any simultaneously occurring issues, and perform targeted troubleshooting based on what you find out. For example:
SELECT count(*) FROM NrIntegrationError WHERE newRelicFeature = 'Metrics' TIMESERIES
- If you’ve validated that you can send data successfully but are unable to query it, you may be running into other kinds of limits, like the inspected count limit. This may manifest itself as an error message during the integration process that says:
Unable to retrieve data for Prometheus data source <name>.
Remove the integration
To remove the integration, remove the configuration code snippet from your Prometheus YML file and restart the server.
When you remove the integration, this stops new data from flowing, but will not purge or remove any historical data.