Configure New Relic as a Prometheus data source for Grafana

You can configure a Prometheus data source in Grafana to query data stored in the New Relic Database (NRDB) using the PromQL query language.

Add a Prometheus data source

Follow these steps to add New Relic as a Prometheus data source for Grafana.

You must complete the Prometheus remote-write integration process prior to beginning the configuration process.

These instructions detail how to complete the process when working with Grafana versions 6.7 and higher.

  1. On New Relic, Create a new Query key.

    Note: In Grafana, you'll need put this in a custom X-Query-Key HTTP header (see step 7 below), but it is the same entity as the New Relic Query key.

  2. From the Grafana Home screen, go to Configuration > Data Sources and click Add data source.
  3. From the Add data source screen under Time series databases options, select Prometheus.
  4. Enter the Name you want to use for your new Prometheus data source.
  5. Set the Default toggle to the on or off position, depending on whether you want this to be your default data source for Prometheus queries.
    • Off: this is not your default data source
    • On: this is your default data source
  6. Enter the correct URL:
  7. Under Custom Headers, select Add Header. Set the Header name to X-Query-Key. For the Value, enter to the Query key you created in step 1.
  8. For HTTP Method select GET.
  9. Click Save & Test.

If your line graphs appear as a series of dots and not as connected lines, you can change in Graphana's Graph panel and selecting Stacking and null value > connected.

New Relic strongly recommends using versions 6.7.x and higher to configure New Relic as a Prometheus data source. If you do chose to complete the configuration while running an earlier version, you will need to do one of the following to successfully configure your data source:

  • Configure the new data source to use basic authentication and then enter the Query-key as the password in the basic authentication workflow.
  • Configure the new data source URL to include the Query-key: https://prometheus-api.newrelic.com/auth/`<query-key>`
Sample configuration image
Screen capture of the add data source workflow in Grafana
Grafana Data Source Config.png, by dbarnesbrown.newrelic.com

Customize Prometheus API behavior

Headers are particularly important if you have connected multiple Prometheus servers to New Relic using the remote write integration. Here are some details about customization.

X-Query-Key (Query key)

The Query key parameter is required to authenticate with New Relic and identify the account containing your metrics. Grafana calls this an X-Query key, while it appears as a Query key in the New Relic UI.

Details:

  • Required
  • An API query key used for authentication
  • If you are using the Prometheus remote write integration, the X-Query-Key should correspond to the same account as the X-License-Key used to integrate for remote write
X-Prometheus-Only

Grafana's auto-complete support doesn't handle metrics that fail to strictly conform to Prometheus naming conventions. New Relic recommends you exclude any metrics that do not conform from this parameter.

Details:

  • Optional
  • Limits metrics exposed by the API to those originating from Prometheus
  • Default = true if not specified
X-Prometheus-Server

This parameter is useful if you are collecting metrics from multiple Prometheus servers. For example, if you are using Grafana, you might want to create a data source for each Prometheus server connected to New Relic and then another data source that can be used to query across all Prometheus servers.

Details:

  • Optional
  • Limits metrics exposed by the API to those collected from the specified Prometheus server
  • This value should match the prometheus_server URL parameter in the remote write URL used to connect to your prometheus server to New Relic
  • Defaults to return metrics collected from all servers

Delete a Prometheus data source

To delete a data source in Grafana:

  1. Go to Configuration > Data Sources.
  2. Click on the data source you want to delete.
  3. Click the Delete button at the bottom of the page.

For more help

If you need more help, check out these support and learning resources: