View and query your Prometheus data

To query and visualize the metrics collected for your Prometheus OpenMetrics or remote write integration with New Relic, you can use NRQL. You can also translate your PromQL-style queries to NRQL using either Grafana or the query builder.

All metrics for Docker and Kubernetes are stored in the Metric type.

Default attributes for the OpenMetrics integration

By default, the following attributes will be added to all metrics for Docker and Kubernetes integrations:

Default attributes
(all integrations)
Description
clusterName The name of the cluster provided in the scraper configuration.
integrationName The name of this integration (nri-prometheus).
integrationVersion The version of the integration; for example, 0.2.0.
metricName The name of the metric itself.
nrMetricType The type of the New Relic Metric type; for example, Gauges.
promMetricType The metric type of the Prometheus metric
scrapedEndpoint The URL of the endpoint is being scraped.

img-integration-k8s@2x.png Kubernetes: If the scraper is running in Kubernetes, New Relic also adds the following attributes to all the metrics:

Additional Kubernetes attributes Description
deploymentName Name of the deployment, if scraping a pod.
label The Kubernetes labels of the object being scraped, prefixed by "label".
namespaceName Name of the namespace.
nodeName Name of the node where the pod being scraped is running, if applicable.
podName Name of the pod being scraped, if applicable.
serviceName Name of the service being scraped, if applicable

Default attributes for the remote write integration

By default, the following attributes will be added to Prometheus remote write metrics:

Default attributes
(all integrations)
Description
prometheus_server A user supplied label specified as a Prometheus remote write URL parameter. The value supplied should be unique as it is intended to differentiate between source Prometheus servers at query time. Unspecified by default.
newrelic.source The name of the New Relic ingest point (prometheusAPI).
instrumentation.provider prometheus
instrumentation.name remote-write
instrumentation.source A user supplied identifier for the source of the Prometheus data that matches the value of prometheus_server.
instrumentation.version Used to identify the version of the remote write API; for example, 0.0.1.

NRQL query examples

When you build queries, be aware that there is no linking between the metrics, entities, and attributes. Use the following NRQL queries to find out which metrics are available and which attributes are present on these metrics:

Get metric names

To get all metric names for OpenMetrics:

FROM Metric SELECT uniques(metricName)

To get metric names for a remote write integration:

FROM Metric SELECT uniques(metricName) WHERE instrumentation.provider='prometheus' AND instrumentation.name='remote-write'

To get metric names for a remote write integration from a single Prometheus source:

FROM Metric SELECT uniques(metricName) WHERE instrumentation.provider='prometheus' AND instrumentation.name='remote-write' AND instrumentation.source='<ds>'

To get metric names for a specific OpenMetrics endpoint:

FROM Metric SELECT uniques(metricName) WHERE scrapedEndpoint='<ep>'

To get metric names for a specific OpenMetrics cluster, namespace, or pod:

FROM Metric SELECT uniques(metricName) WHERE clusterName='<cn>'
FROM Metric SELECT uniques(metricName) WHERE namespaceName='<ns>'
FROM Metric SELECT uniques(metricName) WHERE podName='<pod>'
Get the attributes for a metric

To get all attributes for the selected metric:

FROM Metric SELECT keyset() WHERE metricName='<mn>'
Get the values for an attribute in OpenMetrics

The autocomplete will show all values of the attribute, regardless of the pod. To determine the attribute values for a specific pod:

FROM Metric SELECT uniques(<attribute>) WHERE metricName='<mn>' AND podName='<pod>'

Build the query

Using metric name and attributes, you can query your data. For more information about facets, time series, and time selection, see the NRQL documentation.

To build PromQL-style queries, see our docs.

Get metric values

To get raw metric values:

FROM Metric SELECT <metricName> WHERE <attribute>='<value>'
Get a chart of the metric

To get a chart of the metric with an aggregator of average, min, max, or sum:

FROM Metric SELECT <aggregator>(<metricname>) WHERE <attribute>='<value>' TIMESERIES
Query counter metrics (deltas)

Currently the integration calculates the deltas for counter metrics. This is why queries on counter metrics will show the deltas of the counter instead of the absolute value of the counter.

View connected Redis clients per pod with OpenMetrics

Docker icon Docker: This example assumes you are scraping Redis exporters. To view the number of connected Redis clients per endpoint in a cluster:

FROM Metric SELECT latest(redis_connected_clients) WHERE clusterName='my-cluster' FACET scrapedEndpoint TIMESERIES

img-integration-k8s@2x.png Kubernetes: This example assumes that you have Redis pods with the Redis exporter installed. To view the number of connected Redis clients per pod in the default namespace:

FROM Metric SELECT latest(redis_connected_clients) WHERE namespaceName='default' FACET podName TIMESERIES
Docker iconDocker: View average memory free for scraped endpoints

This example assumes you are scraping node exporters for Docker and want to use OpenMetrics. To view average memory free for all scraped endpoints in a cluster:

FROM Metric SELECT average(node_memory_MemFree_bytes) WHERE clusterName='my-cluster'
img-integration-k8s@2x.png Kubernetes: View average memory usage for pods in a deployment

To view average memory usage for all pods in a Kubernetes deployment using OpenMetrics:

FROM Metric SELECT average(container_memory_usage_bytes) WHERE deploymentName='my-app-deployment' AND namespaceName='default'

View data in New Relic

When you query the data, you can view the results in the New Relic UI. You can also visualize the data as charts, histograms, etc.

To view the NRQL query results for your Prometheus integration's data: Go to one.newrelic.com > Query your data. For more information, see New Relic's query builder documentation.

Create histograms and summaries

With remote write or version 1.2.0 or higher of the Prometheus OpenMetrics integration, you can create histograms and percentiles (summaries) of your data. The OpenMetrics data is based on New Relic's guidelines in GitHub for higher level metric abstractions, while the remote write data closely matches the schema of the original Prometheus data.

Data presentation Comments
Histograms

A bucket <basename>_bucket{le="42"} will be sent as this:

  • For OpenMetrics: <basename>_buckets
  • For remote write: <basename>_bucket

The dimension will be this:

  • For OpenMetrics: {histogram.bucket.upperBound="42"}
  • For remote write: {histogram.bucket.le="42"}
Percentiles

Quantiles (summaries) are transformed into percentiles.

A metric <basename>{quantile="0.3"} will be sent to New Relic as <basename>.percentiles.

The dimension will be this:

{percentile="30"}

NRQL has two functions that work on remote write ingested PromQL: bucketPercentile() and histogram(). The links include query examples.These two functions don't work on OpenMetrics ingested buckets.

To better support visualization of histograms, percentiles are calculated based on the histogram metrics and sent to New Relic. To configure the calculated percentiles for OpenMetrics, use the percentiles configuration option.

For more help

If you need more help, check out these support and learning resources: