All metrics for Docker and Kubernetes are stored in the
By default, the following attributes will be added to all metrics for Docker and Kubernetes integrations:
||The name of the cluster provided in the scraper configuration.|
||The name of this integration (
||The version of the integration; for example,
||The name of the metric itself.|
||The type of the New Relic
||The metric type of the Prometheus metric|
||The URL of the endpoint is being scraped.|
Kubernetes: If the scraper is running in Kubernetes, New Relic also adds the following attributes to all the metrics:
|Additional Kubernetes attributes||Description|
||Name of the deployment, if scraping a pod.|
||The Kubernetes labels of the object being scraped, prefixed by
||Name of the namespace.|
||Name of the node where the pod being scraped is running, if applicable.|
||Name of the pod being scraped, if applicable.|
||Name of the service being scraped, if applicable|
NRQL query examples
When you build queries, be aware that there is no linking between the metrics, entities, and attributes. Use the following NRQL queries to find out which metrics are available and which attributes are present on these metrics:
- Get metric names
To get all metric names:
FROM Metric SELECT uniques(metricName)
To get metric names for a specific endpoint:
FROM Metric SELECT uniques(metricName) WHERE scrapedEndpoint='<ep>'
To get metric names for a specific cluster, namespace, or pod:
FROM Metric SELECT uniques(metricName) WHERE clusterName='<cn>'
FROM Metric SELECT uniques(metricName) WHERE namespaceName='<ns>'
FROM Metric SELECT uniques(metricName) WHERE podName='<pod>'
- Get the attributes for a metric
To get all attributes for the selected metric:
FROM Metric SELECT keyset() WHERE metricName='<mn>'
- Get the values for an attribute
The autocomplete will show all values of the attribute, regardless of the pod. To determine the attribute values for a specific pod:
FROM Metric SELECT uniques(<attribute>) WHERE metricName='<mn>' AND podName='<pod>'
Build the query
- Get metric values
To get raw metric values:
FROM Metric SELECT <metricName> WHERE <attribute>='<value>'
- Get a chart of the metric
To get a chart of the metric with an aggregator of
FROM Metric SELECT <aggregator>(<metricname>) WHERE <attribute>='<value>' TIMESERIES
- Query counter metrics (deltas)
Currently the integration calculates the deltas for counter metrics. This is why queries on counter metrics will show the deltas of the counter instead of the absolute value of the counter.
- View connected Redis clients per pod
Docker: This example assumes you are scraping Redis exporters. To view the number of connected Redis clients per endpoint in a cluster:
FROM Metric SELECT latest(redis_connected_clients) WHERE clusterName='my-cluster' FACET scrapedEndpoint TIMESERIES
Kubernetes: This example assumes that you have Redis pods with the Redis exporter installed. To view the number of connected Redis clients per pod in the default namespace:
FROM Metric SELECT latest(redis_connected_clients) WHERE namespaceName='default' FACET podName TIMESERIES
- Docker: View average memory free for scraped endpoints
This example assumes you are scraping node exporters for Docker. To view average memory free for all scraped endpoints in a cluster:
FROM Metric SELECT average(node_memory_MemFree_bytes) WHERE clusterName='my-cluster'
- Kubernetes: View average memory usage for pods in a deployment
To view average memory usage for all pods in a Kubernetes deployment:
FROM Metric SELECT average(container_memory_usage_bytes) WHERE deploymentName='my-app-deployment' AND namespaceName='default'
View data in New Relic
After you query the data, you can view the results in the New Relic UI. You can also visualize the data as charts, histograms, etc.
Create histograms and summaries
With version 1.2.0 or higher of the Prometheus OpenMetrics integration, you can create histograms and percentiles (summaries) of your data. These are based on New Relic's guidelines in GitHub for higher level metric abstractions.
The dimension will be this:
Quantiles (summaries) are transformed into percentiles.
The dimension will be this:
To better support visualization of histograms, percentiles are calculated based on the histogram metrics and sent to New Relic. To configure the calculated percentiles, use the
percentiles configuration option.