Ignore or include Prometheus metrics

Avoid sending Prometheus OpenMetrics integration data that is not relevant to your monitoring needs. Instead, use filters to ignore or include specific metrics. This will help you control the amount and types of data you send to New Relic. This will also help you avoid additional billing charges, as explained in this document.

Prevent billing increases

We use the Prometheus discovery and scrape annotations. If you set the Prometheus OpenMetrics integration to scrape all the available targets and to send all the data that's exposed from those targets, you may exceed New Relic's platform limits and increase your billing charges. To help prevent this from happening, use the integration's filtering capabilities.

For more information, see the Prometheus OpenMetrics integration requirements for Docker. Also see the troubleshooting procedures for NrIntegrationError events.

Identify metrics to ignore or include

To decide what data to include or exclude, use New Relic's Metric API to explore your metric data. Then, refine your filters to scrape only relevant targets and send useful metrics.

To filter out unwanted metrics from a target, use the ignore_metrics configuration option.
To filter out targets instead of the metrics, use the scrape_enabled_label configuration option.

Caution

The histogram and summary metrics type filtering apply to the base name. You can't filter by the _bucket, _sum, or _count time series for that metric.

The nri-prometheus-latest.yaml manifest file includes the nri-prometheus-cfg config map showing an example configuration. The integration will ignore or include metrics before executing the other functions to add, rename, or copy attributes.

To ignore unwanted metrics, use the following transformation.

Example configuration

To drop all metrics that start with go_ or process_:

transformations:
  - description: "General processing rules"
    ignore_metrics:
      - prefixes:
        - "go_"
        - "process_"

Example of raw Prometheus metrics

go_goroutines 13
process_virtual_memory_bytes 2.062336e+07
mysql_global_status_commands_total{command="ha_close"} 0
mysql_global_status_commands_total{command="ha_open"} 0

This is taken from the MySQL exporter. Besides the MySQL metrics, it also exposes metrics about the exporter that may not be of interest to you.

Example output

After filtering has been applied, the go_ and process_ metrics are dropped and the mysql_ metrics are shipped to New Relic.

mysql_global_status_commands_total{command="ha_close"} 0 
mysql_global_status_commands_total{command="ha_open"} 0

If you only want to include specific metrics, you can use the except list under the ignore_metrics section. As the name implies, this will ignore all the metrics except the ones that contain the with the given prefixes.

Example configuration:

To drop all metrics except kube_hpa_:

transformations:
  - description: "General processing rules"
    ignore_metrics:
      - except:
        - kube_hpa_

You can combine both prefixes and except under the ignore_metrics section for more complex filtering.

Example configuration

To drop all coredns_ metrics, except for coredns_dns_request_count_total and coredns_dns_responses_total:

transformations:
  - description: "CoreDNS Example Metrics"
    ignore_metrics:
      - prefixes:
        - coredns_
        except:
        - coredns_dns_request_count_total
        - coredns_dns_responses_total

Prevent billing increases

Identify metrics to ignore or include

Caution

Filter out unwanted metrics (ignore)

Include only specific metrics (except)

Combination of include and exclude (except and ignore)

Ignore or include Prometheus metrics

Prevent billing increases .css-21sua1{background:none;border:none;width:0;padding:0;}

Identify metrics to ignore or include

Caution

Include only specific metrics (except)

Combination of include and exclude (except and ignore)

Prevent billing increases