If you're using our Prometheus OpenMetrics integration,
nri-prometheus in Kubernetes, you need to migrate to New Relic's Prometheus agent,
newrelic-prometheus-agent, to achieve the same monitoring experience. See how to set up New Relic's Prometheus agent.
newrelic-prometheus-agent is actually running the
Prometheus server in agent mode, the configuration file structure has changed to offer more flexibility and the possibility to use any scrape related feature of the Prometheus server, such as support of many authorization methods, relabel configuration setups, or scaling Prometheus instances.
We translate our simplified configuration file to a Prometheus server configuration file, to give you an easier configuration experience. We suggest that you read the documentation to be able to better follow this migration guide.
To perform the migration, adjust the configuration section in your
values.yml file with the values in this document.
Dashboards and alerts
The metrics are sent in the same way that they were sent using the Prometheus OpenMetrics integration,
nri-prometheus. In some cases you may need to modify some metadata that has changed between Prometheus OpenMetrics integration,
nri-prometheus, and the Prometheus agent,
The following table highlights the metadata renamed between the Prometheus OpenMetrics integration,
nri-prometheus and the Prometheus agent,
If you'd like the previous
nri-prometheus metadata attribute names to persist after the migration, you can enable the following remote write relabel config in the
newrelic-prometheus-agent values.yaml file. This will ensure that dashboards and alerts targeting these attributes will continue to function normally.
newrelic_remote_write: extra_write_relabel_configs: # Enable the extra_write_relabel_configs below for backwards compatibility with legacy POMI labels. # This helpful when migrating from POMI to ensure that Prometheus metrics will contain both labels (e.g. cluster_name and clusterName). # For more migration info, please visit the [migration guide](/docs/infrastructure/prometheus-integrations/install-configure-prometheus-agent/migration-guide/). - source_labels: [namespace] action: replace target_label: namespaceName - source_labels: [node] action: replace target_label: nodeName - source_labels: [pod] action: replace target_label: podName - source_labels: [service] action: replace target_label: serviceName - source_labels: [cluster_name] action: replace target_label: clusterName - source_labels: [job] action: replace target_label: scrapedTargetKind - source_labels: [instance] action: replace target_label: scrapedTargetInstance
To comply with the Prometheus DataModel, Kubernetes label names (
label.kubernetesLabel) are sanitized after scraped to replace any unsupported character to
_. For example, the Kubernetes label
k8s.io/app is added as
label. prefix has been removed. The value of the label remains unchanged.
The following metadata isn't available with the Prometheus agent,
Kubernetes target discovery
We've improved the target discovery configuration by adding
Jobs. See how to configure Kubernetes target discovery.
Default behaviors have changed between Prometheus OpenMetrics and the Prometheus agent. While, by default, Prometheus OpenMetrics scrapes pods and services containing
prometheus.io/scrape=true as a label or annotation, the Prometheus agent scrapes all targets annotated with
newrelic.io/scrape: "true" the pods and services annotated with
prometheus.io/scrape=true will only be scraped by default if it corresponds to one of the supported New Relic's prometheus integrations, check the scrape config documentation for details.
Node metrics are no longer scraped by default in the Prometheus agent, as it may cause metric duplication since it's normally used by New Relic's kubernetes integration.
nri-prometheus, as a custom solution, converts Prometheus metrics to New Relic metrics before sending them, by applying mappings according to the metric metadata type reading from the scraped data:
# TYPE <metric_type>.
On the Prometheus agent,
newrelic-prometheus-agent, which leverages a standard Prometheus server, metrics are sent directly to the New Relic's remote write endpoint, which handles this conversion based on metric names. Some metrics may not be properly converted by the remote write endpoint:
- Prometheus Counter metrics without the name suffix
bucketwill be converted as Gauge.
- Prometheus Gauge metrics with the name suffix
bucketwill be converted as Counter.
To fix this, you can set up metric type mappings relabel options, check the metric and label transformations for details.
nri-prometheus, transformations are now replaced with Prometheus relabel configurations. See a list of examples of relabel configuration setups.
Similar to Prometheus OpenMetrics (
nri-prometheus), the Prometheus agent (
newrelic-prometheus-agent) self-scrapes internal metrics. These metrics have the
prometheus_ prefix and you can use them to observe the status of the Prometheus instance. By default, only a reduced set of these metrics are sent. You can find the list of metrics on the default values.yaml file of the chart, by searching for
We also give you a curated dashboard for self-metrics with performance and health information, and also metrics volume sent. Install the quickstart containing the dashboard for the Prometheus agent in your New Relic account.
Keep Prometheus OpenMetrics integration during the migration
Although our Prometheus OpenMetrics integration in Kubernetes has been replaced by the NR Prometheus agent, it will be supported for a period of time (
June 2023January 2024).
New Relic wants to ensure a smooth and seamless migration to our customers. You can keep either the Prometheus OpenMetrics integration,
nri-prometheus, the Prometheus agent,
newrelic-prometheus-agent, or both.
To preserve nri-prometheus and not using Prometheus agent as of now, set your
values.yaml file as follows:
nri-metadata-injection: enabled: true
nri-prometheus: enabled: true
newrelic-prometheus-agent: enabled: false
Follow steps explained in this document to upgrade the Kubernetes cluster using Helm.