Migrate from the Prometheus OpenMetrics integration

If you're using our Prometheus OpenMetrics integration, nri-prometheus in Kubernetes, you need to migrate to New Relic's Prometheus agent, newrelic-prometheus-agent, to achieve the same monitoring experience. See how to set up New Relic's Prometheus agent.

Since the newrelic-prometheus-agent is actually running the Prometheus server in agent mode, the configuration file structure has changed to offer more flexibility and the possibility to use any scrape related feature of the Prometheus server, such as support of many authorization methods, relabel configuration setups, or scaling Prometheus instances.

We translate our simplified configuration file to a Prometheus server configuration file, to give you an easier configuration experience. We suggest that you read the documentation to be able to better follow this migration guide.

To perform the migration, adjust the configuration section in your values.yml file with the values in this document.

Dashboards and alerts

The metrics are sent in the same way that they were sent using the Prometheus OpenMetrics integration, nri-prometheus. In some cases you may need to modify some metadata that has changed between Prometheus OpenMetrics integration, nri-prometheus, and the Prometheus agent, newrelic-prometheus-agent.

Tip

Read the full list of labels added by the Prometheus agent.

Renamed metadata

The following table highlights the metadata renamed between the Prometheus OpenMetrics integration, nri-prometheus and the Prometheus agent, newrelic-prometheus-agent:

`nri-prometheus`	`newrelic-prometheus-agent`
`namespaceName`	`namespace`
`nodeName`	`node`
`podName`	`pod`
`serviceName`	`service`
`deploymentName`	N/A
`clusterName`	`cluster_name`
`scrapedTargetKind`	`job`. You can also use it to indetify the scrape kind.
`scrapedTargetName`	`instance`. It also contains the `host:port` for the target.

If you'd like the previous nri-prometheus metadata attribute names to persist after the migration, you can enable the following remote write relabel config in the newrelic-prometheus-agent values.yaml file. This will ensure that dashboards and alerts targeting these attributes will continue to function normally.

newrelic_remote_write:
  extra_write_relabel_configs:
    # Enable the extra_write_relabel_configs below for backwards compatibility with legacy POMI labels.
    # This helpful when migrating from POMI to ensure that Prometheus metrics will contain both labels (e.g. cluster_name and clusterName).
    # For more migration info, please visit the [migration guide](/docs/infrastructure/prometheus-integrations/install-configure-prometheus-agent/migration-guide/).
    - source_labels: [namespace]
      action: replace
      target_label: namespaceName
    - source_labels: [node]
      action: replace
      target_label: nodeName
    - source_labels: [pod]
      action: replace
      target_label: podName
    - source_labels: [service]
      action: replace
      target_label: serviceName
    - source_labels: [cluster_name]
      action: replace
      target_label: clusterName
    - source_labels: [job]
      action: replace
      target_label: scrapedTargetKind
    - source_labels: [instance]
      action: replace
      target_label: scrapedTargetInstance

Important

To comply with the Prometheus DataModel, Kubernetes label names (label.kubernetesLabel) are sanitized after scraped to replace any unsupported character to _. For example, the Kubernetes label k8s.io/app is added as k8s_io_app. The label. prefix has been removed. The value of the label remains unchanged.

Removed metadata

The following metadata isn't available with the Prometheus agent, newrelic-prometheus-agent:

deploymentName
integrationName
integrationVersion
nrMetricType
promMetricType
targetName
scrapedTargetName

Kubernetes target discovery

We've improved the target discovery configuration by adding Jobs. See how to configure Kubernetes target discovery.

Default behaviors have changed between Prometheus OpenMetrics and the Prometheus agent. While, by default, Prometheus OpenMetrics scrapes pods and services containing prometheus.io/scrape=true as a label or annotation, the Prometheus agent scrapes all targets annotated with newrelic.io/scrape: "true" the pods and services annotated with prometheus.io/scrape=true will only be scraped by default if it corresponds to one of the supported New Relic's prometheus integrations, check the scrape config documentation for details.

Node metrics are no longer scraped by default in the Prometheus agent, as it may cause metric duplication since it's normally used by New Relic's kubernetes integration.

Metric types

Prometheus OpenMetrics, nri-prometheus, as a custom solution, converts Prometheus metrics to New Relic metrics before sending them, by applying mappings according to the metric metadata type reading from the scraped data: # TYPE <metric_type>.

On the Prometheus agent, newrelic-prometheus-agent, which leverages a standard Prometheus server, metrics are sent directly to the New Relic's remote write endpoint, which handles this conversion based on metric names. Some metrics may not be properly converted by the remote write endpoint:

Prometheus Counter metrics without the name suffix total, count, sum, or bucket will be converted as Gauge.
Prometheus Gauge metrics with the name suffix total, count, sum, or bucket will be converted as Counter.

To fix this, you can set up metric type mappings relabel options, check the metric and label transformations for details.

Transformations

Prometheus OpenMetrics, nri-prometheus, transformations are now replaced with Prometheus relabel configurations. See a list of examples of relabel configuration setups.

Self-instrumentation

Similar to Prometheus OpenMetrics (nri-prometheus), the Prometheus agent (newrelic-prometheus-agent) self-scrapes internal metrics. These metrics have the prometheus_ prefix and you can use them to observe the status of the Prometheus instance. By default, only a reduced set of these metrics are sent. You can find the list of metrics on the default values.yaml file of the chart, by searching for job_name: self-metrics.

We also give you a curated dashboard for self-metrics with performance and health information, and also metrics volume sent. Install the quickstart containing the dashboard for the Prometheus agent in your New Relic account.

Keep Prometheus OpenMetrics integration during the migration

New Relic wants to ensure a smooth and seamless migration to our customers. You can keep either the Prometheus OpenMetrics integration, nri-prometheus, the Prometheus agent, newrelic-prometheus-agent, or both.

To preserve nri-prometheus and not using Prometheus agent as of now, set your values.yaml file as follows:

nri-metadata-injection:
  enabled: true

nri-prometheus:
  enabled: true

newrelic-prometheus-agent:
  enabled: false

Follow steps explained in this document to upgrade the Kubernetes cluster using Helm.