Transition guide for OpenTelemetry cumulative sum metrics

Our OTLP metric pipeline is being upgraded to better support how we handle cumulative metrics. As part of this upgrade we're transitioning the underlying metric data type for monotonic cumulative sums from a gauge type to the cumulativeCount type. The cumulativeCount type allows you to query both the cumulative and delta value associated with reported data.

Starting April 4, 2023, we will ingest cumulative monotonic metrics according to the following end-of-life (EoL) timeline:

• April 4, 2023:
  • The ingest of cumulative monotonic sums as the cumulativeCount metric type is released (non-monotonic cumulative sums are not affected). The prior experience of ingesting these metrics as the gauge metric type is deprecated.
  • Accounts that were actively sending cumulative monotonic sums prior to this date can continue to use gauge type ingest.
  • All other accounts will have cumulative monotonic sums ingested as cumulativeCount metric types.
• April 4 through October 22:
  • Accounts using the deprecated gauge type ingest can opt-in to cumulativeCount type ingest. Early opt-in is the best way to ensure affected queries and alerts are not disrupted by the EoL.
• October 22, 2023: All accounts will be migrated to cumulativeCount type ingest for cumulative monotonic sums.

If you were ingesting cumulative monotonic sums prior to April 4, this 180 day period gives you time to update queries and alerts that make use of the affected metrics.

The sections below will teach you:

• How to know if you're impacted
• How to prepare for the EoL
• How to opt-in to the cumulativeCount functionality prior to the EoL date

Check for impact

If your account was sending OTLP cumulative sum metrics to New Relic before April 4, 2023, then you'll be impacted by the EoL on October 22, 2023.

To see if you've been impacted, run the following NRQL query for the relevant account:

FROM NrIntegrationError SELECT count(*)
WHERE newRelicFeature = 'cumulativeSumAsGaugeEoL'
SINCE 24 hours ago

The query will return results if you've reported metrics impacted by this EoL in the last 24 hours.

To investigate further, you can view the event message to see which metrics are affected:

FROM NrIntegrationError SELECT uniques(metricName)
WHERE newRelicFeature = 'cumulativeSumAsGaugeEoL'
LIMIT MAX SINCE 24 hours ago

Any monitoring or alerting involving these metrics will need to be reviewed and updated prior to the EoL date. The next section contains suggested actions for impacted metrics.

What to do if you're impacted

On October 22, the reporting of OTLP monotonic cumulative sum metrics will switch from using the gauge to cumulativeCount metric type. If you're using those metrics, you'll want to prepare for that. For example, you may want to create duplicate alert conditions with the new metric type so that, when the switch happens, you won't have a disruption (and we explain how to do that below).

Our OpenTelemetry best practices outline the differences in querying between a gauge and a cumulativeCount metric type. Namely, you'll be able to target either the delta value (by default) or the cumulative value by selecting the cumulative field.

Let's take an example query of a cumulative monotonic sum metric metricName. Up until the EoL, a query like:

FROM Metric SELECT latest(<metricName>)

would be evaluating only gauge metrics. Once the EoL happens, the underlying metric type associated with metricName will change to cumulativeCount. Depending on the alert, this change could cause false results as the metric value for a gauge and cumulativeCount type represent different values of the data point.

Below we focus on updating alert conditions, because alerts are generally more important than charts. For your custom charts, you can either:

• Take the same general approach we describe for alerts (create a new chart with an updated query and erase the old chart after the EoL), OR
• Wait until after the EoL and update the charts

Suggested action: Create new versions of alerts

Here are some examples of queries that might be used in alert condition. We'll give you the original query, then a gauge version of the query, and a cumulativeCount version of the query. You should replace your existing alert with the gauge version of the query, and create a duplicate alert condition that uses the cumulativeCount version. That way, when we transition your account to cumulativeCount on October 22, that alert condition will be available when the old query no longer works.

Cumulative queries

Original query:

FROM Metric SELECT latest(<metricName>)

Gauge version of the query:

FROM Metric SELECT latest(<metricName>)
WHERE <metricName>[type] = 'gauge'

Cumulative count version of the query:

FROM Metric SELECT latest(<metricName>[cumulative])
WHERE <metricName>[type] = 'cumulativeCount'

Rate of change queries

Original query:

FROM Metric SELECT derivative(<metricName>, 1 minute)

Gauge version of query:

FROM Metric SELECT derivative(<metricName>, 1 minute)
WHERE <metricName>[type] = 'gauge'

Cumulative count version of query:

FROM Metric SELECT rate(sum(<metricName>[cumulative]), 1 minute)
WHERE <metricName>[type] = 'cumulativeCount'

Suggested action: preview cumulativeCount before the EoL

Affected accounts will continue to have their cumulative sums stored as gauge types until the EoL. However, optionally, you can try out cumulativeCount before the EoL to validate your queries and alerts are behaving as expected. To test cumuativeCount types early, add the following attribute to OTLP data you send to New Relic:

newrelic_metric_type=cumulativeCount

Instrumentation library

If you're using an OTLP instrumentation library within your code base, you should reference the appropriate language SDK for your application. Any method of adding the attribute newrelic_metric_type=cumulativeCount to individual metric data points or to the resource attribute level should be sufficient.

Here's an example of adding the attribute directly to the reporting metric in the Java SDK:

final var meter = openTelemetry.meterBuilder("manual-instrumentation-scope")
                .setInstrumentationVersion("1.0.0")
                .build();
final LongCounter counter = meter.counterBuilder("myCumulativeCounter")
                .build();
counter.add(1, Attributes.of(stringKey("newrelic_metric_type"), "cumulativeCount"));

Collector

If the source of your OTLP monitoring is coming from the OTLP collector, you can use the OpenTelemetry attributes processor to add the attribute before it's sent to your exporters. For example, the following snippet will insert the newrelic_metric_type=cumulativeCount attribute:

…
attributes:
  actions:
    - key: newrelic_metric_type
      action: insert
      value: cumulativeCount
…
service:
  pipelines:
    metrics:
      …
      processors: [...,attributes,...]
      …

Checking results

After setting newrelic_metric_type=cumulativeCount in your metric data, you can check that we're receiving it correctly with this query:

FROM Metric SELECT count(*) WHERE %[type] = 'cumulativeCount' TIMESERIES