• Log inStart now

Introduction to the Amazon CloudWatch Metric Streams integration

AWS CloudWatch Metric Streams integration is the recommended solution to monitor all CloudWatch metrics from all AWS services, including custom namespaces.

Learn how to set up Metric Streams in New Relic.

Why does this matter?

Before CloudWatch metric streams, the only solution for AWS monitoring partners was to deploy a polling fleet and call multiple AWS APIs at regular intervals to retrieve the metrics and metadata. The following table shows the differences between both solutions:

API mode

Stream mode

It requires an integration with each AWS service to collect the metrics.

All CloudWatch metrics from all AWS services and custom namespaces are available in New Relic at once, without needing a specific integration to be built or updated.

There's one exception: metrics that are made available to CloudWatch with more than 2 hours delay are not included in the stream.

It adds an additional delay to metrics being available in New Relic for alerting and dashboarding. The fastest polling interval we offer today is 5 minutes.

Latency is significantly improved, since metrics are streamed in less than two minutes since they are made available in AWS CloudWatch.

It may lead to AWS API throttling for large AWS environments.

AWS API throttling is eliminated.

Want to try out our Amazon CloudWatch Metric Streams integration? Sign up with New Relic for free, forever!

Cost considerations

Consider the following when evaluating the cost of the AWS CloudWatch metric streams integration with New Relic:

Learn about the mechanisms available for data management, including filters in AWS and New Relic side. When applicable, make sure to complete an initial integration in a pre-production environment to evaluate the total cost of the solution based on a limited number of AWS resources and services.

Migrating from AWS API polling integrations

When metrics are sent via Metric Streams to New Relic, if the same metrics are being retrieved using the current poll-based integrations, those metrics will be duplicated. For example, alerts and dashboards that use sum or count will return twice the actual number. This includes alerts and dashboards that use metrics that have a .Sum suffix.

We recommend sending the data to a non-production New Relic account where you can safely do tests. If that is not an option, then AWS CloudWatch Metric Stream filters are available to include or exclude certain namespaces that can cause trouble.

Alternatively, you can use filtering on queries to distinguish between metrics that come from Metric Streams and those that come through polling. All metrics coming from Metric Streams are tagged with collector.name='cloudwatch-metric-streams'.

Migration steps

On a typical deployment, migrating from API polling to metric stream involves the following steps (we recommend trying this on a dev / staging environment first):

  1. Go through the AWS UI in New Relic (or use NerdGraph APIs) to link your AWS account with New Relic. This is currently needed even if your AWS account is already linked with polling integrations.
  2. Make sure you complete the last step in the onboarding, which involves enabling AWS CloudWatch metric stream and the AWS Kinesis Data Firehose to push metrics to New Relic.
  3. Repeat step 2 for any additional AWS region you want to monitor, since AWS CloudWatch requires one stream per region.
  4. Ensure metrics are received from all connected regions and namespaces. This may take several minutes.
  5. Disable all unnecessary polling integrations in the previous AWS provider account. Remember that some integrations still need to be enabled since they aren't fully replaced by metric streams.

Query, dashboard, and alert considerations

AWS Metric Streams integration uses the Metric API to push metrics in the dimensional metric format.

Poll-based integrations push metrics based on events (for example, ComputeSample event), and will be migrated to dimensional metrics in the future.

To assist in this transition, we provide a mechanism (known as shimming) that transparently lets you write queries in any format. Then these queries are processed as expected based on the source that's available (metrics or events). This mechanism works both ways, from events to metrics, and vice versa.

Tip

Learn more about the limitations of the query mechanism that allow customers to use event-based queries (*Samples) with the AWS CloudWatch metric stream integration (Dimensional Metric format).

Please consider the following when migrating from poll-based integrations:

  • Dashboards: Custom dashboards that use poll-based AWS integration events will still work as expected.
  • Alerts: Alert conditions that use poll-based AWS events will still work. We recommend adapting those to the dimensional metric format (using NRQL as source).
  • Entities: New Relic Explorer might show duplicated entities for up to 24 hours.

Integrations not fully replaced by metric streams

The AWS CloudWatch Metric Streams integration focuses on CloudWatch metrics. As a result, the following integrations still need to be configured and enabled to get complete visibility from AWS services.

Polling integrations based on service APIs:

  • AWS Billing
  • AWS CloudTrail
  • AWS Health
  • AWS Trusted Advisor
  • AWS X-Ray

Integrations based on CloudWatch Logs, forwarded to New Relic via Lambda:

  • AWS RDS Enhanced
  • AWS VPC Flow Logs

Metrics not available with CloudWatch metric streams

AWS CloudWatch metric stream will not include metrics that are made available to CloudWatch with more than 2 hours delay. Examples of AWS namespaces that might contain metrics that are aggregated and exposed after 2 hours include: AWS DMS, AWS RDS, AWS DocDB, AWS S3 and AWS DAX.
Please refer to AWS documentation to learn more.

Copyright © 2022 New Relic Inc.