Introduction to the Amazon CloudWatch Metric Streams integration

Our AWS CloudWatch Metric Streams integration is our primary and recommended solution for monitoring all CloudWatch metrics from all AWS services, including custom namespaces.

Learn how to set up AWS Metric Stream monitoring in New Relic.

Why does this matter?

Before CloudWatch Metric Streams, the only solution for AWS monitoring partners was to deploy a polling fleet and call multiple AWS APIs at regular intervals to retrieve the metrics and metadata. The following table shows the differences between both solutions:

API polling mode	Stream mode
It requires an integration with each AWS service to collect the metrics.	All CloudWatch metrics from all AWS services and custom namespaces are available in New Relic at once, without needing a specific integration to be built or updated. There's one exception: When metrics are available to AWS CloudWatch with more than 2 hours delay, these metrics aren't included in the stream.
It adds an additional delay to metrics being available in New Relic for alerting and dashboarding. The fastest polling interval is 5 minutes.	Latency is significantly improved, since metrics are streamed in less than two minutes since they are available in AWS CloudWatch.
It may lead to AWS API throttling for large AWS environments.	AWS API throttling is eliminated.

API polling mode

Stream mode

It requires an integration with each AWS service to collect the metrics.

All CloudWatch metrics from all AWS services and custom namespaces are available in New Relic at once, without needing a specific integration to be built or updated.

There's one exception: When metrics are available to AWS CloudWatch with more than 2 hours delay, these metrics aren't included in the stream.

It adds an additional delay to metrics being available in New Relic for alerting and dashboarding. The fastest polling interval is 5 minutes.

Latency is significantly improved, since metrics are streamed in less than two minutes since they are available in AWS CloudWatch.

It may lead to AWS API throttling for large AWS environments.

AWS API throttling is eliminated.

Want to try out our Amazon CloudWatch Metric Streams integration? Sign up with New Relic for free, forever!

Cost considerations

Consider the following when evaluating the cost of the AWS CloudWatch metric streams integration with New Relic:

AWS CloudWatch metric updates.
AWS Kinesis Firehose ingest.
AWS Kinesis Firehose data transfer.
Optional: AWS Config service used to enrich metrics with resource metadata on select AWS namespaces.

Tip

To detect, identify, and monitor your AWS services, we need access to your AWS Config Service. Without this access, we can't monitor and represent your systems.

Learn about the mechanisms available for data management, including filters in AWS and in New Relic. When applicable, make sure to complete an initial integration in a pre-production environment to evaluate the total cost of the solution based on a limited number of AWS resources and services.

Migrating from AWS API polling integrations

If you're a New Relic customer who currently has our older polling-based AWS integrations, we'll give you guidance on migrating to the new, improved integration.

Note that if you have both forms of integration set up, there can be duplicated metrics. For example, alerts and that use sum or count will return twice the actual number. This includes alerts and dashboards that use metrics that have a .Sum suffix.

We recommend sending the data to a non-production New Relic account where you can safely do tests. If that's not an option, then AWS CloudWatch Metric Stream filters are available to include or exclude certain namespaces that can cause trouble.

Alternatively, you can use filtering on queries to distinguish between metrics that come from Metric Streams and those that come through polling. All metrics coming from Metric Streams are tagged with collector.name='cloudwatch-metric-streams'.

Migration to Metric Streams

We recommend that existing customers migrate from API polling to Metric Streams first in a dev or staging environment. For typical deployments, follow these steps to migrate to AWS Cloudwatch Metric Streams:

Replicate the namespaces from polling to metric streams by going to one.newrelic.com > All capabilities > Infrastructure > AWS. Click on the Migrate to AWS Cloudwatch metric streams button, then configure your AWS Metric Streams account.
Download the customized CloudFormation template. This template contains the preconfigured namespaces that are based on your existing polling configuration.
Add your account details.

If you prefer not to use the CloudFormation template, here's an alternative option:

Go to one.newrelic.com > All capabilities > Infrastructure > AWS. Click on Add an AWS account, then add your AWS account. This step is required even if you've already linked your AWS account with polling integrations.
During the last onboarding step, ensure you've enabled AWS CloudWatch Metric Stream and the AWS Kinesis Data Firehose. This pushes metrics to New Relic. AWS CloudWatch requires one stream per region, so repeat this step for any additional AWS regions you want to monitor.
Ensure metrics are received from all connected regions and namespaces. This may take several minutes.
Disable all unnecessary polling integrations in the previous AWS provider account. Remember that some integrations still need to be enabled because they aren't fully replaced by Metric Streams.

Query, dashboard, and alert considerations

Our AWS Metric Streams integration uses the Metric API to push metrics in the dimensional metric format.

Our poll-based AWS integration pushes metrics based on events (for example, the ComputeSample event), and will be migrated to dimensional metrics in the future.

To assist in this transition, we provide a mechanism (known as shimming) that transparently lets you write queries in any format. Then these queries are processed as expected based on the source that's available (metrics or events). This mechanism works both ways, from events to metrics, and vice versa.

Tip

Learn more about the limitations of the query mechanism that allows customers to use event-based queries (samples) with the AWS CloudWatch Metric Stream integration (dimensional metric format).

Please consider the following when migrating from our poll-based integrations:

Dashboards: Custom that use poll-based AWS integration events will still work as expected.
Alerts: Alert conditions that use poll-based AWS events will still work. We recommend adapting those to the dimensional metric format (using NRQL as source).
Entities: Your list of monitored entities might show duplicated entities for up to 24 hours.
Attributes: Poll-based AWS integrations prefix collected resource tags with label., while the AWS CloudWatch Metric Streams integration prefixes collected resource tags with tags.. If both integrations are enabled for the same AWS account, resource tags will appear under both prefixes when using the Event format.

Integrations not fully replaced by metric streams

The AWS CloudWatch Metric Streams integration focuses on CloudWatch metrics. As a result, the following integrations still need to be configured and enabled to get complete visibility from AWS services.

Polling integrations based on service APIs:

AWS Billing
AWS CloudTrail
AWS Health
AWS Trusted Advisor
AWS X-Ray

Integrations based on CloudWatch Logs, forwarded to New Relic via Lambda:

AWS RDS Enhanced
AWS VPC Flow Logs

Metrics not available with CloudWatch metric streams

When metrics are available to AWS CloudWatch metric stream with more than 2 hours delay, these metrics aren't included in the stream.

Examples of AWS namespaces that might contain metrics that are aggregated and exposed after 2 hours include: AWS DMS, AWS RDS, AWS DocDB, AWS S3, and AWS DAX. See AWS documentation to learn more.