• /
  • Log in

Amazon CloudWatch Metric Streams integration

New Relic currently provides independent integrations with AWS to collect performance metrics and metadata for more than 50 AWS services. With the new AWS Metric Streams integration, you only need a single service, AWS CloudWatch, to gather all AWS metrics and custom namespaces and send them to New Relic.

Why does this matter?

Our current system, which relies on individual integrations, runs on a polling fleet and calls multiple AWS APIs at regular intervals to retrieve the metrics and metadata. Using AWS CloudWatch significantly improves how metrics are gathered, overcoming some of the limitations of using the individual integrations.

API mode

Stream mode

It requires an integration with each AWS service to collect the metrics.

All metrics from all AWS services and custom namespaces are available in New Relic at once, without needing a specific integration to be built or updated.

  • There are two exceptions: percentiles and a small number of metrics that are made available to CloudWatch with more than 2 hours delay.

It adds an additional delay to metrics being available in New Relic for alerting and dashboarding. The fastest polling interval we offer today is 5 minutes.

Latency is significantly improved, since metrics are streamed in less than two minutes since they are made available in AWS CouldWatch.

It may lead to AWS API throttling for large AWS environments.

AWS API throttling is eliminated.

Set up a Metric Stream to send CloudWatch metrics to New Relic

To stream CloudWatch metrics to New Relic you need to create Kinesis Data Firehose and point it to New Relic and then create a CloudWatch Metric Stream that sends metrics to that Firehose.

How to map New Relic and AWS accounts and regions

  • If you manage multiple AWS accounts, then each account needs to be connected to New Relic.
  • If you manage multiple regions within those accounts, then each region needs to be configured with a different Kinesis Data Firehose pointing to New Relic.
  • You will typically map one or many AWS accounts to a single New Relic account.

Guided setup using CloudFormation

First, you need to link each of your AWS accounts with your New Relic account. To do so:

Next, set up the metric stream using the CloudFormation template we provide in the last step of our UI.

Manual setup using AWS Console, API, or calls

  1. Create a Kinesis Data Firehose Delivery Stream and configure the following destination parameters:
  • Source: Direct PUT or other sources
  • Data transformation: Disabled
  • Record format conversion: Disabled
  • Destination: Third-party service provider
  • Ensure the following settings are defined:
    • Third-party service provider: New Relic - Metrics
    • New Relic configuration
      • HTTP endpoint URL - US Datacenter: https://aws-api.newrelic.com/cloudwatch-metrics/v1
      • HTTP endpoint URL - EU Datacenter: https://aws-api.eu01.nr-data.net/cloudwatch-metrics/v1
      • API key: Enter your license key
      • Content encoding: GZIP
      • Retry duration: 60
    • S3 backup mode: Failed data only
    • S3 bucket: select a bucket or create a new one to store metrics that failed to be sent.
    • New Relic buffer conditions
      • Buffer size: 1 MB
      • Buffer interval: 60 (seconds)
    • Permissions IAM role:
      • Create or update IAM role
  1. Create the metric stream.
  • Go to CloudWatch service in your AWS console and select the Streams option under the Metrics menu.

  • Click on Create metric stream.

  • Determine the right configuration based on your use cases:

    • Use inclusion and exclusion filters to select which services should push metrics to New Relic.
    • Select your Kinesis Data Firehose.
    • Define a meaningful name for the stream (for example, newrelic-metric-stream).
  • Change default output format to Open Telemetry 0.7 (JSON is not supported)

  • Confirm the creation of the metric stream.

    Alternatively, you can find instructions on the AWS documentation in order to create the CloudWatch metric stream using a CloudFormation template, API, or the CLI.

  1. Add the new AWS account in the Metric streams mode in the New Relic UI. Go to one.newrelic.com > Infrastructure > AWS, click on Add an AWS account, then on Use metric streams, and follow the steps.

Tip

The following are the minimal permissions that should be granted on the AWS role configured in New Relic so that CloudWatch metrics can be enriched with additional service metadata and custom tags when applicable:

config:BatchGetResourceConfig
config:ListDiscoveredResources
tag:GetResources

The New Relic UI currently recommends the ReadOnlyAccess policy over these individual items so that New Relic has proper permissions to collect service data that's not available in AWS CloudWatch Metric Streams.

Validate your data is received correctly

To confirm you are receiving data from the Metric Streams, follow the steps below:

  1. Go to one.newrelic.com > Infrastructure > AWS, and search for the Stream accounts.
  2. You can check the following:
  • Account status dashboard. Useful to confirm that metric data is being received (errors, number of namespaces/metrics ingested, etc.)
  • Explore your data. Use the Data Explorer to find a specific set of metrics, access all dimensions available for a given metric and more.

Metrics naming convention

Metrics received from AWS CloudWatch are stored in New Relic as dimensional metrics following this convention:

  • Metrics are prefixed by the AWS namespace, all lowercase, where / is replaced with . :
    • AWS/EC2 -> aws.ec2
    • AWS/ApplicationELB -> aws.applicationelb
  • The original AWS metric name with its original case:
    • aws.ec2.CPUUtilization
    • aws.s3.5xxErrors
    • aws.sns.NumberOfMessagesPublished
  • If the resource the metric belongs to has a specific namespace prefix, it is used. If the resource the metric belongs to doesn't have a specific namespace prefix, metrics use the aws. prefix.
    • aws.Region
    • aws.s3.BucketName

Current namespaces supported by AWS can be found in the CloudWatch documentation website.

Query Experience, metric storage and mapping

Metrics coming from AWS CloudWatch are stored as dimensional metrics of type summary and can be queried using NRQL.

We have mapped metrics from the current cloud integrations to the new mappings that will come from AWS Metric Streams. You can continue to use the current metric naming, and queries will continue to work and pick data from AWS Metric Streams and the current cloud integrations.

Check our documentation on how current cloud integrations metrics map to the new metric naming.

All metrics coming from the metric stream will have these attributes:

  • aws.MetricStreamArn
  • collector.name = ‘cloudwatch-metric-streams’.

AWS namespaces' entities in the New Relic Explorer

We generate New Relic entities for most used AWS namespaces and will continue adding support for more namespaces.

When we generate New Relic entities for a namespace you can expect to:

  • Browse those entities in the New Relic Explorer.
  • Access an out-of-the-box entity dashboard for those entities.
  • Get metrics and entities from that namespace decorated with AWS tags. Collecting AWS tags requires that you have given New Relic the tag:GetResources permission which is part of the setup process in the UI. AWS tags show in metrics as tag.AWSTagName; for example, if you have set a Team AWS tag on the resource, it will show as tag.Team.
  • Leverage all the built-in features that are part of the Explorer.

Important

Lookout view in Entity Explorer is not compatible with entities created from the AWS Metric Streams integration at this time.

Set alert conditions

You can create NRQL alert conditions on metrics from a metric stream. Make sure your filter limits data to metrics from the CloudWatch metric stream only. To do that, construct your queries like this:

SELECT sum('aws.s3.5xxErrors') FROM Metric WHERE collector.name = 'cloudwatch-metric-streams' FACET aws.accountId, aws.s3.BucketName

Then, to make sure that alerts processes the data correctly, configure the advanced signal settings. These settings are needed because AWS CloudWatch receives metrics from services with a certain delay (for example, Amazon guarantees that 90% of EC2 metrics are available in CloudWatch within 7 minutes of them being generated). Moreover, streaming metrics from AWS to New Relic adds up to 1 minute additional delay, mostly due to buffering data in the Firehose.

To configure the signal settings, under Condition Settings, click on Advanced Signal Settings and enter the following values:

  1. Aggregation window. We recommend setting it to 1 minute. If you are having issues with flapping alerts or alerts not triggering, consider increasing it to 2 minutes.
  2. Offset evaluation by. Depending on the service, CloudWatch may send metrics with a certain delay. The value is set in windows. With a 1-minute aggregation window, setting the offset to 8 ensures the majority of the metrics are evaluated correctly. You may be able to use a lower offset if the delay introduced by AWS and Firehose is less.
  3. Fill data gaps with. Leave this void, or use Last known value if gaps in the data coming from AWS lead to false positives or negatives.

See our documentation on how to create NRQL alerts for more details.

Tags collection

New Relic provides enhanced dimensions from metrics coming from AWS CloudWatch metric streams. Resource and custom tags are automatically pulled from all services and are used to decorate metrics with additional dimensions. Use the data explorer to see which tags are available on each AWS metric.

The following query shows an example of tags being collected and queried as dimensions in metrics:

SELECT average(`aws.rds.CPUUtilization`) FROM Metric FACET `tags.mycustomtag` SINCE 30 MINUTES AGO TIMESERIES

Metadata collection

Like with custom tags, New Relic also pulls metadata information from relevant AWS services in order to decorate AWS CloudWatch metrics with enriched metadata collected from AWS Services APIs. This metadata is accessible in New Relic as additional dimensions on the metrics provided by AWS CloudWatch. This is an optional capability that's complementary to the CloudWatch Metric Streams integration.

The solution relies on AWS Config, which might incur in additional costs in your AWS account. AWS Config provides granular controls to determine which services and resources are recorded. New Relic will only ingest metadata from the available resources in your AWS account.

The following services / namespaces are supported:

  • EC2
  • Lambda
  • RDS
  • ALB/NLB
  • S3
  • API Gateway (excluding API v1)
  • ELB
  • EBS
  • DynamoDB
  • ECS

Curated dashboards

A set of dashboards for different AWS Services is available in the New Relic One Quickstarts app.

Get access to the Quickstarts App

Follow these steps in order to browse and import dashboards:

  1. Navigate to the Apps catalog in New Relic One.
  2. Search and select the Quickstarts app.
  3. Click on the Add this app link in the top-right corner.
  4. To enable the app, select target accounts and confirm clicking the Update account button.
  5. If applicable, review and confirm the Terms and Conditions of Use.
  6. Note that it might take a few minutes until permissions are applied and the app is ready to be used. Once available, confirm the app is enabled. The Quickstarts app will be listed in the Apps catalog.

Import dashboards from Quickstarts App

Follow these steps in order to import any of the curated dashboards:

  1. Navigate to Apps and open the Quickstarts app.
  2. Browse the AWS Dashboards. If you don't find a dashboard for your AWS service please submit your feedback on the top navigation bar or feel free to contribute with your own dashboard.
  3. Select the dashboard, confirm the target account and initial dashboard name.
  4. A copy of the dashboard should be imported in the target account. Additional customization is possible using any of the New Relic One dashboarding features.

Manage your data

New Relic provides a set of tools to keep track of the data being ingested in your account. Go to Manage your data in the settings menu to see all details. Metrics ingested from AWS Metric Streams integrations are considered in the Metric bucket.

If you need a more granular view of the data you can use the bytecountestimate() function on Metric in order to estimate the data being ingested. For example, the following query represents data ingested from all metrics processed via AWS Metric Streams integration in the last 30 days (in bytes):

FROM Metric SELECT bytecountestimate() where collector.name='cloudwatch-metric-streams' since 30 day ago

We recommend the following actions to control the data being ingested:

  • Make sure metric streams are enabled only on the AWS accounts and regions you want to monitor with New Relic.
  • Use the inclusion and exclusion filters in the CloudWatch Metric Stream in order to select which services / namespaces are being collected.
  • Consider using drop data rules to discard metrics based on custom filters (for example, drop metrics by namespace and tag, tag value, or any other valid NRQL criteria).

Important

Metrics sent via AWS Metric Streams count against your Metric API limits for the New Relic account where data will be ingested.

Migrating from poll-based AWS integrations

When metrics are sent via Metric Streams to New Relic, if the same metrics are being retrieved using the current poll-based integrations, those metrics will be duplicated. For example, alerts and dashboards that use sum or count will return twice the actual number. This includes alerts and dashboards that use metrics that have a .Sum suffix.

We recommend sending the data to a non-production New Relic account where you can safely do tests. If that is not an option, then AWS CloudWatch Metric Stream filters are available to include or exclude certain namespaces that can cause trouble.

Alternatively, you can use filtering on queries to distinguish between metrics that come from Metric Streams and those that come through polling. All metrics coming from Metric Streams are tagged with collector.name='cloudwatch-metric-streams'.

Query, dashboard, and alert considerations

AWS Metric Streams integration uses the Metric API to push metrics in the dimensional metric format.

Poll-based integrations push metrics based on events (for example, ComputeSample event), and will be migrated to dimensional metrics in the future.

To assist in this transition, New Relic provides a mechanism (known as shimming) that transparently lets you write queries in any format. Then these queries are processed as expected based on the source that's available (metrics or events). This mechanism works both ways, from events to metrics, and viceversa.

Please consider the following when migrating from poll-based integrations:

  • Custom dashboards that use poll-based AWS integration events should work as expected.
  • Alert conditions that use poll-based AWS events might need to be adapted with dimensional metric format. Use the NRQL source for the alert condition.
  • Entities in New Relic Explorer might be duplicated for up to 24 hours.
  • The AWS CloudWatch Metric Streams integration only collects metrics. Integrations such as AWS Health, AWS Billing, and AWS CloudTrail collect additional instrumentation that's not available for metric streams. In order to continue collecting that additional data, keep those enabled as polling integrations.

Troubleshooting

No metrics or errors appear on New Relic

If you are not seeing data in New Relic once the AWS CloudWatch Metric Stream has been connected to AWS Kinesis Data Firehose, then follow the steps below to troubleshoot your configuration:

  1. Check that the Metric Stream is in a state of Running via the AWS console or API. Please refer to AWS Troubleshooting docs for additional details.
  2. Check the Metric Stream metrics under AWS/CloudWatch/MetricStreams namespace. You will see a count of metric updates and errors per Metric Streams. This will indicate that the Metric Stream is successfully emitting data.
  3. If you see errors, confirm the IAM role specified in the Metric Streams configuration grants the CloudWatch service principal permissions to write to it.
  4. Check the Monitoring tab of the Kinesis Data Firehose in the Kinesis console to see if the Firehose is successfully receiving data.
  5. You can enable CloudWatch error logging on your Kinesis Data Firehose to get more detailed information for debugging issues. Refer to Amazon Kinesis Data Firehose official documentation for more details.
  6. Confirm that you have configured your Kinesis Data Firehose with the correct destination details:
  • Ensure the New Relic API Key/License Key contains your 40 hexadecimal chars license key.
  • Ensure the right data center US or EU has been selected for your New Relic account (hint: if the license_key starts with “eu” then you need to select the EU data center).
  1. Check that your Kinesis Data Firehose has permissions to write to the configured destination, for example: the S3 bucket policy allows write.

Missing metrics for certain AWS namespaces

New Relic does not apply any filter on the metrics received from the AWS CloudWatch metric stream.

If you are expecting certain metrics to be ingested and it’s not the case, please verify the following:

  • Make sure there’s no Inclusion or Exclusion filter in your CloudWatch Metric Stream.
  • Make sure metrics are available in AWS as part of CloudWatch. Confirm you see the metrics in the AWS CloudWatch interface.

Important

AWS CloudWatch doesn't include metrics that are not available in less than 2 hours. For example, some S3 metrics are aggregated on a daily basis. We plan to make some of these special metrics available in New Relic.

Metric values discrepancies between AWS CloudWatch and New Relic

Metrics are processed, mapped, and stored as received from AWS CloudWatch metric stream. Some discrepancies might be observed when comparing AWS CloudWatch and New Relic dashboards. On limited scenarios, AWS CloudWatch applies specific functions and logic before rendering the metrics.

These guidelines should help understand the root cause of the discrepancy:

  • Check that the same function is used on the metrics (for example average, min, max).
  • On the New Relic side, make sure you filter the same timestamp or timeframe (considering the timezone) to show the exact same time as in AWS CloudWatch.
  • When using timeseries, the New Relic user interface might perform some rounding based on intervals.

You can get a list of the raw metric received by time using a query like this one (note that no function is applied to the selected metric):

FROM Metric
SELECT aws.outposts.InstanceTypeCapacityUtilization
WHERE collector.name = 'cloudwatch-metric-streams'

Remember that AWS fixes the maximum resolution for every metric reported in AWS CloudWatch (for example: 1 minute, 5 minutes, etc).

AWS Metric Streams Operation

You can see the state of the Metric Stream(s) in the Streams tab in the CloudWatch console. In particular, a Metric Stream can be in one of two states: running or stopped.

  • Running: The stream is running correctly. Note that there may not be any metric data been streamed due to the configured filters. Although there is no data corresponding to the configured filters, the status of the Metric Stream can be running still.
  • Stopped: The stream has been explicitly set to the halted state (not because of an error). This state is useful to temporarily stop the streaming of data without deleting the configuration.

Errors in the Status Dashboard

New Relic relies on the AWS Config service to collect additional metadata from resources in order to enrich metrics received via CloudWatch Metric Stream.

Make sure AWS Config is enabled in your AWS Account, and ensure the linked Role has the following permission or inline policy created:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "config:BatchGetResourceConfig",
"Resource": "*"
}
]
}

For more help

If you need more help, check out these support and learning resources:

Create issueEdit page
Copyright © 2021 New Relic Inc.