• Log inStart now

No data appears: AWS CloudWatch metric streams

Problem

You've followed the steps to link your AWS account, configured the AWS CloudWatch metrics stream and AWS Kinesis Data Firehose, and you still don't see the expected metrics in New Relic.

Solutions

No metrics or errors appear on New Relic

If you are not seeing data in New Relic once the AWS CloudWatch Metric Stream has been connected to AWS Kinesis Data Firehose, then follow the steps below to troubleshoot your configuration:

  1. Check that the Metric Stream is in a state of Running via the AWS console or API. Please refer to AWS Troubleshooting docs for additional details.
  2. Check the Metric Stream metrics under AWS/CloudWatch/MetricStreams namespace. You will see a count of metric updates and errors per Metric Streams. This indicates that the Metric Stream is successfully emitting data.
  3. If you see errors, confirm the IAM role specified in the Metric Streams configuration grants the CloudWatch service principal permissions to write and targets the intended Firehose resource.
  4. Check the Monitoring tab of the Kinesis Data Firehose in the Kinesis console to see if the Firehose is successfully receiving data.
  5. You can enable CloudWatch error logging on your Kinesis Data Firehose to get more detailed information for debugging issues. Refer to Amazon Kinesis Data Firehose official documentation for more details.
  6. Confirm that you have configured your Kinesis Data Firehose with the correct destination details:
  • Ensure the New Relic API Key/License Key contains your 40 hexadecimal character license key.
  • Ensure the right data center US or EU has been selected for your New Relic account. Tip: If the license_key starts with “eu” then you need to select the EU data center.
  1. Check that your Kinesis Data Firehose has permissions to write to the configured destination. For example, the S3 bucket policy allows to write.

Missing metrics for certain AWS namespaces

New Relic doesn't apply any filter on the metrics received from the AWS CloudWatch metric stream.

If you're expecting certain metrics to be ingested and they aren't, verify the following:

  • Make sure there’s no Inclusion or Exclusion filter in your CloudWatch Metric Stream.
  • Make sure metrics are available in AWS CloudWatch and can be queried in the AWS CloudWatch interface. For some specific AWS services, such as ECS/ EKS container insights, enhanced monitoring needs to be explicitly enabled in AWS-side before getting access to the metrics.

Important

AWS CloudWatch doesn't include metrics that are not available in less than 2 hours. For example, some S3 metrics are aggregated on a daily basis. We plan to make some of these special metrics available in New Relic.

Metric values discrepancies between AWS CloudWatch and New Relic

Metrics are processed, mapped, and stored as received from AWS CloudWatch metric stream. Some discrepancies might be observed when comparing AWS CloudWatch and New Relic dashboards. On limited scenarios, AWS CloudWatch applies specific functions and logic before rendering the metrics.

These guidelines should help understand the root cause of the discrepancy:

  • Check that the same function is used on the metrics (for example average, min, max).
  • On the New Relic side, make sure you filter the same timestamp or timeframe (considering the time zone) to show the exact same time as in AWS CloudWatch.
  • When using timeseries, the New Relic user interface might perform some rounding based on intervals.

You can get a list of the raw metric received by time using a query like this one (note that no function is applied to the selected metric):

FROM Metric
SELECT aws.outposts.InstanceTypeCapacityUtilization
WHERE collector.name = 'cloudwatch-metric-streams'

Remember that AWS fixes the maximum resolution (1 minute, 5 minutes, etc.) for every metric reported in AWS CloudWatch.

AWS Metric Streams Operation

You can see the state of the Metric Stream(s) in the Streams tab in the CloudWatch console. In particular, a Metric Stream can be in one of two states: running or stopped.

  • Running: The stream is running correctly. Even if it's running, there may not be any metric data being streamed due to the configured filters.
  • Stopped: The stream has been explicitly set to the halted state (not because of an error). This state is useful to temporarily stop the streaming of data without deleting the configuration.

Errors in the Status Dashboard

New Relic relies on the AWS Config service to collect additional metadata from resources in order to enrich metrics received via CloudWatch Metric Stream.

Make sure AWS Config is enabled in your AWS Account, and ensure the linked Role has the following permission or inline policy created:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "config:BatchGetResourceConfig",
"Resource": "*"
}
]
}
Copyright © 2022 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.