• EnglishEspañol日本語한국어Português
  • Log inStart now

Amazon Kinesis Data Streams monitoring integration

Important

Enable the AWS CloudWatch Metric Streams integration to monitor all CloudWatch metrics from your AWS services, including custom namespaces. Individual integrations are no longer our recommended option.

New Relic includes an integration for monitoring Amazon Kinesis Data Streams. This document explains how to activate the integration and describes the data that can be reported.

Features

Amazon Kinesis Data Streams is a platform for streaming data on AWS, making it easy to load and analyze data in real time. It also gives you the ability to build custom streaming data applications for specialized needs.

New Relic's Kinesis Data Streams integration gathers metric and configuration data on all of the streams associated with your account.

Data collected includes bytes put to and retrieved from the stream, records put and retrieved, time taken by operations, and other metrics. You can view your Kinesis Data Streams data in pre-built dashboards and create custom queries and charts in New Relic.

You also have an option for enabling shard data collection. AWS data records are sequentially processed in shards, which are in turn grouped in streams. The number of shards in a stream determines the total data read and write capacity of the service.

Activate integration

To enable this integration follow standard procedures to Connect AWS services to New Relic.

Polling and configuration

You can change the polling frequency and filter data using configuration options.

Default polling information for the Amazon Kinesis Data Streams integration:

  • New Relic polling interval: 15 minutes
  • Amazon CloudWatch data interval: 1 minute

Shard data: To collect shard data, enable Collect shards data in the integration's configuration options in the UI.

Important

If you enable collecting shard data, this will increase the number of API calls to your AWS CloudWatch service. There will be seven calls per shard per polling interval.

Find and use data

To find your integration data, go to one.newrelic.com > All capabilities > Infrastructure > AWS and select one of the Kinesis Data Streams integration links.

You can query and explore your data using the QueueSample event type. The provider value includes:

  • Stream metrics: KinesisStream
  • Shard metrics: KinesisStreamShard

For more on how to use your data, see Understand and use integration data.

Metric data

This integration collects the following metrics. For additional information, see Amazon's documentation for monitoring Amazon Kinesis Data Streams.

This integration collects the following metrics:

Name

Description

getRecordsBytes

The number of bytes retrieved from the Kinesis stream, measured over the specified time period. Minimum, Maximum, and Average statistics represent the bytes in a single GetRecords operation for the stream in the specified time period.

getRecordsIteratorAgeMilliseconds

The age of the last record in all GetRecords calls made against an Kinesis stream, measured over the specified time period, in milliseconds. Age is the difference between the current time and when the last record of the GetRecords call was written to the stream. The Minimum and Maximum statistics can be used to track the progress of Kinesis consumer applications. A value of zero indicates that the records being read are completely caught up with the stream.

getRecordsLatency

The time taken in milliseconds per GetRecords operation, measured over the specified time period.

getRecordsRecords

The number of records retrieved from the shard, measured over the specified time period. Minimum, Maximum, and Average statistics represent the records in a single GetRecords operation for the stream in the specified time period.

incomingBytes

The number of bytes successfully put to the Kinesis stream over the specified time period. This metric includes bytes from PutRecord and PutRecords operations. Minimum, Maximum, and Average statistics represent the bytes in a single put operation for the stream in the specified time period.

incomingRecords

The number of records successfully put to the Kinesis stream over the specified time period. This metric includes record counts from PutRecord and PutRecords operations. Minimum, Maximum, and Average statistics represent the records in a single put operation for the stream in the specified time period.

putRecordBytes

The number of bytes put to the Kinesis stream using the PutRecord operation over the specified time period.

putRecordLatency

The time taken in milliseconds per PutRecord operation, measured over the specified time period.

putRecordSuccess

The number of successful PutRecord operations per Kinesis stream, measured over the specified time period. Average reflects the percentage of successful writes to a stream.

putRecordsBytes

The number of bytes put to the Kinesis stream using the PutRecords operation over the specified time period.

putRecordsLatency

The time taken in milliseconds per PutRecords operation, measured over the specified time period.

putRecordsRecords

The number of successful records in a PutRecords operation per Kinesis stream, measured over the specified time period.

putRecordsSuccess

The number of PutRecords operations where at least one record succeeded, per Kinesis stream, measured over the specified time period.

readProvisionedThroughputExceeded

The number of GetRecords calls throttled for the stream over the specified time period. The most commonly used statistic for this metric is Average.

writeProvisionedThroughputExceeded

The number of records rejected due to throttling for the stream over the specified time period. This metric includes throttling from PutRecord and PutRecords operations. The most commonly used statistic for this metric is Average.

When the Minimum statistic has a non-zero value, records were being throttled for the stream during the specified time period.

When the Maximum statistic has a value of 0 (zero), no records were being throttled for the stream during the specified time period.

Copyright © 2024 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.