Amazon Kinesis Data Streams monitoring integration

Access to this feature depends on your subscription level. Requires Infrastructure Pro.

New Relic Infrastructure includes an integration for monitoring Amazon Kinesis Data Streams in New Relic. This document explains how to activate the integration and describes the data that can be reported.

Features

Amazon Kinesis Data Streams is a platform for streaming data on AWS, making it easy to load and analyze data in real time. It also gives you the ability to build custom streaming data applications for specialized needs.

New Relic Infrastructure's Kinesis Data Streams integration gathers metric and configuration data on all of the streams associated with your account.

Data collected includes bytes put to and retrieved from the stream, records put and retrieved, time taken by operations, and other metrics. You can view your Kinesis Data Streams data in pre-built dashboards and create custom queries and charts in New Relic Insights.

You also have an option for enabling shard data collection. AWS data records are sequentially processed in shards, which are in turn grouped in streams. The number of shards in a stream determines the total data read and write capacity of the service.

Activate integration

To enable this integration follow standard procedures to Connect AWS services to Infrastructure.

Polling and configuration

You can change the polling frequency and filter data using configuration options.

Default polling information for the Amazon Kinesis Data Streams integration:

  • New Relic polling interval: 15 minutes
  • Amazon CloudWatch data interval: 1 minute

Shard data: To collect shard data, enable Collect shards data in the integration's configuration options in the UI.

If you enable collecting shard data, this will increase the number of API calls to your AWS CloudWatch service. There will be seven calls per shard per polling interval.

Find and use data

To find your integration data in Infrastructure, go to infrastructure.newrelic.com > Integrations > Amazon Web Services and select one of the Kinesis Data Streams integration links.

In New Relic Insights, data is attached to the QueueSample event type. The provider value includes:

  • Stream metrics: KinesisStream
  • Shard metrics: KinesisStreamShard

For more on how to use your data, see Understand and use integration data.

Metric data

This integration collects the following metrics. For additonal information, see Amazon's documentation for monitoring Amazon Kinesis Data Streams.

This integration collects the following metrics:

Name Description
getRecordsBytes The number of bytes retrieved from the Kinesis stream, measured over the specified time period. Minimum, Maximum, and Average statistics represent the bytes in a single GetRecords operation for the stream in the specified time period.
getRecordsIteratorAgeMilliseconds The age of the last record in all GetRecords calls made against an Kinesis stream, measured over the specified time period, in milliseconds. Age is the difference between the current time and when the last record of the GetRecords call was written to the stream. The Minimum and Maximum statistics can be used to track the progress of Kinesis consumer applications. A value of zero indicates that the records being read are completely caught up with the stream.
getRecordsLatency The time taken in milliseconds per GetRecords operation, measured over the specified time period.
getRecordsRecords The number of records retrieved from the shard, measured over the specified time period. Minimum, Maximum, and Average statistics represent the records in a single GetRecords operation for the stream in the specified time period.
incomingBytes The number of bytes successfully put to the Kinesis stream over the specified time period. This metric includes bytes from PutRecord and PutRecords operations. Minimum, Maximum, and Average statistics represent the bytes in a single put operation for the stream in the specified time period.
incomingRecords The number of records successfully put to the Kinesis stream over the specified time period. This metric includes record counts from PutRecord and PutRecords operations. Minimum, Maximum, and Average statistics represent the records in a single put operation for the stream in the specified time period.
putRecordBytes The number of bytes put to the Kinesis stream using the PutRecord operation over the specified time period.
putRecordLatency The time taken in milliseconds per PutRecord operation, measured over the specified time period.
putRecordSuccess The number of successful PutRecord operations per Kinesis stream, measured over the specified time period. Average reflects the percentage of successful writes to a stream.
putRecordsBytes The number of bytes put to the Kinesis stream using the PutRecords operation over the specified time period.
putRecordsLatency The time taken in milliseconds per PutRecords operation, measured over the specified time period.
putRecordsRecords The number of successful records in a PutRecords operation per Kinesis stream, measured over the specified time period.
putRecordsSuccess The number of PutRecords operations where at least one record succeeded, per Kinesis stream, measured over the specified time period.
readProvisionedThroughputExceeded The number of GetRecords calls throttled for the stream over the specified time period. The most commonly used statistic for this metric is Average.
writeProvisionedThroughputExceeded

The number of records rejected due to throttling for the stream over the specified time period. This metric includes throttling from PutRecord and PutRecords operations. The most commonly used statistic for this metric is Average.

When the Minimum statistic has a non-zero value, records were being throttled for the stream during the specified time period.

When the Maximum statistic has a value of 0 (zero), no records were being throttled for the stream during the specified time period.

Inventory data

Tags (indicated with an *) are only fetched when tags collection is on.

aws/kinesis/stream

Name Description

awsRegion

The AWS region that the stream was provisioned in.

streamName

The name of the stream.

status

The current status of the stream being described (CREATING, DELETING, ACTIVE, UPDATING).

retentionPeriodHours

The current retention period, in hours.

openShardsCount

Number of opened shards.

closedShardsCount

Number of closed shards.

tags*

Stream tags.

aws/kinesis/shard

Name Description

awsRegion

The AWS region that the shard was provisioned in.

streamName

The name of the stream.

shardId

The unique identifier of the shard within the stream.

parentShardId

The shard ID of the shard's parent.

adjacentParentShardId

The shard ID of the shard adjacent to the shard's parent.

startingHashKey

The starting hash key of the hash key range.

endingHashKey

The ending hash key of the hash key range.

startingSequenceNumber

The starting sequence number for the range.

endingSequenceNumber

The ending sequence number for the range. Shards that are in the OPEN state have an ending sequence number of null.

tags*

Shard tags.

For more help

Recommendations for learning more: