New Relic includes an integration for monitoring Amazon Kinesis Data Streams. This document explains how to activate the integration and describes the data that can be reported.
Features
Amazon Kinesis Data Streams is a platform for streaming data on AWS, making it easy to load and analyze data in real time. It also gives you the ability to build custom streaming data applications for specialized needs.
New Relic's Kinesis Data Streams integration gathers metric and configuration data on all of the streams associated with your account.
Data collected includes bytes put to and retrieved from the stream, records put and retrieved, time taken by operations, and other metrics. You can view your Kinesis Data Streams data in pre-built dashboards and create custom queries and charts in New Relic.
You also have an option for enabling shard data collection. AWS data records are sequentially processed in shards, which are in turn grouped in streams. The number of shards in a stream determines the total data read and write capacity of the service.
Activate integration
To enable this integration follow standard procedures to Connect AWS services to New Relic.
Polling and configuration
You can change the polling frequency and filter data using configuration options.
Default polling information for the Amazon Kinesis Data Streams integration:
- New Relic polling interval: 15 minutes
- Amazon CloudWatch data interval: 1 minute
Shard data: To collect shard data, enable Collect shards data in the integration's configuration options in the UI.
If you enable collecting shard data, this will increase the number of API calls to your AWS CloudWatch service. There will be seven calls per shard per polling interval.
Find and use data
To find your integration data, go to one.newrelic.com > Infrastructure > AWS and select one of the Kinesis Data Streams integration links.
You can query and explore your data using the QueueSample event type. The provider value includes:
- Stream metrics:
KinesisStream - Shard metrics:
KinesisStreamShard
For more on how to use your data, see Understand and use integration data.
Metric data
This integration collects the following metrics. For additional information, see Amazon's documentation for monitoring Amazon Kinesis Data Streams.
This integration collects the following metrics:
| Name | Description |
|---|---|
getRecordsBytes |
The number of bytes retrieved from the Kinesis stream, measured over the specified time period. Minimum, Maximum, and Average statistics represent the bytes in a single GetRecords operation for the stream in the specified time period. |
getRecordsIteratorAgeMilliseconds |
The age of the last record in all GetRecords calls made against an Kinesis stream, measured over the specified time period, in milliseconds. Age is the difference between the current time and when the last record of the GetRecords call was written to the stream. The Minimum and Maximum statistics can be used to track the progress of Kinesis consumer applications. A value of zero indicates that the records being read are completely caught up with the stream. |
getRecordsLatency |
The time taken in milliseconds per GetRecords operation, measured over the specified time period. |
getRecordsRecords |
The number of records retrieved from the shard, measured over the specified time period. Minimum, Maximum, and Average statistics represent the records in a single GetRecords operation for the stream in the specified time period. |
incomingBytes |
The number of bytes successfully put to the Kinesis stream over the specified time period. This metric includes bytes from PutRecord and PutRecords operations. Minimum, Maximum, and Average statistics represent the bytes in a single put operation for the stream in the specified time period. |
incomingRecords |
The number of records successfully put to the Kinesis stream over the specified time period. This metric includes record counts from PutRecord and PutRecords operations. Minimum, Maximum, and Average statistics represent the records in a single put operation for the stream in the specified time period. |
putRecordBytes |
The number of bytes put to the Kinesis stream using the PutRecord operation over the specified time period. |
putRecordLatency |
The time taken in milliseconds per PutRecord operation, measured over the specified time period. |
putRecordSuccess |
The number of successful PutRecord operations per Kinesis stream, measured over the specified time period. Average reflects the percentage of successful writes to a stream. |
putRecordsBytes |
The number of bytes put to the Kinesis stream using the PutRecords operation over the specified time period. |
putRecordsLatency |
The time taken in milliseconds per PutRecords operation, measured over the specified time period. |
putRecordsRecords |
The number of successful records in a PutRecords operation per Kinesis stream, measured over the specified time period. |
putRecordsSuccess |
The number of PutRecords operations where at least one record succeeded, per Kinesis stream, measured over the specified time period. |
readProvisionedThroughputExceeded |
The number of GetRecords calls throttled for the stream over the specified time period. The most commonly used statistic for this metric is Average. |
writeProvisionedThroughputExceeded |
The number of records rejected due to throttling for the stream over the specified time period. This metric includes throttling from When the Minimum statistic has a non-zero value, records were being throttled for the stream during the specified time period. When the Maximum statistic has a value of 0 (zero), no records were being throttled for the stream during the specified time period. |
Inventory data
Tags (indicated with an *) are only fetched when tags collection is on.
aws/kinesis/stream
-
Name Description awsRegionThe AWS region that the stream was provisioned in.
streamNameThe name of the stream.
statusThe current status of the stream being described (CREATING, DELETING, ACTIVE, UPDATING).
retentionPeriodHoursThe current retention period, in hours.
openShardsCountNumber of opened shards.
closedShardsCountNumber of closed shards.
tags*Stream tags.
aws/kinesis/shard
-
Name Description awsRegionThe AWS region that the shard was provisioned in.
streamNameThe name of the stream.
shardIdThe unique identifier of the shard within the stream.
parentShardIdThe shard ID of the shard's parent.
adjacentParentShardIdThe shard ID of the shard adjacent to the shard's parent.
startingHashKeyThe starting hash key of the hash key range.
endingHashKeyThe ending hash key of the hash key range.
startingSequenceNumberThe starting sequence number for the range.
endingSequenceNumberThe ending sequence number for the range. Shards that are in the OPEN state have an ending sequence number of null.
tags*Shard tags.