AWS Kinesis Data Analytics monitoring integration

Our Infrastructure monitoring integrations include an integration for reporting your AWS Kinesis Data Analytics data to our products. Read on to learn more about how to activate the integration and what data it collects.

Activate integration

To enable this integration follow standard procedures to Connect AWS services to Infrastructure.

Configuration and polling

You can change the polling frequency and filter data using configuration options.

Default polling information for the AWS Kinesis Data Analytics integration:

  • New Relic polling interval: 5 minutes
  • Amazon CloudWatch data interval: 1 minute

Find and use data

To find your integration data in Infrastructure, go to infrastructure.newrelic.com > AWS and select an integration.

Data is attached to the following event type:

Entity Event Type Provider
Application AwsKinesisAnalyticsApplicationSample AwsKinesisAnalyticsApplication
ApplicationKpu AwsKinesisAnalyticsApplicationSample AwsKinesisAnalyticsApplication
Task AwsKinesisAnalyticsTaskSample AwsKinesisAnalyticsTask
TaskOperator AwsKinesisAnalyticsTaskOperatorSample AwsKinesisAnalyticsTaskOperator
TaskParallelism AwsKinesisAnalyticsTaskParallelismSample AwsKinesisAnalyticsTaskParallelism
TaskOperatorParallelism AwsKinesisAnalyticsTaskOperatorParallelismSample AwsKinesisAnalyticsTaskOperatorParallelism

For more on how to use your data, see Understand and use integration data.

Metric data

This integration collects AWS Kinesis Data Analytics data for Application, ApplicationKpu, Task, TaskOperator, TaskParallelism and TaskOperatorParallelism.

Kinesis Data Analytics Application data

Metric Unit Description

bytesConsumedRate

BytesPerSecond The average number of bytes consumed per second for a topic.

commitsFailed

Count The total number of offset commit failures to Kafka, if offset committing and checkpointing are enabled.

commitsSucceeded

Count The total number of successful offset commits to Kafka, if offset committing and checkpointing are enabled.

cpuUtilization

Percent Overall percentage of CPU utilization across task managers. For example, if there are five task managers, Kinesis Data Analytics publishes five samples of this metric per reporting interval.

currentInputWatermark

Milliseconds The last watermark this application/operator/task/thread has received.

currentOutputWatermark

Milliseconds The last watermark this application/operator/task/thread has emitted.

downtime

Milliseconds For jobs currently in a failing/recovering situation, the time elapsed during this outage.

fullRestarts

Count The total number of times this job has fully restarted since it was submitted. This metric does not measure fine-grained restarts.

heapMemoryUtilization

Percent Overall heap memory utilization across task managers. For example, if there are five task managers, Kinesis Data Analytics publishes five samples of this metric per reporting interval.

lastCheckpointDuration

Milliseconds The time it took to complete the last checkpoint.

lastCheckpointSize

Bytes The total size of the last checkpoint.

numRecordsInPerSecond

CountPerSecond The total number of records this operator or task has received per second.

numRecordsOut

Count The total number of records this operator or task has emitted.

numRecordsOutPerSecond

CountPerSecond The total number of records this operator or task has emitted per second.

numberOfFailedCheckpoints

Count The number of times checkpointing has failed.

numRecordsIn

Count The total number of records this operator or task has received.

oldGenerationGcCount

Count The total number of old garbage collection operations that have occurred across all task managers.

oldGenerationGcTime

Milliseconds The total time spent performing old garbage collection operations.

recordsLagMax

Count The maximum lag in terms of number of records for any partition in this window.

threadsCount

Count The total number of live threads used by the application.

uptime

Milliseconds The time that the job has been running without interruption.

bytes

Bytes The number of bytes read (per input stream) or written (per output stream).

inputProcessing.DroppedRecords

Count The number of records returned by a Lambda function that were marked with Dropped status.

inputProcessing.Duration

Milliseconds The time taken for each AWS Lambda function invocation performed by Kinesis Data Analytics.

inputProcessing.OkBytes

Count The sum of bytes of the records returned by a Lambda function that were marked with Ok status.

inputProcessing.OkRecords

Count The number of records returned by a Lambda function that were marked with Ok status.

inputProcessing.ProcessingFailedRecords

Count The number of records returned by a Lambda function that were marked with ProcessingFailed status.

inputProcessing.Success

Count The number of successful Lambda invocations by Kinesis Data Analytics.

lambdaDelivery.DeliveryFailedRecords

Count The number of records returned by a Lambda function that were marked with DeliveryFailed status.

lambdaDelivery.Duration

Milliseconds The time taken for each Lambda function invocation performed by Kinesis Data Analytics.

lambdaDelivery.OkRecords

Count The number of records returned by a Lambda function that were marked with Ok status.

millisBehindLatest

Milliseconds Indicates how far behind from the current time an application is reading from the streaming source.

records

Count The number of records read (per input stream) or written (per output stream).

success

Count 1 for each successful delivery attempt to the destination configured for your application; 0 for every failed delivery attempt. The average value of this metric indicates how many successful deliveries are performed.

bytesRequestedPerFetch

Bytes The bytes requested in a single call to getRecords.

committedOffsets

Other The last successfully committed offsets to Kafka, for each partition. A particular partition's metric can be specified by topic name and partition id.

currentOffsets

Other The consumer's current read offset, for each partition. A particular partition's metric can be specified by topic name and partition id.

Kinesis Data Analytics ApplicationKpu data

Metric Unit Description

KPUs

Count

The number of Kinesis Processing Units (KPUs) currently in use.

Known issue: as this metric has a longer lag than the rest, in some cases, it's not shown

Kinesis Data Analytics Task data

Metric Unit Description

commitsFailed

Count The total number of offset commit failures to Kafka, if offset committing and checkpointing are enabled.

commitsSucceeded

Count The total number of successful offset commits to Kafka, if offset committing and checkpointing are enabled.

currentInputWatermark

Milliseconds The last watermark this task has received.

currentOutputWatermark

Milliseconds The currently output watermark.

numLateRecordsDropped

Count The number of records this task has dropped due to arriving late.

numRecordsIn

Count The total number of records this task has received.

numRecordsInPerSecond

CountPerSecond The total number of records this task has received per second.

numRecordsOut

Count The total number of records this task has emitted.

numRecordsOutPerSecond

CountPerSecond The total number of records this task has emitted per second.

recordsLagMax

Count The maximum lag in terms of number of records for any partition in this window.

Kinesis Data Analytics TaskOperator data

Metric Unit Description

bytesConsumedRate

BytesPerSecond The average number of bytes consumed per second for a topic.

commitsFailed

Count The total number of offset commit failures to Kafka, if offset committing and checkpointing are enabled.

commitsSucceeded

Count The total number of successful offset commits to Kafka, if offset committing and checkpointing are enabled.

currentInputWatermark

Milliseconds The last watermark this task has received.

currentOutputWatermark

Milliseconds The last watermark this application/operator/task/thread has emitted.

numLateRecordsDropped

Count The number of records this operator or task has dropped due to arriving late.

numRecordsIn

Count The total number of records this task has received.

numRecordsInPerSecond

CountPerSecond The total number of records this task has received per second.

numRecordsOut

Count The total number of records this task has emitted.

numRecordsOutPerSecond

CountPerSecond The total number of records this task has emitted per second.

recordsLagMax

Count The maximum lag in terms of number of records for any partition in this window.

Kinesis Data Analytics TaskParallelism data

Metric Unit Description

currentInputWatermark

Milliseconds The last watermark this application/operator/task/thread has received.

numLateRecordsDropped

Count The number of records this task has dropped due to arriving late.

numRecordsIn

Count The total number of records this task has received.

numRecordsInPerSecond

CountPerSecond The total number of records this task has received per second.

numRecordsOut

Count The total number of records this task has emitted.

numRecordsOutPerSecond

CountPerSecond The total number of records this task has emitted per second.

Kinesis Data Analytics TaskOperatorParallelism data

Metric Unit Description

bytesConsumedRate

BytesPerSecond The average number of bytes consumed per second for a topic.

commitsFailed

Count The total number of offset commit failures to Kafka, if offset committing and checkpointing are enabled.

commitsSucceeded

Count The total number of successful offset commits to Kafka, if offset committing and checkpointing are enabled.

currentInputWatermark

Milliseconds The last watermark this application/operator/task/thread has received.

currentOutputWatermark

Milliseconds The last watermark this application/operator/task/thread has emitted.

numRecordsIn

Count The total number of records this operator or task has received.

numRecordsInPerSecond

CountPerSecond The total number of records this operator or task has received per second.

numRecordsOut

Count The total number of records this operator or task has emitted.

numRecordsOutPerSecond

CountPerSecond The total number of records this operator or task has emitted per second.

recordsLagMax

Count The maximum lag in terms of number of records for any partition in this window.

For more help

If you need more help, check out these support and learning resources: