• /
  • EnglishEspañol日本語한국어Português
  • Log inStart now

AWS Kinesis Data Analytics monitoring integration

Important

Enable the AWS CloudWatch Metric Streams integration to monitor all CloudWatch metrics from your AWS services, including custom namespaces. Individual integrations are no longer our recommended option.

Our infrastructure monitoring integrations include an integration for reporting your AWS Kinesis Data Analytics data to our products. Read on to learn more about how to activate the integration and what data it collects.

Activate integration

To enable this integration follow standard procedures to Connect AWS services to New Relic.

Configuration and polling

You can change the polling frequency and filter data using configuration options.

Default polling information for the AWS Kinesis Data Analytics integration:

  • New Relic polling interval: 5 minutes
  • Amazon CloudWatch data interval: 1 minute

Find and use data

To find your integration data in our infrastructure UI, go to one.newrelic.com > All capabilities > Infrastructure > AWS and select an integration.

Data is attached to the following event type:

Entity

Event Type

Provider

Application

AwsKinesisAnalyticsApplicationSample

AwsKinesisAnalyticsApplication

ApplicationKpu

AwsKinesisAnalyticsApplicationSample

AwsKinesisAnalyticsApplication

Task

AwsKinesisAnalyticsTaskSample

AwsKinesisAnalyticsTask

TaskOperator

AwsKinesisAnalyticsTaskOperatorSample

AwsKinesisAnalyticsTaskOperator

TaskParallelism

AwsKinesisAnalyticsTaskParallelismSample

AwsKinesisAnalyticsTaskParallelism

TaskOperatorParallelism

AwsKinesisAnalyticsTaskOperatorParallelismSample

AwsKinesisAnalyticsTaskOperatorParallelism

For more on how to use your data, see Understand and use integration data.

Metric data

This integration collects AWS Kinesis Data Analytics data for Application, ApplicationKpu, Task, TaskOperator, TaskParallelism and TaskOperatorParallelism.

Kinesis Data Analytics Application data

Metric

Unit

Description

bytesConsumedRate

BytesPerSecond

The average number of bytes consumed per second for a topic.

commitsFailed

Count

The total number of offset commit failures to Kafka, if offset committing and checkpointing are enabled.

commitsSucceeded

Count

The total number of successful offset commits to Kafka, if offset committing and checkpointing are enabled.

cpuUtilization

Percent

Overall percentage of CPU utilization across task managers. For example, if there are five task managers, Kinesis Data Analytics publishes five samples of this metric per reporting interval.

currentInputWatermark

Milliseconds

The last watermark this application/operator/task/thread has received.

currentOutputWatermark

Milliseconds

The last watermark this application/operator/task/thread has emitted.

downtime

Milliseconds

For jobs currently in a failing/recovering situation, the time elapsed during this outage.

fullRestarts

Count

The total number of times this job has fully restarted since it was submitted. This metric does not measure fine-grained restarts.

heapMemoryUtilization

Percent

Overall heap memory utilization across task managers. For example, if there are five task managers, Kinesis Data Analytics publishes five samples of this metric per reporting interval.

lastCheckpointDuration

Milliseconds

The time it took to complete the last checkpoint.

lastCheckpointSize

Bytes

The total size of the last checkpoint.

numRecordsInPerSecond

CountPerSecond

The total number of records this operator or task has received per second.

numRecordsOut

Count

The total number of records this operator or task has emitted.

numRecordsOutPerSecond

CountPerSecond

The total number of records this operator or task has emitted per second.

numberOfFailedCheckpoints

Count

The number of times checkpointing has failed.

numRecordsIn

Count

The total number of records this operator or task has received.

oldGenerationGcCount

Count

The total number of old garbage collection operations that have occurred across all task managers.

oldGenerationGcTime

Milliseconds

The total time spent performing old garbage collection operations.

recordsLagMax

Count

The maximum lag in terms of number of records for any partition in this window.

threadsCount

Count

The total number of live threads used by the application.

uptime

Milliseconds

The time that the job has been running without interruption.

bytes

Bytes

The number of bytes read (per input stream) or written (per output stream).

inputProcessing.DroppedRecords

Count

The number of records returned by a Lambda function that were marked with Dropped status.

inputProcessing.Duration

Milliseconds

The time taken for each AWS Lambda function invocation performed by Kinesis Data Analytics.

inputProcessing.OkBytes

Count

The sum of bytes of the records returned by a Lambda function that were marked with Ok status.

inputProcessing.OkRecords

Count

The number of records returned by a Lambda function that were marked with Ok status.

inputProcessing.ProcessingFailedRecords

Count

The number of records returned by a Lambda function that were marked with ProcessingFailed status.

inputProcessing.Success

Count

The number of successful Lambda invocations by Kinesis Data Analytics.

lambdaDelivery.DeliveryFailedRecords

Count

The number of records returned by a Lambda function that were marked with DeliveryFailed status.

lambdaDelivery.Duration

Milliseconds

The time taken for each Lambda function invocation performed by Kinesis Data Analytics.

lambdaDelivery.OkRecords

Count

The number of records returned by a Lambda function that were marked with Ok status.

millisBehindLatest

Milliseconds

Indicates how far behind from the current time an application is reading from the streaming source.

records

Count

The number of records read (per input stream) or written (per output stream).

success

Count

1 for each successful delivery attempt to the destination configured for your application; 0 for every failed delivery attempt. The average value of this metric indicates how many successful deliveries are performed.

bytesRequestedPerFetch

Bytes

The bytes requested in a single call to getRecords.

committedOffsets

Other

The last successfully committed offsets to Kafka, for each partition. A particular partition's metric can be specified by topic name and partition id.

currentOffsets

Other

The consumer's current read offset, for each partition. A particular partition's metric can be specified by topic name and partition id.

Kinesis Data Analytics ApplicationKpu data

Metric

Unit

Description

KPUs

Count

The number of Kinesis Processing Units (KPUs) currently in use.

Known issue: as this metric has a longer lag than the rest, in some cases, it's not shown

Kinesis Data Analytics Task data

Metric

Unit

Description

commitsFailed

Count

The total number of offset commit failures to Kafka, if offset committing and checkpointing are enabled.

commitsSucceeded

Count

The total number of successful offset commits to Kafka, if offset committing and checkpointing are enabled.

currentInputWatermark

Milliseconds

The last watermark this task has received.

currentOutputWatermark

Milliseconds

The currently output watermark.

numLateRecordsDropped

Count

The number of records this task has dropped due to arriving late.

numRecordsIn

Count

The total number of records this task has received.

numRecordsInPerSecond

CountPerSecond

The total number of records this task has received per second.

numRecordsOut

Count

The total number of records this task has emitted.

numRecordsOutPerSecond

CountPerSecond

The total number of records this task has emitted per second.

recordsLagMax

Count

The maximum lag in terms of number of records for any partition in this window.

Kinesis Data Analytics TaskOperator data

Metric

Unit

Description

bytesConsumedRate

BytesPerSecond

The average number of bytes consumed per second for a topic.

commitsFailed

Count

The total number of offset commit failures to Kafka, if offset committing and checkpointing are enabled.

commitsSucceeded

Count

The total number of successful offset commits to Kafka, if offset committing and checkpointing are enabled.

currentInputWatermark

Milliseconds

The last watermark this task has received.

currentOutputWatermark

Milliseconds

The last watermark this application/operator/task/thread has emitted.

numLateRecordsDropped

Count

The number of records this operator or task has dropped due to arriving late.

numRecordsIn

Count

The total number of records this task has received.

numRecordsInPerSecond

CountPerSecond

The total number of records this task has received per second.

numRecordsOut

Count

The total number of records this task has emitted.

numRecordsOutPerSecond

CountPerSecond

The total number of records this task has emitted per second.

recordsLagMax

Count

The maximum lag in terms of number of records for any partition in this window.

Kinesis Data Analytics TaskParallelism data

Metric

Unit

Description

currentInputWatermark

Milliseconds

The last watermark this application/operator/task/thread has received.

numLateRecordsDropped

Count

The number of records this task has dropped due to arriving late.

numRecordsIn

Count

The total number of records this task has received.

numRecordsInPerSecond

CountPerSecond

The total number of records this task has received per second.

numRecordsOut

Count

The total number of records this task has emitted.

numRecordsOutPerSecond

CountPerSecond

The total number of records this task has emitted per second.

Kinesis Data Analytics TaskOperatorParallelism data

Metric

Unit

Description

bytesConsumedRate

BytesPerSecond

The average number of bytes consumed per second for a topic.

commitsFailed

Count

The total number of offset commit failures to Kafka, if offset committing and checkpointing are enabled.

commitsSucceeded

Count

The total number of successful offset commits to Kafka, if offset committing and checkpointing are enabled.

currentInputWatermark

Milliseconds

The last watermark this application/operator/task/thread has received.

currentOutputWatermark

Milliseconds

The last watermark this application/operator/task/thread has emitted.

numRecordsIn

Count

The total number of records this operator or task has received.

numRecordsInPerSecond

CountPerSecond

The total number of records this operator or task has received per second.

numRecordsOut

Count

The total number of records this operator or task has emitted.

numRecordsOutPerSecond

CountPerSecond

The total number of records this operator or task has emitted per second.

recordsLagMax

Count

The maximum lag in terms of number of records for any partition in this window.

Copyright © 2024 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.