AWS Managed Kafka (MSK) integration

BETA

Access to this feature depends on your subscription level. Requires Infrastructure Pro.

New Relic Infrastructure's integrations include an integration for reporting your Amazon Web Services Managed Kafka data to New Relic products. This document explains how to activate this integration and describes the data that can be reported.

Activate integration

To enable this integration, follow standard procedures to connect AWS services to Infrastructure.

Configuration and polling

To change the polling frequency and filter data, use configuration options.

Default polling information for the AWS Managed Kafka integration:

  • New Relic polling interval: 5 minutes
  • Amazon CloudWatch data interval: 1 minute

View and use data

To view your integration data in Infrastructure, go to infrastructure.newrelic.com > AWS and select an integration.

In New Relic Insights, data is attached to the following event type:

Entity Event Type Provider
Cluster AwsMskClusterSample AwsMskCluster
Broker AwsMskBrokerSample AwsMskBroker
Topic AwsMskTopicSample AwsMskTopic

For more on how to use your data, see Understand and use integration data.

Metric data

This integration records AWS Managed Kafka data for cluster, broker, and topic entities.

Managed Kafka Cluster data

Metric Unit Description

ActiveControllerCount

Count Only one controller per cluster should be active at any given time.

GlobalPartitionCount

Count Total number of partitions across all brokers in the cluster.

GlobalTopicCount

Count Total number of topics across all brokers in the cluster.

OfflinePartitionsCount

Count Total number of partitions that are offline in the cluster.

Managed Kafka Broker data

Metric Unit Description

BytesInPerSec

BytesPerSecond The number of bytes per second received from clients.

BytesOutPerSec

BytesPerSecond The number of bytes per second sent to clients.

CpuSystem

Percent The percentage of CPU in kernel space.

CpuUser

Percent The percentage of CPU in user space.

CpuIdle

Percent The percentage of CPU idle time.

FetchConsumerLocalTimeMsMean

Milliseconds The mean time in milliseconds that the consumer request is processed at the leader.

FetchConsumerRequestQueueTimeMsMean

Milliseconds The mean time in milliseconds that the consumer request waits in the request queue.

FetchConsumerResponseQueueTimeMsMean

Milliseconds The mean time in milliseconds that the consumer request waits in the response queue.

FetchConsumerResponseSendTimeMsMean

Milliseconds The mean time in milliseconds for the consumer to send a response.

FetchConsumerTotalTimeMsMean

Milliseconds The mean total time in milliseconds that consumers spend on fetching data from the broker.

FetchFollowerLocalTimeMsMean

Milliseconds The mean time in milliseconds that the follower request is processed at the leader.

FetchFollowerRequestQueueTimeMsMean

Milliseconds The mean time in milliseconds that the follower request waits in the request queue.

FetchFollowerResponseQueueTimeMsMean

Milliseconds The mean time in milliseconds that the follower request waits in the response queue.

FetchFollowerResponseSendTimeMsMean

Milliseconds The mean time in milliseconds for the follower to send a response.

FetchFollowerTotalTimeMsMean

Milliseconds The mean total time in milliseconds that followers spend on fetching data from the broker.

FetchMessageConversionsPerSec

CountPerSecond The number of fetch message conversions per second for the broker.

FetchThrottleTime

Milliseconds The average fetch throttle time in milliseconds.

FetchThrottleByteRate

BytesPerSecond The number of throttled bytes per second.

FetchThrottleQueueSize

Count The number of messages in the throttle queue.

KafkaAppLogsDiskUsed

Percent The percentage of disk space used for application logs.

KafkaDataLogsDiskUsed

Percent The percentage of disk space used for data logs.

LeaderCount

Count The number of leader replicas.

MemoryBuffered

Bytes The size in bytes of buffered memory for the broker.

MemoryCached

Bytes The size in bytes of cached memory for the broker.

MemoryFree

Bytes The size in bytes of memory that is free and available for the broker.

MemoryUsed

Bytes The size in bytes of memory that is in use for the broker.

MessagesInPerSec

CountPerSecond The number of incoming messages per second for the broker.

NetworkProcessorAvgIdlePercent

Percent The average percentage of the time the network processors are idle.

NetworkRxDropped

Count The number of dropped receive packages.

NetworkRxErrors

Count The number of network receive errors for the broker.

NetworkRxPackets

Count The number of packets received by the broker.

NetworkTxDropped

Count The number of dropped transmit packages.

NetworkTxErrors

Count The number of network transmit errors for the broker.

NetworkTxPackets

Count The number of packets transmitted by the broker.

PartitionCount

Count The number of partitions for the broker.

ProduceLocalTimeMsMean

Milliseconds The mean time in milliseconds for the follower to send a response.

ProduceMessageConversionsPerSec

CountPerSecond The number of produce message conversions per second for the broker.

ProduceMessageConversionsTimeMsMean

Milliseconds The mean time in milliseconds spent on message format conversions.

ProduceRequestQueueTimeMsMean

Milliseconds The mean time in milliseconds that request messages spend in the queue.

ProduceResponseQueueTimeMsMean

Milliseconds The mean time in milliseconds that response messages spend in the queue.

ProduceResponseSendTimeMsMean

Milliseconds The mean time in milliseconds spent on sending response messages.

ProduceTotalTimeMsMean

Milliseconds The mean produce time in milliseconds.

ProduceThrottleTime

Milliseconds The average produce throttle time in milliseconds.

ProduceThrottleByteRate

BytesPerSecond The number of throttled bytes per second.

ProduceThrottleQueueSize

Count The number of messages in the throttle queue.

RequestBytesMean

Count The mean number of request bytes for the broker.

RequestExemptFromThrottleTime

Milliseconds The average time spent in broker network and I/O threads to process requests that are exempt from throttling.

RequestHandlerAvgIdlePercent

Percent The average percentage of the time the request handler threads are idle.

RequestThrottleQueueSize

Count The number of messages in the throttle queue.

RequestThrottleTime

Milliseconds The average request throttle time in milliseconds.

RequestTime

Milliseconds The average time spent in broker network and I/O threads to process requests.

RootDiskUsed

Percent The percentage of the root disk used by the broker.

SwapUsed

Bytes The size in bytes of swap memory that is in use for the broker.

SwapFree

Bytes The size in bytes of swap memory that is available for the broker.

UnderMinIsrPartitionCount

Count The number of under minIsr partitions for the broker.

UnderReplicatedPartitions

Count The number of under-replicated partitions for the broker.

ZooKeeperRequestLatencyMsMean

Milliseconds Mean latency in milliseconds for ZooKeeper requests from broker.

ZooKeeperSessionState

Other Connection status of broker's ZooKeeper session which may be one of the following: NOT_CONNECTED: '0.0', ASSOCIATING: '0.1', CONNECTING: '0.5', CONNECTEDREADONLY: '0.8', CONNECTED: '1.0', CLOSED: '5.0', AUTH_FAILED: '10.0'.

Managed Kafka Topic data

Metric Unit Description

BytesInPerSec

BytesPerSecond The number of bytes received per second.

BytesOutPerSec

BytesPerSecond The number of bytes sent per second.

FetchMessageConversionsPerSec

CountPerSecond The number of fetched messages converted per second.

MessagesInPerSec

CountPerSecond The number of messages received per second.

ProduceMessageConversionsPerSec

CountPerSecond The number of conversions per second for produced messages.

For more help

Recommendations for learning more: