• /
  • Log in

Amazon Managed Kafka (MSK) integration

New Relic offers an integration for collecting your Amazon Web Services Managed Streaming for Apache Kafka data. This document explains how to activate this integration and describes the data that can be reported.

Activate integration

To enable this integration, follow standard procedures to connect AWS services to New Relic.

Configuration and polling

To change the polling frequency and filter data, use configuration options.

Default polling information for the Amazon Managed Kafka integration:

  • New Relic polling interval: 5 minutes
  • Amazon CloudWatch data interval: 1 minute

View and use data

To view your integration data, go to one.newrelic.com > Infrastructure > AWS and select an integration.

You can query and explore your data using the following event type:

Entity

Event Type

Provider

Cluster

AwsMskClusterSample

AwsMskCluster

Broker

AwsMskBrokerSample

AwsMskBroker

Topic

AwsMskTopicSample

AwsMskTopic

For more on how to use your data, see Understand and use integration data.

Metric data

This integration records Amazon Managed Kafka data for cluster, broker, and topic entities.

Managed Kafka Cluster data

Metric

Unit

Description

activeControllerCount

Count

Only one controller per cluster should be active at any given time.

globalPartitionCount

Count

Total number of partitions across all brokers in the cluster.

globalTopicCount

Count

Total number of topics across all brokers in the cluster.

offlinePartitionsCount

Count

Total number of partitions that are offline in the cluster.

Managed Kafka Broker data

Metric

Unit

Description

bytesInPerSec

BytesPerSecond

The number of bytes per second received from clients.

bytesOutPerSec

BytesPerSecond

The number of bytes per second sent to clients.

cpuSystem

Percent

The percentage of CPU in kernel space.

cpuUser

Percent

The percentage of CPU in user space.

cpuIdle

Percent

The percentage of CPU idle time.

fetchConsumerLocalTimeMsMean

Milliseconds

The mean time in milliseconds that the consumer request is processed at the leader.

fetchConsumerRequestQueueTimeMsMean

Milliseconds

The mean time in milliseconds that the consumer request waits in the request queue.

fetchConsumerResponseQueueTimeMsMean

Milliseconds

The mean time in milliseconds that the consumer request waits in the response queue.

fetchConsumerResponseSendTimeMsMean

Milliseconds

The mean time in milliseconds for the consumer to send a response.

fetchConsumerTotalTimeMsMean

Milliseconds

The mean total time in milliseconds that consumers spend on fetching data from the broker.

fetchFollowerLocalTimeMsMean

Milliseconds

The mean time in milliseconds that the follower request is processed at the leader.

fetchFollowerRequestQueueTimeMsMean

Milliseconds

The mean time in milliseconds that the follower request waits in the request queue.

fetchFollowerResponseQueueTimeMsMean

Milliseconds

The mean time in milliseconds that the follower request waits in the response queue.

fetchFollowerResponseSendTimeMsMean

Milliseconds

The mean time in milliseconds for the follower to send a response.

fetchFollowerTotalTimeMsMean

Milliseconds

The mean total time in milliseconds that followers spend on fetching data from the broker.

fetchMessageConversionsPerSec

CountPerSecond

The number of fetch message conversions per second for the broker.

fetchThrottleTime

Milliseconds

The average fetch throttle time in milliseconds.

fetchThrottleByteRate

BytesPerSecond

The number of throttled bytes per second.

fetchThrottleQueueSize

Count

The number of messages in the throttle queue.

kafkaAppLogsDiskUsed

Percent

The percentage of disk space used for application logs.

kafkaDataLogsDiskUsed

Percent

The percentage of disk space used for data logs.

leaderCount

Count

The number of leader replicas.

memoryBuffered

Bytes

The size in bytes of buffered memory for the broker.

memoryCached

Bytes

The size in bytes of cached memory for the broker.

memoryFree

Bytes

The size in bytes of memory that is free and available for the broker.

memoryUsed

Bytes

The size in bytes of memory that is in use for the broker.

messagesInPerSec

CountPerSecond

The number of incoming messages per second for the broker.

networkProcessorAvgIdlePercent

Percent

The average percentage of the time the network processors are idle.

networkRxDropped

Count

The number of dropped receive packages.

networkRxErrors

Count

The number of network receive errors for the broker.

networkRxPackets

Count

The number of packets received by the broker.

networkTxDropped

Count

The number of dropped transmit packages.

networkTxErrors

Count

The number of network transmit errors for the broker.

networkTxPackets

Count

The number of packets transmitted by the broker.

partitionCount

Count

The number of partitions for the broker.

produceLocalTimeMsMean

Milliseconds

The mean time in milliseconds for the follower to send a response.

produceMessageConversionsPerSec

CountPerSecond

The number of produce message conversions per second for the broker.

produceMessageConversionsTimeMsMean

Milliseconds

The mean time in milliseconds spent on message format conversions.

produceRequestQueueTimeMsMean

Milliseconds

The mean time in milliseconds that request messages spend in the queue.

produceResponseQueueTimeMsMean

Milliseconds

The mean time in milliseconds that response messages spend in the queue.

produceResponseSendTimeMsMean

Milliseconds

The mean time in milliseconds spent on sending response messages.

produceTotalTimeMsMean

Milliseconds

The mean produce time in milliseconds.

produceThrottleTime

Milliseconds

The average produce throttle time in milliseconds.

produceThrottleByteRate

BytesPerSecond

The number of throttled bytes per second.

produceThrottleQueueSize

Count

The number of messages in the throttle queue.

requestBytesMean

Count

The mean number of request bytes for the broker.

requestExemptFromThrottleTime

Milliseconds

The average time spent in broker network and I/O threads to process requests that are exempt from throttling.

requestHandlerAvgIdlePercent

Percent

The average percentage of the time the request handler threads are idle.

requestThrottleQueueSize

Count

The number of messages in the throttle queue.

requestThrottleTime

Milliseconds

The average request throttle time in milliseconds.

requestTime

Milliseconds

The average time spent in broker network and I/O threads to process requests.

rootDiskUsed

Percent

The percentage of the root disk used by the broker.

swapUsed

Bytes

The size in bytes of swap memory that is in use for the broker.

swapFree

Bytes

The size in bytes of swap memory that is available for the broker.

underMinIsrPartitionCount

Count

The number of under minIsr partitions for the broker.

underReplicatedPartitions

Count

The number of under-replicated partitions for the broker.

zooKeeperRequestLatencyMsMean

Milliseconds

Mean latency in milliseconds for ZooKeeper requests from broker.

zooKeeperSessionState

Other

Connection status of broker's ZooKeeper session which may be one of the following: NOT_CONNECTED: '0.0', ASSOCIATING: '0.1', CONNECTING: '0.5', CONNECTEDREADONLY: '0.8', CONNECTED: '1.0', CLOSED: '5.0', AUTH_FAILED: '10.0'.

Managed Kafka Topic data

Metric

Unit

Description

bytesInPerSec

BytesPerSecond

The number of bytes received per second.

bytesOutPerSec

BytesPerSecond

The number of bytes sent per second.

fetchMessageConversionsPerSec

CountPerSecond

The number of fetched messages converted per second.

messagesInPerSec

CountPerSecond

The number of messages received per second.

produceMessageConversionsPerSec

CountPerSecond

The number of conversions per second for produced messages.

For more help

If you need more help, check out these support and learning resources:

Create issueEdit page
Copyright © 2021 New Relic Inc.