--- title: Amazon Managed Kafka (MSK) integration source: https://docs.newrelic.com/docs/message-queues-streaming/installation/aws-managed-kafka-msk-integration --- > #### ⚠️ IMPORTANT > > Enable the [AWS CloudWatch Metric Streams integration](https://docs.newrelic.com/docs/infrastructure/amazon-integrations/aws-integrations-list/aws-metric-stream/) to monitor all CloudWatch metrics from your AWS services, including custom namespaces. Individual integrations are no longer our recommended option. New Relic offers an integration for collecting your [Amazon Web Services Managed Streaming for Apache Kafka](https://aws.amazon.com/msk/) data. This document explains how to activate this integration and describes the data that can be reported. ## Activate integration [#activate] To enable this integration, follow standard procedures to [connect AWS services to New Relic](https://docs.newrelic.com/docs/infrastructure/infrastructure-integrations/getting-started/connect-aws-integrations-infrastructure). ## Configuration and polling [#polling] To change the polling frequency and filter data, use [configuration options](https://docs.newrelic.com/docs/integrations/new-relic-integrations/getting-started/configure-polling-frequency-data-collection-cloud-integrations). Default [polling](https://docs.newrelic.com/docs/infrastructure/amazon-integrations/aws-integrations-list/aws-polling-intervals-infrastructure-integrations) information for the Amazon Managed Kafka integration: - New Relic polling interval: 5 minutes - Amazon CloudWatch data interval: 1 minute ## View and use data [#find-data] To view your integration data, go to **[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Infrastructure > AWS** and select an integration. You can [query and explore your data](https://docs.newrelic.com/docs/using-new-relic/data/understand-data/query-new-relic-data) using the following [event type](https://docs.newrelic.com/docs/data-apis/understand-data/new-relic-data-types/#event-data): | Entity | Event Type | Provider | | ------- | --------------------- | --------------- | | Cluster | `AwsMskClusterSample` | `AwsMskCluster` | | Broker | `AwsMskBrokerSample` | `AwsMskBroker` | | Topic | `AwsMskTopicSample` | `AwsMskTopic` | For more on how to use your data, see [Understand and use integration data](https://docs.newrelic.com/docs/infrastructure/integrations/find-use-infrastructure-integration-data). ## Metric data [#metrics] This integration records Amazon Managed Kafka data for cluster, broker, and topic entities. ### Managed Kafka Cluster data | Metric | Unit | Description | | ------------------------ | ----- | ------------------------------------------------------------------- | | `activeControllerCount` | Count | Only one controller per cluster should be active at any given time. | | `globalPartitionCount` | Count | Total number of partitions across all brokers in the cluster. | | `globalTopicCount` | Count | Total number of topics across all brokers in the cluster. | | `offlinePartitionsCount` | Count | Total number of partitions that are offline in the cluster. | ### Managed Kafka Broker data | Metric | Unit | Description | | -------------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `bytesInPerSec` | BytesPerSecond | The number of bytes per second received from clients. | | `bytesOutPerSec` | BytesPerSecond | The number of bytes per second sent to clients. | | `cpuSystem` | Percent | The percentage of CPU in kernel space. | | `cpuUser` | Percent | The percentage of CPU in user space. | | `cpuIdle` | Percent | The percentage of CPU idle time. | | `fetchConsumerLocalTimeMsMean` | Milliseconds | The mean time in milliseconds that the consumer request is processed at the leader. | | `fetchConsumerRequestQueueTimeMsMean` | Milliseconds | The mean time in milliseconds that the consumer request waits in the request queue. | | `fetchConsumerResponseQueueTimeMsMean` | Milliseconds | The mean time in milliseconds that the consumer request waits in the response queue. | | `fetchConsumerResponseSendTimeMsMean` | Milliseconds | The mean time in milliseconds for the consumer to send a response. | | `fetchConsumerTotalTimeMsMean` | Milliseconds | The mean total time in milliseconds that consumers spend on fetching data from the broker. | | `fetchFollowerLocalTimeMsMean` | Milliseconds | The mean time in milliseconds that the follower request is processed at the leader. | | `fetchFollowerRequestQueueTimeMsMean` | Milliseconds | The mean time in milliseconds that the follower request waits in the request queue. | | `fetchFollowerResponseQueueTimeMsMean` | Milliseconds | The mean time in milliseconds that the follower request waits in the response queue. | | `fetchFollowerResponseSendTimeMsMean` | Milliseconds | The mean time in milliseconds for the follower to send a response. | | `fetchFollowerTotalTimeMsMean` | Milliseconds | The mean total time in milliseconds that followers spend on fetching data from the broker. | | `fetchMessageConversionsPerSec` | CountPerSecond | The number of fetch message conversions per second for the broker. | | `fetchThrottleTime` | Milliseconds | The average fetch throttle time in milliseconds. | | `fetchThrottleByteRate` | BytesPerSecond | The number of throttled bytes per second. | | `fetchThrottleQueueSize` | Count | The number of messages in the throttle queue. | | `kafkaAppLogsDiskUsed` | Percent | The percentage of disk space used for application logs. | | `kafkaDataLogsDiskUsed` | Percent | The percentage of disk space used for data logs. | | `leaderCount` | Count | The number of leader replicas. | | `memoryBuffered` | Bytes | The size in bytes of buffered memory for the broker. | | `memoryCached` | Bytes | The size in bytes of cached memory for the broker. | | `memoryFree` | Bytes | The size in bytes of memory that is free and available for the broker. | | `memoryUsed` | Bytes | The size in bytes of memory that is in use for the broker. | | `messagesInPerSec` | CountPerSecond | The number of incoming messages per second for the broker. | | `networkProcessorAvgIdlePercent` | Percent | The average percentage of the time the network processors are idle. | | `networkRxDropped` | Count | The number of dropped receive packages. | | `networkRxErrors` | Count | The number of network receive errors for the broker. | | `networkRxPackets` | Count | The number of packets received by the broker. | | `networkTxDropped` | Count | The number of dropped transmit packages. | | `networkTxErrors` | Count | The number of network transmit errors for the broker. | | `networkTxPackets` | Count | The number of packets transmitted by the broker. | | `partitionCount` | Count | The number of partitions for the broker. | | `produceLocalTimeMsMean` | Milliseconds | The mean time in milliseconds for the follower to send a response. | | `produceMessageConversionsPerSec` | CountPerSecond | The number of produce message conversions per second for the broker. | | `produceMessageConversionsTimeMsMean` | Milliseconds | The mean time in milliseconds spent on message format conversions. | | `produceRequestQueueTimeMsMean` | Milliseconds | The mean time in milliseconds that request messages spend in the queue. | | `produceResponseQueueTimeMsMean` | Milliseconds | The mean time in milliseconds that response messages spend in the queue. | | `produceResponseSendTimeMsMean` | Milliseconds | The mean time in milliseconds spent on sending response messages. | | `produceTotalTimeMsMean` | Milliseconds | The mean produce time in milliseconds. | | `produceThrottleTime` | Milliseconds | The average produce throttle time in milliseconds. | | `produceThrottleByteRate` | BytesPerSecond | The number of throttled bytes per second. | | `produceThrottleQueueSize` | Count | The number of messages in the throttle queue. | | `requestBytesMean` | Count | The mean number of request bytes for the broker. | | `requestExemptFromThrottleTime` | Milliseconds | The average time spent in broker network and I/O threads to process requests that are exempt from throttling. | | `requestHandlerAvgIdlePercent` | Percent | The average percentage of the time the request handler threads are idle. | | `requestThrottleQueueSize` | Count | The number of messages in the throttle queue. | | `requestThrottleTime` | Milliseconds | The average request throttle time in milliseconds. | | `requestTime` | Milliseconds | The average time spent in broker network and I/O threads to process requests. | | `rootDiskUsed` | Percent | The percentage of the root disk used by the broker. | | `swapUsed` | Bytes | The size in bytes of swap memory that is in use for the broker. | | `swapFree` | Bytes | The size in bytes of swap memory that is available for the broker. | | `underMinIsrPartitionCount` | Count | The number of under minIsr partitions for the broker. | | `underReplicatedPartitions` | Count | The number of under-replicated partitions for the broker. | | `zooKeeperRequestLatencyMsMean` | Milliseconds | Mean latency in milliseconds for ZooKeeper requests from broker. | | `zooKeeperSessionState` | Other | Connection status of broker's ZooKeeper session which may be one of the following: NOT_CONNECTED: '0.0', ASSOCIATING: '0.1', CONNECTING: '0.5', CONNECTEDREADONLY: '0.8', CONNECTED: '1.0', CLOSED: '5.0', AUTH_FAILED: '10.0'. | ### Managed Kafka Topic data | Metric | Unit | Description | | --------------------------------- | -------------- | ----------------------------------------------------------- | | `bytesInPerSec` | BytesPerSecond | The number of bytes received per second. | | `bytesOutPerSec` | BytesPerSecond | The number of bytes sent per second. | | `fetchMessageConversionsPerSec` | CountPerSecond | The number of fetched messages converted per second. | | `messagesInPerSec` | CountPerSecond | The number of messages received per second. | | `produceMessageConversionsPerSec` | CountPerSecond | The number of conversions per second for produced messages. | ## What's next [Data and UI](https://docs.newrelic.com/docs/message-queues-streaming/ui-data/understand-ui) Learn how to use New Relic to monitor your Kafka clusters