New Relic Queues and Streams offers a comprehensive solution for monitoring your Kafka clusters, providing detailed insights into potential issues such as failures, bottlenecks, and throttling.
This document helps you to:
- Find your Kafka telemetry data in the New Relic UI.
- Understand the key metrics and data available in the New Relic UI.
- Learn how to use the New Relic UI to monitor and observe the performance of your Kafka message queues and streaming data.
Accessing queues and streams data in the New Relic UI
To view your cluster data in the New Relic UI, Go to one.newrelic.com > All capabilities > Queues & Streams.
To view the Kafka cluster data, in the Queues & Streams summary page, under Name column, select the Kafka cluster provider. You can also use the search bar or filter to search for a specific Kafka clusters based on Provider, Account name, or their Account ID.
Key metrics and data in the New Relic UI
With New Relic Queues and Streams for Kafka, you get:
Instant real-time Kafka insights: Scrape key client Kafka metrics effortlessly with the Java agent to quickly gain real time insights by connecting the dots across your hosts and service through an unified UI. Spot outliers and troubleshoot faster: Instantly spot anomalies in key metrics across all Kafka client components - producers and consumers - to ensure smooth data flow Pinpoint bottlenecks: Using latency metrics, throttling rates, queue depth, and retry counts identify bottlenecks that impact data flow and increase latency Uncover hidden broker issues: Analyze client-side metrics like producer/consumer throttle time and retry rate growth to detect potential broker-side issues and resource limitations Boost data flow with client-side optimization: Identify uncompressed data streams through metrics such as average producer compression ratio and implement efficient compression strategies to shrink data size and improve throughput
Summary
Summary section provides an overview of the Kafka cluster, including the number of brokers, topics, and partitions.
Kafka Navigator provides a pictorial representation of the healthy and unhealthy Kafka clusters, brokers, or topics. The green honeycomb tile represents the healthy, while the yellow honeycomb represents the unhealthy status. Hover over the honeycomb tile to view the details of the cluster, broker, or topic. You can also click the honeycomb tile to view additional details of the cluster, broker, or topic.
Incoming throughput by clusters (bytes/sec) provides the time series data of incoming throughput of the Kafka cluster in bytes per second.
By Cluster bar chart visualisations provides count of topics being written to or read from all brokers in the kafka cluster, along with the number of partitions and brokers in these cluster.
Topics
The topics section provides a detailed view of the top 20 topic entities in the Kafka cluster, including incoming and outgoing throughput, and message rate. You can click on the topic entity to view additional details of the topic, such as Bytes in, Bytes out, Messages in, fetch Messages etc.