Find and query Kafka data in New Relic

Once your Kafka monitoring is configured with OpenTelemetry, your metrics will appear across multiple locations in New Relic. This guide shows you where to find your data and how to query it effectively.

Navigate to your Kafka data

Your Kafka metrics appear in several places across the New Relic platform, each optimized for different use cases:

1. Entity explorer

Best for: Quick health checks and entity relationships

Go to one.newrelic.com > All capabilities > On host integrations
View Kafka broker, cluster, and topics entity types

Entity explorer showing Kafka broker, cluster, and topics entity types

2. Queues & Streams

Best for: Message queue focused monitoring

Go to one.newrelic.com > All capabilities > Queues & Streams
Filter by provider=opentelemetry and click the account row where you instrumented Kafka to view your data

Queues & Streams view showing Kafka data filtered by OpenTelemetry provider

3. Pre-built dashboards

Best for: Comprehensive monitoring and visualization

Go to one.newrelic.com > Dashboards > Recommended dashboards (View all)
Search for "OTel Kafka Dashboard"

Pre-built dashboard showing Kafka cluster health, network throughput, and broker metrics

Query Kafka data with NRQL

Use NRQL to create custom queries and dashboards for your Kafka data.

View all Kafka metrics:

FROM Metric SELECT *
WHERE instrumentation.provider='opentelemetry'
AND kafka.cluster.name IS NOT NULL
LIMIT 100

Monitor specific cluster:

FROM Metric SELECT *
WHERE kafka.cluster.name = 'production-cluster'
AND instrumentation.provider='opentelemetry'

Consumer lag by group:

FROM Metric SELECT latest(kafka.consumer_group.lag)
WHERE instrumentation.provider='opentelemetry'
FACET kafka.consumer_group.name, kafka.topic.name
SINCE 1 hour ago TIMESERIES

High consumer lag alert:

FROM Metric SELECT max(kafka.consumer_group.lag)
WHERE instrumentation.provider='opentelemetry'
AND kafka.consumer_group.name = 'your-consumer-group'

Active brokers count:

FROM Metric SELECT latest(kafka.brokers)
WHERE instrumentation.provider='opentelemetry'
AND kafka.cluster.name = 'your-cluster'
TIMESERIES

Under-replicated partitions:

FROM Metric SELECT latest(kafka.partition.under_replicated)
WHERE instrumentation.provider='opentelemetry'
FACET kafka.topic.name SINCE 1 hour ago

Message throughput by topic:

FROM Metric SELECT rate(sum(kafka.topic.messages), 1 minute) as 'Messages/min'
WHERE instrumentation.provider='opentelemetry'
FACET kafka.topic.name SINCE 1 hour ago TIMESERIES

Network I/O by broker:

FROM Metric SELECT rate(sum(kafka.network.io.bytes), 1 minute) as 'Bytes/min'
WHERE instrumentation.provider='opentelemetry'
FACET kafka.broker.id, direction SINCE 1 hour ago TIMESERIES

Advanced queries

For more comprehensive query examples organized by monitoring focus area:

Get high-level cluster statistics:

FROM Metric SELECT
  latest(kafka.brokers) AS 'Active Brokers',
  latest(kafka.cluster.topic.count) AS 'Total Topics',
  latest(kafka.cluster.partition.count) AS 'Total Partitions'
WHERE kafka.cluster.name = 'production-cluster'
  AND instrumentation.provider='opentelemetry'
TIMESERIES

Monitor individual broker health and leadership:

FROM Metric SELECT
  latest(kafka.partition.under_replicated) AS 'Under-replicated Partitions',
  latest(kafka.broker.leader.count) AS 'Leader Count',
  latest(kafka.partition.count) AS 'Total Partitions'
WHERE kafka.cluster.name = 'production-cluster'
  AND instrumentation.provider='opentelemetry'
FACET broker.id
TIMESERIES

Track overall cluster health and partition status:

FROM Metric SELECT
  latest(kafka.partition.offline) AS 'Offline Partitions',
  latest(kafka.cluster.partition.count) AS 'Total Partitions',
  latest(kafka.cluster.topic.count) AS 'Total Topics'
WHERE kafka.cluster.name = 'production-cluster'
  AND instrumentation.provider='opentelemetry'
TIMESERIES

Monitor message rates and data throughput by topic:

FROM Metric SELECT
  rate(sum(kafka.prod.msg.count), 1 minute) AS 'Messages/min',
  rate(filter(sum(kafka.topic.io), WHERE direction = 'in'), 1 minute) AS 'Bytes In/min',
  rate(filter(sum(kafka.topic.io), WHERE direction = 'out'), 1 minute) AS 'Bytes Out/min'
WHERE kafka.cluster.name = 'production-cluster'
  AND instrumentation.provider='opentelemetry'
FACET topic
TIMESERIES

Track consumer group lag and processing delays:

FROM Metric SELECT
  latest(kafka.consumer_group.lag_sum) AS 'Total Lag'
WHERE kafka.cluster.name = 'production-cluster'
  AND instrumentation.provider='opentelemetry'
FACET group
TIMESERIES

Monitor request processing times and performance:

FROM Metric SELECT
  latest(kafka.request.time.avg) AS 'Avg Latency',
  latest(kafka.request.time.99p) AS '99th Percentile'
WHERE kafka.cluster.name = 'production-cluster'
  AND instrumentation.provider='opentelemetry'
FACET type
TIMESERIES

Monitor JVM metrics for Kafka brokers:

FROM Metric SELECT
  (latest(jvm.memory.heap.used) / latest(jvm.memory.heap.max)) * 100 AS 'Heap Usage %',
  rate(sum(jvm.gc.collections.count), 1 minute) AS 'GC Count/min'
WHERE kafka.cluster.name = 'production-cluster'
  AND instrumentation.provider='opentelemetry'
FACET broker.id
TIMESERIES

Create custom dashboards

Combine multiple queries to create comprehensive Kafka monitoring dashboards:

Go to one.newrelic.com > Dashboards
Click "Create a dashboard"
Add widgets using the NRQL queries above
Organize widgets by focus area:
- Cluster health: Broker count, under-replicated partitions, controller status
- Consumer performance: Lag, throughput, group membership
- Topic metrics: Message rates, partition counts, replication status
- Resource utilization: Network I/O, JVM metrics, disk usage

Set up alerts

ヒント

New Relic provides pre-configured golden metric alerts for Kafka. When creating a new alert condition, select Guided mode → Host integrations → Kafka (brokers, cluster, or topics) to view recommended alert templates.

You can also create custom alerts using NRQL queries:

Alert when consumer lag exceeds your acceptable threshold:

FROM Metric SELECT
  latest(kafka.consumer_group.lag_sum)
WHERE kafka.cluster.name = 'production-cluster'
  AND instrumentation.provider='opentelemetry'
FACET group

Recommended threshold: Alert when lag > 10,000 messages for critical consumer groups.

Alert when partitions lose replication:

FROM Metric SELECT
  sum(kafka.partition.under_replicated)
WHERE kafka.cluster.name = 'production-cluster'
  AND instrumentation.provider='opentelemetry'

Recommended threshold: Alert when any partitions become under-replicated (> 0).

Alert immediately when partitions go offline:

FROM Metric SELECT
  latest(kafka.partition.offline)
WHERE kafka.cluster.name = 'production-cluster'
  AND instrumentation.provider='opentelemetry'

Recommended threshold: Alert immediately when any partitions go offline (> 0).

Alert when partitions fall below minimum in-sync replicas:

FROM Metric SELECT
  latest(kafka.partition.under_min_isr)
WHERE kafka.cluster.name = 'production-cluster'
  AND instrumentation.provider='opentelemetry'
FACET broker.id

Recommended threshold: Alert when any partitions fall below minimum ISR (> 0).

Alert when heap usage is critical:

FROM Metric SELECT
  (latest(jvm.memory.heap.used) / latest(jvm.memory.heap.max)) * 100
WHERE kafka.cluster.name = 'production-cluster'
  AND instrumentation.provider='opentelemetry'
FACET broker.id

Recommended threshold: Alert when heap usage > 85% for sustained periods.

Alert on high request failure rates:

FROM Metric SELECT
  rate(sum(kafka.request.failed), 1 minute)
WHERE kafka.cluster.name = 'production-cluster'
  AND instrumentation.provider='opentelemetry'
FACET type

Recommended threshold: Alert when failure rate > 1% of total requests.

Alert on unclean leader elections (indicates broker failures):

FROM Metric SELECT
  rate(sum(kafka.unclean.election.rate), 1 minute)
WHERE kafka.cluster.name = 'production-cluster'
  AND instrumentation.provider='opentelemetry'

Recommended threshold: Alert on any unclean leader elections (> 0).

Next steps

Kafka metrics reference - Complete list of available metrics with descriptions
Create NRQL alert conditions - Set up proactive monitoring
NRQL tutorial - Learn advanced querying techniques

Navigate to your Kafka data

1. Entity explorer

2. Queues & Streams

3. Pre-built dashboards

Query Kafka data with NRQL

Basic queries

Consumer lag monitoring

Broker health queries

Performance monitoring

Advanced queries

Cluster overview

Broker health

Cluster health

Topic throughput and I/O

Consumer lag

Request latency

JVM health

Create custom dashboards

Set up alerts

ヒント

High consumer lag

Under-replicated partitions

Offline partitions

Partitions under minimum ISR

High JVM memory usage

Request failures

Unclean leader elections

Next steps

Find and query Kafka data in New Relic

Navigate to your Kafka data .css-21sua1{background:none;border:none;width:0;padding:0;}

1. Entity explorer

2. Queues & Streams

3. Pre-built dashboards

Query Kafka data with NRQL

Consumer lag monitoring

Broker health queries

Performance monitoring

Advanced queries

Cluster overview

Broker health

Cluster health

Topic throughput and I/O

Consumer lag

Request latency

JVM health

Create custom dashboards

Set up alerts

ヒント

High consumer lag

Under-replicated partitions

Offline partitions

Partitions under minimum ISR

High JVM memory usage

Request failures

Unclean leader elections

Next steps

Navigate to your Kafka data