• /
  • Log in
  • Free account

Java agent: Instrument Kafka message queues

The New Relic Java agent automatically collects data from Kafka's Java clients library. Because Kafka is a high-performance messaging system that generates a lot of data, you can customize the agent for your app's specific throughput and use cases.

This document explains how to collect and view three types of Kafka data:

Kafka instrumentation is available in Java agent versions 4.12.0 or higher. For supported Kafka client versions, see Java compatibility and requirements.

Tip

For our Kafka integration, see Kafka monitoring integration.

View Kafka metrics

After installation, the agent automatically reports rich Kafka metrics with information about messaging rates, latency, lag, and more. The Java agent collects all Kafka consumer and producer metrics (but not connect or stream metrics).

To view these metrics, create a custom dashboard:

  1. Go to the New Relic metric explorer.

  2. Use the metric explorer to locate your metrics. You can find Kafka metrics in this metric folder:

    MessageBroker/Kafka/Internal/KafkaMetricName

    For example, the request-rate metric is located at:

    MessageBroker/Kafka/Internal/consumer-metrics/request-rate

    Tip

    For a full list of Kafka consumer and producer metrics, see the Kafka documentation.

  3. Add the metrics you want to monitor to a dashboard by clicking Add to dashboard.

Enable Kafka event collection

You can configure the agent to collect event data instead of metric timeslice data (for the difference between metric timeslice and event data, see data collection). This allows you to use NRQL to filter and facet the default Kafka metrics. When enabled, the agent collects one Kafka event every 30 seconds. This event contains all of the the data from Kafka consumer and produce metrics captured since the previous event.

Important

The agent records up to 2000 events per harvest cycle, though you can change this value with max_samples_stored. Kafka event data is included in this pool. If you use the recordCustomEvent() API call to send custom events to New Relic and you send more than 2000 events, the agent will discard some Kafka or custom events.

To enable Kafka event collection:

  1. Add the kafka.metrics.as_events.enabled element to your newrelic.yml config file:

    kafka.metrics.as_events.enabled: true
  2. Restart your JVM.

  3. Use the event explorer to view your Kafka events, located in the KafkaMetrics event type. Or, use NRQL to query your events directly. For example:

    SELECT average('producer-metrics.record-send-rate') from KafkaMetrics SINCE 30 minutes ago timeseries

Enable Kafka distributed traces

The Java agent can also collect distributed traces from Kafka clients. Enabling traces doesn't affect the agent's normal operations: it will still report metric or event data from Kafka.

Impacts and requirements to consider before enabling:

  • The instrumentation adds a 150 to 200 byte payload to message headers. If your Kafka messages are very small, traces can add significant processing and storage overhead. This additional payload size could cause Kafka to drop messages if they exceed your Kafka messaging size limit. For this reason, we recommend testing out Kafka distributed traces in a dev environment before enabling them in production.
  • Distributed tracing is only available for Kafka client versions 0.11.0.0 or higher.
  • If you have not enabled distributed tracing for your app before, read the Transition guide before enabling.
  • To propagate W3C trace context via Kafka message headers, see the distributed tracing API usage guide for details on APIs that were released in Java agent 6.4.0. Note that adding additional headers to Kafka messages will further increase the payload size. To see these APIs in action, see Using Java agent trace APIs with Kafka.

The complete process of enabling this is below but at a high level it involves these basic steps: 1) enable tracing via the agent config, and 2) calling the Java agent API to instrument transactions on both the producer and consumer side.

To collect distributed traces from Kafka:

For more help

If you need more help, check out these support and learning resources:

Create issueEdit page
Copyright © 2021 New Relic Inc.