The New Relic Java agent automatically collects data from Kafka's Java clients library. Because Kafka is a high-performance messaging system that generates a lot of data, you can customize the agent for your app's specific throughput and use cases.
This document explains how to collect and view three types of Kafka data:
Kafka instrumentation is available in Java agent versions 4.12.0 or higher. For supported Kafka client versions, see Java compatibility and requirements.
For our Kafka integration, see Kafka monitoring integration.
After installation, the agent automatically reports rich Kafka metrics with information about messaging rates, latency, lag, and more. The Java agent collects all Kafka consumer and producer metrics (but not connect or stream metrics).
To view these metrics, create a custom dashboard:
Go to the New Relic metric explorer.
Use the metric explorer to locate your metrics. You can find Kafka metrics in this metric folder:
For example, the
request-ratemetric is located at:MessageBroker/Kafka/Internal/consumer-metrics/request-rate
For a full list of Kafka consumer and producer metrics, see the Kafka documentation.
Add the metrics you want to monitor to a dashboard by clicking Add to dashboard.
You can configure the agent to collect event data instead of metric timeslice data (for the difference between metric timeslice and event data, see data collection). This allows you to use NRQL to filter and facet the default Kafka metrics. When enabled, the agent collects one Kafka event every 30 seconds. This event contains all of the the data from Kafka consumer and produce metrics captured since the previous event.
The agent records up to 2000 events per harvest cycle, though you can change this value with
max_samples_stored. Kafka event data is included in this pool. If you use the
recordCustomEvent() API call to send custom events to New Relic and you send more than 2000 events, the agent will discard some Kafka or custom events.
To enable Kafka event collection:
kafka.metrics.as_events.enabledelement to your
newrelic.ymlconfig file:kafka.metrics.as_events.enabled: true
Restart your JVM.
- SELECT average('producer-metrics.record-send-rate') from KafkaMetrics SINCE 30 minutes ago timeseries
The Java agent can also collect distributed traces from Kafka clients. Enabling traces doesn't affect the agent's normal operations: it will still report metric or event data from Kafka.
Impacts and requirements to consider before enabling:
- The instrumentation adds a 150 to 200 byte payload to message headers. If your Kafka messages are very small, traces can add significant processing and storage overhead. This additional payload size could cause Kafka to drop messages if they exceed your Kafka messaging size limit. For this reason, we recommend testing out Kafka distributed traces in a dev environment before enabling them in production.
- Distributed tracing is only available for Kafka client versions 0.11.0.0 or higher.
- If you have not enabled distributed tracing for your app before, read the Transition guide before enabling.
- To propagate W3C trace context via Kafka message headers, see the distributed tracing API usage guide for details on APIs that were released in Java agent 6.4.0. Note that adding additional headers to Kafka messages will further increase the payload size. To see these APIs in action, see Using Java agent trace APIs with Kafka.
The complete process of enabling this is below but at a high level it involves these basic steps: 1) enable tracing via the agent config, and 2) calling the Java agent API to instrument transactions on both the producer and consumer side.
To collect distributed traces from Kafka: