Google Cloud Managed Kafka monitoring integration

New Relic integrations with the Google Cloud Platform (GCP) include one that reports Google Cloud Managed Service for Apache Kafka data to New Relic. This document explains how to activate the GCP Managed Kafka integration and describes the data it reports.

Features

Managed Service for Apache Kafka is Google Cloud's fully managed, Apache Kafka–compatible streaming platform, including Kafka Connect. New Relic Managed Kafka integration collects throughput, lag, retention, and connector-health metrics across clusters, topics, partitions, consumer groups, and Kafka Connect clusters and connectors.

Activate integration

To enable the integration, follow standard procedures to connect your GCP service to New Relic:

Connect with Workload Identity Federation

Polling frequency

New Relic integrations query your GCP services according to a polling interval that varies by integration. The polling frequency for Google Cloud Managed Kafka is 5 minutes. The resolution is 1 data point every minute.

중요

Managed Service for Apache Kafka supports 1-minute polling intervals for WIF integrations in limited preview. To request access, contact New Relic support.

Workload Identity Federation

Find and use data

After you enable the integration, your Managed Kafka resources appear as entities in the New Relic entity explorer. To see dashboards and manage services, go to one.newrelic.com > All capabilities > Infrastructure > GCP.

All Managed Kafka metrics available in GCP Cloud Monitoring are collected as dimensional metrics in the Metric event type. Additional metrics beyond this table are collected automatically. See Google's Managed Kafka metrics documentation for the complete list.

Entities

Entity	Entity type	Resource type
Cluster	`KAFKACLUSTER`	`managedkafka.googleapis.com/Cluster`
Topic	`KAFKATOPIC`	`managedkafka.googleapis.com/Topic`
Topic Partition	`GCPMANAGEDKAFKATOPICPARTITION`	`managedkafka.googleapis.com/TopicPartition`
Consumer Group	`GCPMANAGEDKAFKATOPICPARTITIONCONSUMERGROUP`	`managedkafka.googleapis.com/TopicPartitionConsumerGroup`
Connect Cluster	`GCPMANAGEDKAFKACONNECTCLUSTER`	`managedkafka.googleapis.com/ConnectCluster`
Connector	`GCPMANAGEDKAFKACONNECTOR`	`managedkafka.googleapis.com/Connector`

Metric data

Key metrics — Cluster

Metric name	Unit	Description
`gcp.managedkafka.cluster.cpu.utilization`	Percent	CPU utilization across broker nodes in the Managed Kafka cluster.
`gcp.managedkafka.cluster.memory.utilization`	Percent	Memory utilization across broker nodes in the Managed Kafka cluster.
`gcp.managedkafka.cluster.storage.used_bytes`	Bytes	Storage in use by the Managed Kafka cluster.
`gcp.managedkafka.cluster.broker_count`	Count	Number of brokers currently running in the Managed Kafka cluster.
`gcp.managedkafka.cluster.topic_count`	Count	Number of topics on the Managed Kafka cluster.

Key metrics — Topic

Metric name	Unit	Description
`gcp.managedkafka.topic.byte_in_count`	Bytes	Bytes produced to the Managed Kafka topic.
`gcp.managedkafka.topic.byte_out_count`	Bytes	Bytes consumed from the Managed Kafka topic.
`gcp.managedkafka.topic.message_in_count`	Count	Messages produced to the Managed Kafka topic.
`gcp.managedkafka.topic.partition_count`	Count	Number of partitions in the Managed Kafka topic.
`gcp.managedkafka.topic.byte_size`	Bytes	Total on-disk size of the Managed Kafka topic across partitions.

Key metrics — Topic Partition

Metric name	Unit	Description
`gcp.managedkafka.byte_size`	Bytes	On-disk size of the topic partition.
`gcp.managedkafka.last_offset`	Count	Most recent offset produced to the topic partition.
`gcp.managedkafka.first_offset`	Count	Earliest available offset in the topic partition.
`gcp.managedkafka.log_end_offset`	Count	Log-end offset for the topic partition.
`gcp.managedkafka.consumer_lag`	Count	Number of messages the partition is ahead of its slowest consumer.

Key metrics — Consumer Group

Metric name	Unit	Description
`gcp.managedkafka.offset_lag`	Count	Number of messages the consumer group is behind the latest offset, faceted by partition.
`gcp.managedkafka.committed_offset`	Count	Last offset committed by the consumer group.
`gcp.managedkafka.consumer_group.member_count`	Count	Number of active members in the consumer group.
`gcp.managedkafka.consumer_group.rebalance_count`	Count	Number of consumer-group rebalances observed.

Key metrics — Connect Cluster

Metric name	Unit	Description
`gcp.managedkafka.cpu.worker_core_usage_time`	Seconds	CPU-seconds consumed by workers in the Kafka Connect cluster.
`gcp.managedkafka.memory.worker_usage`	Bytes	Memory currently in use by workers in the Kafka Connect cluster.
`gcp.managedkafka.connector_request_rate`	Count	Number of connector API requests processed by the Kafka Connect cluster.
`gcp.managedkafka.connect_cluster.worker_count`	Count	Number of workers currently running in the Kafka Connect cluster.
`gcp.managedkafka.connect_cluster.request_latencies`	Milliseconds	Latency distribution for connector API requests on the Kafka Connect cluster.

Key metrics — Connector

Metric name	Unit	Description
`gcp.managedkafka.connector_tasks`	Count	Number of tasks running for the connector, faceted by task state.
`gcp.managedkafka.connector_total_error_record_count`	Count	Number of records that produced errors on the connector.
`gcp.managedkafka.connector_sink_active_records`	Count	Active records buffered on sink connectors.
`gcp.managedkafka.connector_source_records_written_count`	Count	Number of records written by source connectors to Kafka.
`gcp.managedkafka.connector_sink_records_read_count`	Count	Number of records read by sink connectors from Kafka.

For the complete list of Managed Kafka metrics, see Google's Managed Kafka metrics documentation.