• English日本語한국어
  • Log inStart now

Collector for Confluent Cloud & Kafka monitoring

You can collect metrics about your Confluent Cloud-managed Kafka deployment with the OpenTelemetry Collector. The collector is a component of OpenTelemetry that collects, processes, and exports telemetry data to New Relic (or any observability backend).

Complete the steps below to collect Kafka metrics from Confluent using an OpenTelemetry collector running in docker.

Step 1 of 5

Make sure you're set up

Before you start, you need to have the for the account you want to report data to. You should also verify that:

Step 2 of 5

Download or clone the example repos

Download New Relic's OpenTelemetry Examples repoas this setup uses its example collector configuration. Once installed, open the Confluent Cloud example directory. For more information, you can check the README there as well.

Step 3 of 5

Add the authentication files

This example setup uses TLS to authenticate the request to Confluent Cloud. There are multiple methods to authenticate, so you should follow your company best practices and authentication methods.

  • TLS/SSL requires you to create keys and certificates, create your own Certificate Authority (CA), and sign the certificate.

  • Doing this should leave you with three files which need to be added to this directory.

  • Those files are referenced in this example as the follwing files: key.pem, cert.pem, ca.pem.

    Important

    For more information about TLS authentication with Confluent Cloud, check the documentation on authenticating with TLS as well as the security tutorial.

    For dev/test Confluent environments, you can simplify this by using plain text authentication.

Step 4 of 5

Set environment variables and run the collector

Export the following variables or add them in a .env file, then run the docker compose up command.

bash
$
# Open the confluent cloud example directory
$
cd newrelic-opentelemetry-examples/other-examples/collector/confluentcloud
$
$
# Set environment variables.
$
export NEW_RELIC_API_KEY=<YOUR_API_KEY>
$
export NEW_RELIC_OTLP_ENDPOINT=https://otlp.nr-data.net
$
export CLUSTER_ID=<YOUR_CLUSTER_ID>
$
export CLUSTER_API_KEY=<YOUR_CLUSTER_API_KEY>
$
export CLUSTER_API_SECRET=<YOUR_CLUSTER_API_SECRET>
$
export CLUSTER_BOOTSTRAP_SERVER=<YOUR_CLUSTER_BOOTSTRAP_SERVER>
$
$
# run the collector in docker
$
docker compose up

Local Variable information

Variable

Description

Docs

NEW_RELIC_API_KEY

New Relic Ingest API Key

API Key docs

NEW_RELIC_OTLP_ENDPOINT

New Relic OTLP endpoint is https://otlp.nr-data.net:4318

OTLP endpoint config docs

CLUSTER_ID

ID of cluster from Confluent Cloud

Available in your Confluent cluster settings

CONFLUENT_API_ID

Cloud API key

Cloud API key docs

CONFLUENT_API_SECRET

Cloud API secret

Cloud API key docs

CLUSTER_BOOTSTRAP_SERVER

Bootstrap Server for cluster

Available in your cluster settings

Step 5 of 5

View your data in New Relic

You can view your Confluent Cloud data in a few different ways.

  • Navigate to the New Relic marketplace and search for Confluent. The available dashboards can be installed right onto your account!
  • Navigate to the metrics explorer and filter for confluent_kafka. This data can be added to any custom alert or dashboard.

Confluent Cloud metrics

This integration covers all the Exportable metrics within the Confluent Cloud Metrics API. We have a partial list of the Exportable metrics below:

Name

Description

confluent_kafka_server_received_bytes

The delta count of bytes of the customer's data received from the network. Each sample is the number of bytes received since the previous data sample. The count is sampled every 60 seconds.

confluent_kafka_server_sent_bytes

The delta count of bytes of the customer's data sent over the network. Each sample is the number of bytes sent since the previous data point. The count is sampled every 60 seconds.

confluent_kafka_server_received_records

The delta count of records received. Each sample is the number of records received since the previous data sample. The count is sampled every 60 seconds.

confluent_kafka_server_sent_records

The delta count of records sent. Each sample is the number of records sent since the previous data point. The count is sampled every 60 seconds.

confluent_kafka_server_retained_bytes

The current count of bytes retained by the cluster. The count is sampled every 60 seconds.

confluent_kafka_server_active_connection_count

The count of active authenticated connections.

confluent_kafka_server_request_count

The delta count of requests received over the network. Each sample is the number of requests received since the previous data point. The count sampled every 60 seconds.

confluent_kafka_server_partition_count

The number of partitions

confluent_kafka_server_successful_authentication_count

The delta count of successful authentications. Each sample is the number of successful authentications since the previous data point. The count sampled every 60 seconds.

confluent_kafka_server_consumer_lag_offsets

The lag between a group member's committed offset and the partition's high watermark.

Copyright © 2023 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.