Kafka's integration configuration settings

This integration is open source software. That means you can browse its source code and send improvements or create your own fork and build it.

Labels and custom attributes

Environment variables can be used to control config settings, such as your , and are then passed through to the infrastructure agent. For instructions on how to use this feature, see Configure the infrastructure agent.

You can further decorate your metrics using labels. Labels allow you to add key/value pairs attributes to your metrics which you can then use to query, filter or group your metrics on.
Our default sample config file includes examples of labels but, as they are not mandatory, you can remove, modify or add new ones of your choice.

labels:
    env: production
    role: kafka

For more about the general structure of on-host integration configuration, see the configuration.

Inventory data

The Kafka integration captures the non-default broker and topic configuration parameters, and collects the topic partition schemes as reported by ZooKeeper. The data is available on the Inventory UI page under the config/kafka source.

Configure KafkaBrokerSample and KafkaTopicSample collection

The Kafka integration collects both Metrics and Inventory information. Check the Applies To column below to see the settings available to each collection:

Setting	Description	Default	Applies To
CLUSTER_NAME	User-defined name to uniquely identify the cluster being monitored. Required.	N/A	M/I
KAFKA_VERSION	The version of the Kafka broker you're connecting to, used for setting optimum API versions. It must match -or be lower than- the version from the broker. Versions older than 1.0.0 may be missing some features. Note that if the broker binary name is `kafka_2.12-2.7.0`, the Kafka API version to be used is 2.7.0 and the preceding 2.12 is the Scala language version.	1.0.0	M/I
AUTODISCOVER_STRATEGY	the method of discovering brokers. Options are `zookeeper` or `bootstrap`.	zookeeper	M/I
METRICS	Set to `true` to enable Metrics only collection.	false
INVENTORY	Set to `true` to enable Inventory only collection.	false

Zookeeper autodiscovery arguments

These are only relevant when the autodiscover_strategy option is set to zookeeper.

Setting	Description	Default	Applies To
ZOOKEEPER_HOSTS	The list of Apache ZooKeeper hosts (in JSON format) that need to be connected. If `CONSUMER_OFFSET` is set to `false` `KafkaBrokerSamples` and `KafkaTopicSamples` will be collected.	[]	M/I
ZOOKEEPER_AUTH_SCHEME	The ZooKeeper authentication scheme that is used to connect. Currently, the only supported value is `digest`. If omitted, no authentication is used.	N/A	M/I
ZOOKEEPER_AUTH_SECRET	The ZooKeeper authentication secret that is used to connect. Should be of the form `username:password`. Only required if `zookeeper_auth_scheme` is specified.	N/A	M/I
ZOOKEEPER_PATH	The Zookeeper node under which the Kafka configuration resides. Defaults to `/`.	N/A	M/I
PREFERRED_LISTENER	Use a specific listener to connect to a broker. If unset, the first listener that passes a successful test connection is used. Supported values are `PLAINTEXT`, `SASL_PLAINTEXT`, `SSL`, and `SASL_SSL`. *Note the `SASL_` protocols only support Kerberos (GSSAPI) authentication.**	N/A	M/I

Bootstrap broker discovery arguments

These are only relevant when the autodiscover_strategy option is set tobootstrap

Setting	Description	Default	Applies To
BOOTSTRAP_BROKER_HOST	The host for the bootstrap broker. If `CONSUMER_OFFSET` is set to `false` `KafkaBrokerSamples` and `KafkaTopicSamples` will be collected.	N/A	M/I
BOOTSTRAP_BROKER_KAFKA_PORT	The Kafka port for the bootstrap broker.	N/A	M/I
BOOTSTRAP_BROKER_KAFKA_PROTOCOL	The protocol to use to connect to the bootstrap broker. Supported values are `PLAINTEXT`, `SASL_PLAINTEXT`, `SSL`, and `SASL_SSL`. *Note the `SASL_` protocols only support Kerberos (GSSAPI) authentication.**	PLAINTEXT	M/I
BOOTSTRAP_BROKER_JMX_PORT	The JMX port to use for collection on each broker in the cluster. Note that all discovered brokers should have JMX active on this port	N/A	M/I
BOOTSTRAP_BROKER_JMX_USER	The JMX user to use for collection on each broker in the cluster.	N/A	M/I
BOOTSTRAP_BROKER_JMX_PASSWORD	The JMX password to use for collection on each broker in the cluster.	N/A	M/I

JMX options

These options apply to all JMX connections on the instance.

Setting	Description	Default	Applies To
KEY_STORE	The filepath of the keystore containing the JMX client's SSL certificate.	N/A	M/I
KEY_STORE_PASSWORD	The password for the JMX SSL key store.	N/A	M/I
TRUST_STORE	The filepath of the trust keystore containing the JMX server's SSL certificate.	N/A	M/I
TRUST_STORE_PASSWORD	The password for the JMX trust store.	N/A	M/I
DEFAULT_JMX_USER	The default user that is connecting to the JMX host to collect metrics. If the username field is omitted for a JMX host, this value will be used.	admin	M/I
DEFAULT_JMX_PASSWORD	The default password to connect to the JMX host. If the password field is omitted for a JMX host, this value will be used.	admin	M/I
TIMEOUT	The timeout for individual JMX queries in milliseconds.	10000	M/I

Broker TLS connection options

You need these options if the broker protocol is SSL or SASL_SSL.

Setting	Description	Default	Applies To
TLS_CA_FILE	The certificate authority file for SSL and SASL_SSL listeners, in PEM format.	N/A	M/I
TLS_CERT_FILE	The client certificate file for SSL and SASL_SSL listeners, in PEM format.	N/A	M/I
TLS_KEY_FILE	The client key file for SSL and SASL_SSL listeners, in PEM format.	N/A	M/I
TLS_INSECURE_SKIP_VERIFY	Skip verifying the server's certificate chain and host name.	false	M/I

Broker SASL and Kerberos connection options

You need these options if the broker protocol is SASL_PLAINTEXT or SASL_SSL.

Setting	Description	Default	Applies To
SASL_MECHANISM	The type of SASL authentication to use. Supported options are `SCRAM-SHA-512`, `SCRAM-SHA-256`, `PLAIN`, and `GSSAPI`.	N/A	M/I
SASL_USERNAME	SASL username required with the PLAIN and SCRAM mechanisms.	N/A	M/I
SASL_PASSWORD	SASL password required with the PLAIN and SCRAM mechanisms.	N/A	M/I
SASL_GSSAPI_REALM	Kerberos realm required with the GSSAPI mechanism.	N/A	M/I
SASL_GSSAPI_SERVICE_NAME	Kerberos service name required with the GSSAPI mechanism.	N/A	M/I
SASL_GSSAPI_USERNAME	Kerberos username required with the GSSAPI mechanism.	N/A	M/I
SASL_GSSAPI_KEY_TAB_PATH	Kerberos key tab path required with the GSSAPI mechanism.	N/A	M/I
SASL_GSSAPI_KERBEROS_CONFIG_PATH	Kerberos config path required with the GSSAPI mechanism.	/etc/krb5.conf	M/I
SASL_GSSAPI_DISABLE_FAST_NEGOTIATION	Disable FAST negotiation.	false	M/I

Broker Collection filtering

Setting	Description	Default	Applies To
LOCAL_ONLY_COLLECTION	Collect only the metrics related to the configured bootstrap broker. Only used if `autodiscover_strategy` is `bootstrap`. Environments that use discovery (such as Kubernetes) must be set to true because othwerwise brokers will be discovered twice: By the integration, and by the discovery mechanism, leading to duplicate data. Note that activating this flag will skip KafkaTopicSample collection	false	M/I
TOPIC_MODE	Determines how many topics we collect. Options are `all`, `none`, `list`, or `regex`.	none	M/I
TOPIC_LIST	JSON array of topic names to monitor. Only in effect if `topic_mode` is set to `list`.	[]	M/I
TOPIC_REGEX	Regex pattern that matches the topic names to monitor. Only in effect if `topic_mode` is set to `regex`.	N/A	M/I
TOPIC_BUCKET	Used to split topic collection across multiple instances. Should be of the form `<bucket number>/<number of buckets>`.	1/1	M/I
COLLECT_TOPIC_SIZE	Collect the metric Topic size. Options are `true` or `false`, defaults to `false`. This is a resource-intensive metric to collect, especially against many topics.	false	M/I
COLLECT_TOPIC_OFFSET	Collect the metric Topic offset. Options are `true` or `false`, defaults to `false`. This is a resource-intensive metric to collect, especially against many topics.	false	M/I

Configure KafkaConsumerSample and KafkaProducerSample collection

The Kafka integration collects both Metrics(M) and Inventory(I) information. Check the Applies To column below to find which settings can be used for each specific collection:

Setting	Description	Default	Applies To
CLUSTER_NAME	User-defined name to uniquely identify the cluster being monitored. Required.	N/A	M/I
PRODUCERS	Producers to collect. For each producer a `name`, `hostname`, `port`, `username`, and `password` can be specified in JSON form. `name` is the producer’s name as it appears in Kafka. If it is not set, metrics from all producers in the host:port will be gathered. `host`, `port`, `username`, and `password` are the optional JMX settings and use the default if unspecified. It is also possible to set the value `default` to let the `name` undefined and use the default values for `host`, `port`, `username` and `password`. Required to produce KafkaProducerSample. Examples: `[{"host": "localhost", "port": 24, "username": "me", "password": "secret"}]` `[{"name": "myProducer", "host": "localhost", "port": 24, "username": "me", "password": "secret"}]`	[]	M/I
CONSUMERS	Consumers to collect. For each consumer a `name`, `hostname`, `port`, `username`, and `password` can be specified in JSON form. `name` is the consumer’s name as it appears in Kafka. If it is not set, metrics from all consumers in the host:port will be gathered. `host`, `port`, `username`, and `password` are the optional JMX settings and use the default if unspecified. It is also possible to set the value `default` to let the `name` undefined and use the default values for `host`, `port`, `username` and `password`. Required to produce KafkaConsumerSample. Examples: `[{"host": "localhost", "port": 24, "username": "me", "password": "secret"}]` `[{"name": "myConsumer", "host": "localhost", "port": 24, "username": "me", "password": "secret"}]`	[]	M/I
DEFAULT_JMX_HOST	The default host to collect JMX metrics. If the host field is omitted from a producer or consumer configuration, this value will be used.	localhost	M/I
DEFAULT_JMX_PORT	The default port to collect JMX metrics. If the port field is omitted from a producer or consumer configuration, this value will be used.	9999	M/I
DEFAULT_JMX_USER	The default user that is connecting to the JMX host to collect metrics. If the username field is omitted from a producer or consumer configuration, this value will be used.	admin	M/I
DEFAULT_JMX_PASSWORD	The default password to connect to the JMX host. If the password field is omitted from a producer or consumer configuration, this value will be used.	admin	M/I
METRICS	Set to `true` to enable Metrics only collection.	false
INVENTORY	Set to `true` to enable Inventory only collection.	false

JMX SSL and timeout options

These options apply to all JMX connections on the instance.

Setting	Description	Default	Applies To
KEY_STORE	The filepath of the keystore containing the JMX client's SSL certificate.	N/A	M/I
KEY_STORE_PASSWORD	The password for the JMX SSL key store.	N/A	M/I
TRUST_STORE	The filepath of the trust keystore containing the JMX server's SSL certificate.	N/A	M/I
TRUST_STORE_PASSWORD	The password for the JMX trust store.	N/A	M/I
TIMEOUT	The timeout for individual JMX queries in milliseconds.	10000	M/I

Configure KafkaOffsetSample collection

The Kafka integration collects both Metrics and Inventory information. Check the Applies To column below to find which settings can be used for each specific collection:

Setting	Description	Default	Applies To
CLUSTER_NAME	User-defined name to uniquely identify the cluster being monitored. Required.	N/A	M/I
KAFKA_VERSION	The version of the Kafka broker you're connecting to, used for setting optimum API versions. It must match -or be lower than- the version from the broker. Versions older than 1.0.0 may be missing some features. Note that if the broker binary name is kafka_2.12-2.7.0 the Kafka api version to be used is 2.7.0, the preceding 2.12 is the Scala language version.	1.0.0	M/I
AUTODISCOVER_STRATEGY	the method of discovering brokers. Options are `zookeeper` or `bootstrap`.	zookeeper	M/I
CONSUMER_OFFSET	Populate consumer offset data in KafkaOffsetSample if set to true. Note that this option will skip Broker/Consumer/Producer collection and only collect KafkaOffsetSample	false	M/I
CONSUMER_GROUP_REGEX	regex pattern that matches the consumer groups to collect offset statistics for. This is limited to collecting statistics for 300 consumer groups. Note: This option must be set when CONSUMER_OFFSET is true.	N/A	M/I
INACTIVE_CONSUMER_GROUP_OFFSET	Collects offset metrics from consumer groups without any active consumer, it requires CONSUMER_OFFSET set to true.	false	M/I
CONSUMER_GROUP_OFFSET_BY_TOPIC	Activates an extra metric aggregation for consumerGroup by topic. it requires CONSUMER_OFFSET set to true.	N/A	M/I
METRICS	Set to `true` to enable Metrics only collection.	false
INVENTORY	Set to `true` to enable Inventory only collection.	false

Zookeeper autodiscovery arguments

This is only relevant when the autodiscover_strategy option is set to zookeeper.

Setting	Description	Default	Applies To
ZOOKEEPER_HOSTS	The list of Apache ZooKeeper hosts (in JSON format) that need to be connected. If `CONSUMER_OFFSET` is set to `false` `KafkaBrokerSamples` and `KafkaTopicSamples` will be collected.	[]	M/I
ZOOKEEPER_AUTH_SCHEME	The ZooKeeper authentication scheme that is used to connect. Currently, the only supported value is `digest`. If omitted, no authentication is used.	N/A	M/I
ZOOKEEPER_AUTH_SECRET	The ZooKeeper authentication secret that is used to connect. Should be of the form `username:password`. Only required if `zookeeper_auth_scheme` is specified.	N/A	M/I
ZOOKEEPER_PATH	The Zookeeper node under which the Kafka configuration resides. Defaults to `/`.	N/A	M/I
PREFERRED_LISTENER	Use a specific listener to connect to a broker. If unset, the first listener that passes a successful test connection is used. Supported values are `PLAINTEXT`, `SASL_PLAINTEXT`, `SSL`, and `SASL_SSL`. *Note the `SASL_` protocols only support Kerberos (GSSAPI) authentication.**	N/A	M/I

Bootstrap broker discovery arguments

This is only relevant when the autodiscover_strategy option is set to bootstrap.

Setting	Description	Default	Applies To
BOOTSTRAP_BROKER_HOST	The host for the bootstrap broker. If `CONSUMER_OFFSET` is set to `false` `KafkaBrokerSamples` and `KafkaTopicSamples` will be collected.	N/A	M/I
BOOTSTRAP_BROKER_KAFKA_PORT	The Kafka port for the bootstrap broker.	N/A	M/I
BOOTSTRAP_BROKER_KAFKA_PROTOCOL	The protocol to use to connect to the bootstrap broker. Supported values are `PLAINTEXT`, `SASL_PLAINTEXT`, `SSL`, and `SASL_SSL`. *Note the `SASL_` protocols only support Kerberos (GSSAPI) authentication.**	PLAINTEXT	M/I
BOOTSTRAP_BROKER_JMX_PORT	The JMX port to use for collection on each broker in the cluster. Note that all discovered brokers should have JMX active on this port	N/A	M/I
BOOTSTRAP_BROKER_JMX_USER	The JMX user to use for collection on each broker in the cluster.	N/A	M/I
BOOTSTRAP_BROKER_JMX_PASSWORD	The JMX password to use for collection on each broker in the cluster.	N/A	M/I

JMX SSL and timeout options

These apply to all JMX connections on an instance.

Setting	Description	Default	Applies To
KEY_STORE	The filepath of the keystore containing the JMX client's SSL certificate.	N/A	M/I
KEY_STORE_PASSWORD	The password for the JMX SSL key store.	N/A	M/I
TRUST_STORE	The filepath of the trust keystore containing the JMX server's SSL certificate.	N/A	M/I
TRUST_STORE_PASSWORD	The password for the JMX trust store.	N/A	M/I
DEFAULT_JMX_USER	The default user that is connecting to the JMX host to collect metrics. If the username field is omitted for a JMX host, this value will be used.	admin	M/I
DEFAULT_JMX_PASSWORD	The default password to connect to the JMX host. If the password field is omitted for a JMX host, this value will be used.	admin	M/I
TIMEOUT	The timeout for individual JMX queries in milliseconds.	10000	M/I

Troubleshooting

The Kafka integration uses a JMX helper tool called nrjmx to retrieve JMX metrics from brokers, consumers, and producers. JMX needs to be enabled and configured on all brokers in the cluster. Also, firewalls need to be tuned to allow connections from the host running the integration to the brokers over the JMX port.

To check whether JMX is correctly configured, run the following command for each broker from the machine running the Kafka integration. Replace the PORT, USERNAME, and PASSWORD variables with the corresponding JMX settings for the brokers:

bash

$echo "*:*" | nrjmx -hostname MY_HOSTNAME -port MY_PORT -v -username MY_USERNAME -password MY_PASSWORD

The command should generate the output showing a long series of metrics without any errors.

The integration might show an error like the following:

bash

$KRB Error: (6) KDC_ERR_C_PRINCIPAL_UNKNOWN Client not found in Kerberos database

Check the keytab with kinit command. Replace the highlighted fields with your values:

bash

$$ kinit -k -t KEY_TAB_PATH USERNAME

If the username/keytab combination is correct, the command above should finish without printing any errors.

Check the realm using klist command:

bash

$$ klist |grep "Default principal:"

You should see something like this:

bash

$Default principal: johndoe@a_realm_name

Check that the printed user name and realm match the sasl_gssapi_realm and sasl_gssapi_username parameters in the integration configuration.

Kafka's integration configuration settings

Labels and custom attributes

Inventory data

Configure KafkaBrokerSample and KafkaTopicSample collection

Zookeeper autodiscovery arguments

Bootstrap broker discovery arguments

JMX options

Broker TLS connection options

Broker SASL and Kerberos connection options

Broker Collection filtering

Configure KafkaConsumerSample and KafkaProducerSample collection

JMX SSL and timeout options

Configure KafkaOffsetSample collection

Zookeeper autodiscovery arguments

Bootstrap broker discovery arguments

JMX SSL and timeout options

Troubleshooting

Duplicate data being reported

Integration is logging errors 'zk: node not found'

JMX connection errors

Kerberos authentication failing

Kafka's integration configuration settings

Labels and custom attributes .css-21sua1{background:none;border:none;width:0;padding:0;}

Inventory data

Configure KafkaBrokerSample and KafkaTopicSample collection

Zookeeper autodiscovery arguments

Bootstrap broker discovery arguments

JMX options

Broker TLS connection options

Broker SASL and Kerberos connection options

Broker Collection filtering

Configure KafkaConsumerSample and KafkaProducerSample collection

JMX SSL and timeout options

Configure KafkaOffsetSample collection

Zookeeper autodiscovery arguments

Bootstrap broker discovery arguments

JMX SSL and timeout options

Troubleshooting

Integration is logging errors 'zk: node not found'

JMX connection errors

Kerberos authentication failing

Labels and custom attributes