Monitorer Kafka sur Kubernetes (Strimzi) avec OpenTelemetry

Monitorez votre cluster Kafka fonctionnant sur Kubernetes avec l'opérateur Strimzi en déployant l'OpenTelemetry Collector. Le collecteur découvre automatiquement les pods de broker Kafka et collecte des métriques complètes.

Architecture

Le diagramme suivant illustre l'architecture de monitoring et le flux de données vers New Relic.

Étapes d'installation

Suivez ces étapes pour configurer la surveillance de votre cluster Kafka :

Avant de commencer

Assurez-vous d'avoir :

Un compte New Relic avec un
Cluster Kubernetes avec accès kubectl
Kafka déployé via l'opérateur Strimzi

Configurer le cluster Kafka pour les métriques JMX Kafka

Configurez votre cluster Strimzi Kafka pour exposer les métriques JMX Kafka via le Prometheus JMX Exporter. Cette configuration sera déployée en tant que ConfigMap et référencée par votre cluster Kafka.

Step 1. Create JMX metrics ConfigMap

Créez une ConfigMap avec des patterns JMX Exporter qui définissent les métriques Kafka à collecter. Enregistrer sous kafka-jmx-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-jmx-metrics
  namespace: newrelic
data:
  kafka-metrics-config.yml: |
    startDelaySeconds: 0
    lowercaseOutputName: true
    lowercaseOutputLabelNames: true

    rules:
      # Cluster-level controller metrics
      - pattern: 'kafka.controller<type=KafkaController, name=GlobalTopicCount><>Value'
        name: kafka_cluster_topic_count
        type: GAUGE

      - pattern: 'kafka.controller<type=KafkaController, name=GlobalPartitionCount><>Value'
        name: kafka_cluster_partition_count
        type: GAUGE

      - pattern: 'kafka.controller<type=KafkaController, name=FencedBrokerCount><>Value'
        name: kafka_broker_fenced_count
        type: GAUGE

      - pattern: 'kafka.controller<type=KafkaController, name=PreferredReplicaImbalanceCount><>Value'
        name: kafka_partition_non_preferred_leader
        type: GAUGE

      - pattern: 'kafka.controller<type=KafkaController, name=OfflinePartitionsCount><>Value'
        name: kafka_partition_offline
        type: GAUGE

      - pattern: 'kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value'
        name: kafka_controller_active_count
        type: GAUGE

      # Broker-level replica metrics
      - pattern: 'kafka.server<type=ReplicaManager, name=UnderMinIsrPartitionCount><>Value'
        name: kafka_partition_under_min_isr
        type: GAUGE

      - pattern: 'kafka.server<type=ReplicaManager, name=LeaderCount><>Value'
        name: kafka_broker_leader_count
        type: GAUGE

      - pattern: 'kafka.server<type=ReplicaManager, name=PartitionCount><>Value'
        name: kafka_partition_count
        type: GAUGE

      - pattern: 'kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value'
        name: kafka_partition_under_replicated
        type: GAUGE

      - pattern: 'kafka.server<type=ReplicaManager, name=IsrShrinksPerSec><>Count'
        name: kafka_isr_operation_count
        type: COUNTER
        labels:
          operation: "shrink"

      - pattern: 'kafka.server<type=ReplicaManager, name=IsrExpandsPerSec><>Count'
        name: kafka_isr_operation_count
        type: COUNTER
        labels:
          operation: "expand"

      - pattern: 'kafka.server<type=ReplicaFetcherManager, name=MaxLag, clientId=Replica><>Value'
        name: kafka_max_lag
        type: GAUGE

      # Broker topic metrics (totals)
      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec><>Count'
        name: kafka_message_count
        type: COUNTER

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=TotalFetchRequestsPerSec><>Count'
        name: kafka_request_count
        type: COUNTER
        labels:
          type: "fetch"

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=TotalProduceRequestsPerSec><>Count'
        name: kafka_request_count
        type: COUNTER
        labels:
          type: "produce"

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=FailedFetchRequestsPerSec><>Count'
        name: kafka_request_failed
        type: COUNTER
        labels:
          type: "fetch"

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=FailedProduceRequestsPerSec><>Count'
        name: kafka_request_failed
        type: COUNTER
        labels:
          type: "produce"

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=BytesInPerSec><>Count'
        name: kafka_network_io
        type: COUNTER
        labels:
          direction: "in"

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=BytesOutPerSec><>Count'
        name: kafka_network_io
        type: COUNTER
        labels:
          direction: "out"

      # Per-topic metrics (only appear after traffic flows)
      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec, topic=(.+)><>Count'
        name: kafka_prod_msg_count
        type: COUNTER
        labels:
          topic: "$1"

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=BytesInPerSec, topic=(.+)><>Count'
        name: kafka_topic_io
        type: COUNTER
        labels:
          topic: "$1"
          direction: "in"

      - pattern: 'kafka.server<type=BrokerTopicMetrics, name=BytesOutPerSec, topic=(.+)><>Count'
        name: kafka_topic_io
        type: COUNTER
        labels:
          topic: "$1"
          direction: "out"

      # Request metrics
      - pattern: 'kafka.network<type=RequestMetrics, name=TotalTimeMs, request=(Produce|FetchConsumer|FetchFollower)><>99thPercentile'
        name: kafka_request_time_99p
        type: GAUGE
        labels:
          type: "$1"

      - pattern: 'kafka.network<type=RequestChannel, name=RequestQueueSize><>Value'
        name: kafka_request_queue
        type: GAUGE

      - pattern: 'kafka.server<type=DelayedOperationPurgatory, name=PurgatorySize, delayedOperation=(.+)><>Value'
        name: kafka_purgatory_size
        type: GAUGE
        labels:
          type: "$1"

      # Controller stats
      - pattern: 'kafka.controller<type=ControllerStats, name=LeaderElectionRateAndTimeMs><>Count'
        name: kafka_leader_election_rate
        type: COUNTER

      - pattern: 'kafka.controller<type=ControllerStats, name=UncleanLeaderElectionsPerSec><>Count'
        name: kafka_unclean_election_rate
        type: COUNTER

      # JVM Garbage Collection
      - pattern: 'java.lang<name=(.+), type=GarbageCollector><>CollectionCount'
        name: jvm_gc_collections_count
        type: COUNTER
        labels:
          name: "$1"

      # JVM Memory
      - pattern: 'java.lang<type=Memory><HeapMemoryUsage>max'
        name: jvm_memory_heap_max
        type: GAUGE

      - pattern: 'java.lang<type=Memory><HeapMemoryUsage>used'
        name: jvm_memory_heap_used
        type: GAUGE

      # JVM Threading and System
      - pattern: 'java.lang<type=Threading><>ThreadCount'
        name: jvm_thread_count
        type: GAUGE

      - pattern: 'java.lang<type=OperatingSystem><>SystemCpuLoad'
        name: jvm_system_cpu_utilization
        type: GAUGE

      # Broker uptime
      - pattern: 'java.lang<type=Runtime><>Uptime'
        name: kafka_broker_uptime
        type: GAUGE

      # Additional metrics — remove this section to reduce data ingest

      # Request latency: total count, 50th percentile, and average (99p kept above)
      - pattern: 'kafka.network<type=RequestMetrics, name=TotalTimeMs, request=(Produce|FetchConsumer|FetchFollower)><>Count'
        name: kafka_request_time_total
        type: COUNTER
        labels:
          type: "$1"

      - pattern: 'kafka.network<type=RequestMetrics, name=TotalTimeMs, request=(Produce|FetchConsumer|FetchFollower)><>50thPercentile'
        name: kafka_request_time_50p
        type: GAUGE
        labels:
          type: "$1"

      - pattern: 'kafka.network<type=RequestMetrics, name=TotalTimeMs, request=(Produce|FetchConsumer|FetchFollower)><>Mean'
        name: kafka_request_time_avg
        type: GAUGE
        labels:
          type: "$1"

      # Log flush metrics
      - pattern: 'kafka.log<type=LogFlushStats, name=LogFlushRateAndTimeMs><>Count'
        name: kafka_logs_flush_count
        type: COUNTER

      - pattern: 'kafka.log<type=LogFlushStats, name=LogFlushRateAndTimeMs><>50thPercentile'
        name: kafka_logs_flush_time_50p
        type: GAUGE

      - pattern: 'kafka.log<type=LogFlushStats, name=LogFlushRateAndTimeMs><>99thPercentile'
        name: kafka_logs_flush_time_99p
        type: GAUGE

      # JVM GC elapsed time
      - pattern: 'java.lang<name=(.+), type=GarbageCollector><>CollectionTime'
        name: jvm_gc_collections_elapsed
        type: COUNTER
        labels:
          name: "$1"

      # JVM Memory heap committed
      - pattern: 'java.lang<type=Memory><HeapMemoryUsage>committed'
        name: jvm_memory_heap_committed
        type: GAUGE

      # JVM class loading
      - pattern: 'java.lang<type=ClassLoading><>LoadedClassCount'
        name: jvm_class_count
        type: GAUGE

      # Additional JVM OS metrics
      - pattern: 'java.lang<type=OperatingSystem><>SystemLoadAverage'
        name: jvm_system_cpu_load_1m
        type: GAUGE

      - pattern: 'java.lang<type=OperatingSystem><>AvailableProcessors'
        name: jvm_cpu_count
        type: GAUGE

      - pattern: 'java.lang<type=OperatingSystem><>ProcessCpuLoad'
        name: jvm_cpu_recent_utilization
        type: GAUGE

      - pattern: 'java.lang<type=OperatingSystem><>OpenFileDescriptorCount'
        name: jvm_file_descriptor_count
        type: GAUGE

      # JVM Memory Pool
      - pattern: 'java.lang<type=MemoryPool, name=(.+)><Usage>used'
        name: jvm_memory_pool_used
        type: GAUGE
        labels:
          name: "$1"

      - pattern: 'java.lang<type=MemoryPool, name=(.+)><Usage>max'
        name: jvm_memory_pool_max
        type: GAUGE
        labels:
          name: "$1"

      - pattern: 'java.lang<type=MemoryPool, name=(.+)><CollectionUsage>used'
        name: jvm_memory_pool_used_after_last_gc
        type: GAUGE
        labels:
          name: "$1"

Conseil

Personnaliser les métriques: cette ConfigMap inclut des métriques complètes sur les brokers Kafka, les topics, les requêtes, les contrôleurs et la JVM. Vous pouvez ajouter ou modifier des motifs en consultant les exemples Prometheus JMX Exporter et la documentation des MBeans Kafka. Consultez la documentation des règles de JMX Exporter pour des configurations supplémentaires.

Important

Exigence relative à l'espace de noms: La ConfigMap des métriques JMX et votre cluster Kafka doivent se trouver dans le même espace de noms. Dans ce guide, les deux sont déployés dans l'espace de noms newrelic.

Appliquer la ConfigMap :

bash

$kubectl apply -f kafka-jmx-config.yaml

Step 2. Update Kafka cluster to use JMX Exporter

Mettez à jour votre ressource Strimzi Kafka pour référencer la ConfigMap des métriques :

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: newrelic
spec:
  kafka:
    version: X.X.X
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-jmx-metrics
          key: kafka-metrics-config.yml
    # ...rest of your Kafka configuration

Appliquer les modifications. Strimzi effectuera un redémarrage progressif de vos brokers Kafka :

bash

$kubectl apply -f kafka-cluster.yaml

Une fois le redémarrage progressif terminé, chaque broker Kafka exposera des métriques Prometheus sur le port 9404.

Déployer le Collecteur OpenTelemetry

Déployez le Collecteur OpenTelemetry pour monitorer votre cluster Kafka. Choisissez votre méthode d'installation préférée :

La méthode d'installation Helm est l'approche recommandée pour déployer OpenTelemetry Collector dans Kubernetes.

Step 1. Create New Relic credentials secret

Créez un secret Kubernetes contenant votre clé de licence New Relic et votre point de terminaison OTLP. Choisissez le point de terminaison pour votre région New Relic :

bash

$kubectl create secret generic newrelic-otlp-secret \
>  --namespace newrelic \
>  --from-literal=NEW_RELIC_LICENSE_KEY='your-license-key-here' \
>  --from-literal=NEW_RELIC_OTLP_ENDPOINT='https://otlp.nr-data.net:4317'

bash

$kubectl create secret generic newrelic-otlp-secret \
>  --namespace newrelic \
>  --from-literal=NEW_RELIC_LICENSE_KEY='your-license-key-here' \
>  --from-literal=NEW_RELIC_OTLP_ENDPOINT='https://otlp.eu01.nr-data.net:4317'

Conseil

Pour d'autres configurations de point de terminaison, consultez Configurer votre point de terminaison OTLP.

Step 2. Create values.yaml with collector configuration

Créez un fichier values.yaml contenant la configuration complète du collecteur OpenTelemetry. Les collecteurs NRDOT et OpenTelemetry utilisent une configuration identique et offrent les mêmes capacités du monitoring Kafka. Choisissez votre image de collecteur préférée :

NRDOT est la distribution de l'OpenTelemetry Collector prise en charge par New Relic, offrant un support complet de New Relic. Pour plus d'informations, consultez le dépôt GitHub NRDOT Collector.

Créez values.yaml avec le contenu suivant :

# Deployment mode
mode: deployment
replicaCount: 1

# Use NRDOT collector image
image:
  repository: newrelic/nrdot-collector
  tag: "latest"
  pullPolicy: Always

# Service account configuration
serviceAccount:
  create: true
  name: otel-collector

# RBAC for Kubernetes service discovery
clusterRole:
  create: true
  rules:
    - apiGroups: [""]
      resources: ["pods", "nodes"]
      verbs: ["get", "list", "watch"]

# Pod security context
podSecurityContext:
  runAsNonRoot: true
  runAsUser: 10001

# Container security context
securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

# Resource limits
resources:
  requests:
    memory: 512Mi
    cpu: 250m
  limits:
    memory: 1Gi
    cpu: 500m

# Load environment variables from secret
extraEnvsFrom:
  - secretRef:
      name: newrelic-otlp-secret

# Disable unused default ports
ports:
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false

# OpenTelemetry Collector Configuration
config:
  receivers:
    # Disable default receivers not needed in NRDOT
    jaeger: null
    zipkin: null

    kafkametrics/cluster:
      brokers:
        # TODO#1: Replace with your Kafka bootstrap service
        - "my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092"
      collection_interval: 30s
      protocol_version: 2.0.0
      scrapers:
        - brokers
        - topics
        - consumers
      # Exclude internal Kafka topics (prefixed with __) at the source
      topic_match: "^[^_].*$"
      metrics:
        kafka.topic.min_insync_replicas:
          enabled: true
        kafka.topic.replication_factor:
          enabled: true
        kafka.partition.replicas:
          enabled: false
        kafka.partition.oldest_offset:
          enabled: false
        kafka.partition.current_offset:
          enabled: false

    prometheus/kafka-jmx:
      config:
        scrape_configs:
          - job_name: 'kafka-jmx-metrics'
            scrape_interval: 30s
            kubernetes_sd_configs:
              - role: pod
                namespaces:
                  names:
                    # TODO#2: Replace with the namespace where your Kafka cluster is deployed
                    - newrelic
            relabel_configs:
              # Filter for Kafka broker pods
              - source_labels: [__meta_kubernetes_pod_label_strimzi_io_name]
                action: keep
                # TODO#3: Replace with your Strimzi Kafka cluster name followed by '-kafka'
                regex: my-cluster-kafka

              - source_labels: [__meta_kubernetes_pod_label_strimzi_io_cluster]
                action: keep
                # TODO#4: Replace with your Strimzi Kafka cluster name
                regex: my-cluster

              # Extract broker ID from pod name
              - source_labels: [__meta_kubernetes_pod_name]
                target_label: broker.id
                regex: '.*-(\\d+)$'
                replacement: '$1'

              # Set scrape target to pod IP on port 9404
              - source_labels: [__meta_kubernetes_pod_ip]
                target_label: __address__
                replacement: '$1:9404'

  exporters:
    # New Relic OTLP exporter
    otlp/backend:
      endpoint: ${NEW_RELIC_OTLP_ENDPOINT}
      tls:
        insecure: false
      sending_queue:
        num_consumers: 12
        queue_size: 5000
      retry_on_failure:
        enabled: true
      headers:
        api-key: ${NEW_RELIC_LICENSE_KEY}

  processors:
    # Batch processor for efficient export
    batch/export:
      send_batch_size: 1024
      timeout: 30s

    # Memory limiter to prevent OOM
    memory_limiter:
      limit_percentage: 80
      spike_limit_percentage: 30
      check_interval: 1s

    # Transform metric naming conventions
    transform/metric-naming:
      metric_statements:
        - context: metric
          statements:
            - replace_pattern(name, "_", ".")
            - replace_pattern(name, "\\.load\\.1", ".load_1")
            - replace_pattern(name, "\\.recent\\.util", ".recent_util")
            - replace_pattern(name, "file\\.descriptor\\.count", "file_descriptor.count")
            - replace_pattern(name, "\\.memory\\.pool\\.used\\.bytes$", ".memory.pool.used")
            - replace_pattern(name, "\\.memory\\.pool\\.max\\.bytes$", ".memory.pool.max")
            - replace_pattern(name, "\\.memory\\.pool\\.collection\\.used\\.bytes$", ".memory.pool.used_after_last_gc")
            - replace_pattern(name, "\\.non\\.preferred\\.leader", ".non_preferred_leader")
            - replace_pattern(name, "\\.under\\.min\\.isr", ".under_min_isr")
            - replace_pattern(name, "\\.under\\.replicated", ".under_replicated")
            - replace_pattern(name, "\\.total$", "") where name != "kafka.request.time.total"
        - context: datapoint
          statements:
            - set(attributes["name"], attributes["gc"]) where attributes["gc"] != nil
            - delete_key(attributes, "gc") where attributes["gc"] != nil
            - set(attributes["name"], attributes["pool"]) where attributes["pool"] != nil
            - delete_key(attributes, "pool") where attributes["pool"] != nil

    # Add cluster name to all metrics
    resource/cluster-name:
      attributes:
        - key: kafka.cluster.name
          # TODO#5: Replace with your Kafka cluster name (used to identify and filter metrics in New Relic)
          value: my-cluster
          action: upsert

    # Remove broker.id for cluster-level metrics
    transform/remove_broker_id:
      metric_statements:
        - context: datapoint
          statements:
            - delete_key(attributes, "broker.id")

    # Filter out scrape overhead metrics
    filter/scrape-overhead:
      metrics:
        exclude:
          match_type: regexp
          metric_names:
            - "^jmx_.*"
            - "^process_.*"
            - "^jvm_buffer_pool_.*"
            - "^jvm_threads_.*"
            - "^jvm_classes_.*"
            - "^jvm_memory_(heap|non_heap)_(committed|init|max|used)_bytes$"
            - "^jvm_compilation_.*"
            - "^jvm_(runtime|info).*"
            - "^jvm_memory_pool_(allocated_bytes_total|committed_bytes|init_bytes|collection_(committed|init|max)_bytes)$"

    # Include only cluster-level metrics for cluster pipeline
    filter/include_cluster_metrics:
      metrics:
        include:
          match_type: regexp
          metric_names:
            - "^kafka\\.partition\\.offline$"
            - "^kafka\\.(leader|unclean)\\.election\\.rate$"
            - "^kafka\\.partition\\.non_preferred_leader$"
            - "^kafka\\.broker\\.fenced\\.count$"
            - "^kafka\\.cluster\\.partition\\.count$"
            - "^kafka\\.cluster\\.topic\\.count$"

    # Exclude cluster-level metrics from broker pipeline
    filter/exclude_cluster_metrics:
      metrics:
        exclude:
          match_type: regexp
          metric_names:
            - "^kafka\\.partition\\.offline$"
            - "^kafka\\.(leader|unclean)\\.election\\.rate$"
            - "^kafka\\.partition\\.non_preferred_leader$"
            - "^kafka\\.broker\\.fenced\\.count$"
            - "^kafka\\.cluster\\.partition\\.count$"
            - "^kafka\\.cluster\\.topic\\.count$"

    # Remove unnecessary attributes
    transform/remove_attributes:
      metric_statements:
        - context: metric
          statements:
            - set(description, "") where description != ""
            - set(unit, "") where unit != ""
        - context: resource
          statements:
            - delete_key(attributes, "server.address")
            - delete_key(attributes, "server.port")
            - delete_key(attributes, "service.instance.id")
            - delete_key(attributes, "host.name")
            - delete_key(attributes, "k8s.pod.uid")
            - delete_key(attributes, "url.scheme")

    # Aggregate partition metrics to topic level
    metricstransform/topic-aggregation:
      transforms:
        - include: kafka.partition.replicas_in_sync
          action: insert
          new_name: kafka.partition.replicas_in_sync.total
          operations:
            - action: aggregate_labels
              label_set: [topic]
              aggregation_type: sum
        - include: kafka.partition.replicas
          action: insert
          new_name: kafka.partition.replicas.total
          operations:
            - action: aggregate_labels
              label_set: [topic]
              aggregation_type: sum

    # Filter out original partition replicas metric
    filter/exclude_partition_replicas_metric:
      metrics:
        exclude:
          match_type: strict
          metric_names:
            - kafka.partition.replicas_in_sync

    # Filter internal Kafka topics as a safety net (topic_match handles the receiver side)
    filter/internal_topics:
      metrics:
        datapoint:
          - 'attributes["topic"] != nil and IsMatch(attributes["topic"], "^__.*")'

    # Convert cumulative to delta metrics
    cumulativetodelta:

    groupbyattrs/cluster:
      keys: [kafka.cluster.name]

    metricstransform/cluster_max:
      transforms:
        - include: "kafka\\.partition\\.offline|kafka\\.leader\\.election\\.rate|kafka\\.unclean\\.election\\.rate|kafka\\.partition\\.non_preferred_leader|kafka\\.broker\\.fenced\\.count|kafka\\.cluster\\.partition\\.count|kafka\\.cluster\\.topic\\.count"
          match_type: regexp
          action: update
          operations:
            - action: aggregate_labels
              aggregation_type: max
              label_set: []

  service:
    pipelines:
      # Suppress default pipelines — only custom Kafka metrics pipelines are used
      traces: null
      logs: null
      metrics: null

      # Broker-level metrics from Prometheus JMX scraping
      metrics/broker:
        receivers:
          - prometheus/kafka-jmx
        processors:
          - resource/cluster-name
          - filter/scrape-overhead
          - transform/metric-naming
          - transform/remove_attributes
          - filter/exclude_cluster_metrics
          - memory_limiter
          - cumulativetodelta
          - batch/export
        exporters:
          - otlp/backend

      # Cluster-level metrics from Prometheus JMX scraping
      metrics/cluster/prometheus:
        receivers:
          - prometheus/kafka-jmx
        processors:
          - resource/cluster-name
          - filter/scrape-overhead
          - transform/metric-naming
          - transform/remove_attributes
          - filter/include_cluster_metrics
          - transform/remove_broker_id
          - memory_limiter
          - cumulativetodelta
          - groupbyattrs/cluster
          - metricstransform/cluster_max
          - batch/export
        exporters:
          - otlp/backend

      # Cluster-level metrics from Kafka metrics receiver
      metrics/cluster/kafkametrics:
        receivers:
          - kafkametrics/cluster
        processors:
          - resource/cluster-name
          - filter/internal_topics
          - transform/remove_attributes
          - metricstransform/topic-aggregation
          - filter/exclude_partition_replicas_metric
          - memory_limiter
          - cumulativetodelta
          - batch/export
        exporters:
          - otlp/backend

Paramètres de configuration

Le tableau suivant décrit les principaux paramètres de configuration :

paramètres	Description
`config.receivers.kafkametrics/cluster.brokers`	Replace with your Kafka bootstrap service
`config.receivers.prometheus/kafka-jmx.config.scrape_configs[0].kubernetes_sd_configs[0].namespaces.names`	Replace with the namespace where your Kafka cluster is deployed
`config.receivers.prometheus/kafka-jmx.config.scrape_configs[0].relabel_configs[strimzi_io_name].regex`	Replace with your Strimzi Kafka cluster name followed by `-kafka`
`config.receivers.prometheus/kafka-jmx.config.scrape_configs[0].relabel_configs[strimzi_io_cluster].regex`	Replace with your Strimzi Kafka cluster name
`config.processors.resource/cluster-name.attributes[kafka.cluster.name].value`	Replace with your Kafka cluster name (this will be used to identify and filter your metrics in New Relic)
`resources.limits` et `resources.requests`	Adjust according to your workload needs

Utilisez l'OpenTelemetry Collector communautaire pour une flexibilité maximale et un déploiement indépendant des fournisseurs.

Créez values.yaml avec le contenu suivant :

# Deployment mode
mode: deployment
replicaCount: 1

# Use contrib image for kafkametrics receiver
image:
  repository: otel/opentelemetry-collector-contrib
  tag: "latest"
  pullPolicy: Always

# Service account configuration
serviceAccount:
  create: true
  name: otel-collector

# RBAC for Kubernetes service discovery
clusterRole:
  create: true
  rules:
    - apiGroups: [""]
      resources: ["pods", "nodes"]
      verbs: ["get", "list", "watch"]

# Pod security context
podSecurityContext:
  runAsNonRoot: true
  runAsUser: 10001

# Container security context
securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

# Resource limits
resources:
  requests:
    memory: 512Mi
    cpu: 250m
  limits:
    memory: 1Gi
    cpu: 500m

# Load environment variables from secret
extraEnvsFrom:
  - secretRef:
      name: newrelic-otlp-secret

# OpenTelemetry Collector Configuration
config:
  receivers:
    # Kafka metrics receiver for cluster-level metrics
    kafkametrics/cluster:
      brokers:
        # TODO#1: Replace with your Kafka bootstrap service
        - "my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092"
      collection_interval: 30s
      protocol_version: 2.0.0
      scrapers:
        - brokers
        - topics
        - consumers
      # Exclude internal Kafka topics (prefixed with __) at the source
      topic_match: "^[^_].*$"
      metrics:
        kafka.topic.min_insync_replicas:
          enabled: true
        kafka.topic.replication_factor:
          enabled: true
        kafka.partition.replicas:
          enabled: false
        kafka.partition.oldest_offset:
          enabled: false
        kafka.partition.current_offset:
          enabled: false

    # Prometheus receiver for JMX metrics from Kafka brokers
    prometheus/kafka-jmx:
      config:
        scrape_configs:
          - job_name: 'kafka-jmx-metrics'
            scrape_interval: 30s
            kubernetes_sd_configs:
              - role: pod
                namespaces:
                  names:
                    # TODO#2: Replace with the namespace where your Kafka cluster is deployed
                    - newrelic
            relabel_configs:
              # Filter for Kafka broker pods
              - source_labels: [__meta_kubernetes_pod_label_strimzi_io_name]
                action: keep
                # TODO#3: Replace with your Strimzi Kafka cluster name followed by '-kafka'
                regex: my-cluster-kafka

              - source_labels: [__meta_kubernetes_pod_label_strimzi_io_cluster]
                action: keep
                # TODO#4: Replace with your Strimzi Kafka cluster name
                regex: my-cluster

              # Extract broker ID from pod name
              - source_labels: [__meta_kubernetes_pod_name]
                target_label: broker.id
                regex: '.*-(\\d+)$'
                replacement: '$1'

              # Set scrape target to pod IP on port 9404
              - source_labels: [__meta_kubernetes_pod_ip]
                target_label: __address__
                replacement: '$1:9404'

  exporters:
    # New Relic OTLP exporter
    otlp/backend:
      endpoint: ${NEW_RELIC_OTLP_ENDPOINT}
      tls:
        insecure: false
      sending_queue:
        num_consumers: 12
        queue_size: 5000
      retry_on_failure:
        enabled: true
      headers:
        api-key: ${NEW_RELIC_LICENSE_KEY}

  processors:
    # Batch processor for efficient export
    batch/export:
      send_batch_size: 1024
      timeout: 30s

    # Memory limiter to prevent OOM
    memory_limiter:
      limit_percentage: 80
      spike_limit_percentage: 30
      check_interval: 1s

    # Transform metric naming conventions
    transform/metric-naming:
      metric_statements:
        - context: metric
          statements:
            - replace_pattern(name, "_", ".")
            - replace_pattern(name, "\\.load\\.1", ".load_1")
            - replace_pattern(name, "\\.recent\\.util", ".recent_util")
            - replace_pattern(name, "file\\.descriptor\\.count", "file_descriptor.count")
            - replace_pattern(name, "\\.memory\\.pool\\.used\\.bytes$", ".memory.pool.used")
            - replace_pattern(name, "\\.memory\\.pool\\.max\\.bytes$", ".memory.pool.max")
            - replace_pattern(name, "\\.memory\\.pool\\.collection\\.used\\.bytes$", ".memory.pool.used_after_last_gc")
            - replace_pattern(name, "\\.non\\.preferred\\.leader", ".non_preferred_leader")
            - replace_pattern(name, "\\.under\\.min\\.isr", ".under_min_isr")
            - replace_pattern(name, "\\.under\\.replicated", ".under_replicated")
            - replace_pattern(name, "\\.total$", "") where name != "kafka.request.time.total"
        - context: datapoint
          statements:
            - set(attributes["name"], attributes["gc"]) where attributes["gc"] != nil
            - delete_key(attributes, "gc") where attributes["gc"] != nil
            - set(attributes["name"], attributes["pool"]) where attributes["pool"] != nil
            - delete_key(attributes, "pool") where attributes["pool"] != nil

    # Add cluster name to all metrics
    resource/cluster-name:
      attributes:
        - key: kafka.cluster.name
          # TODO#5: Replace with your Kafka cluster name (used to identify and filter metrics in New Relic)
          value: my-cluster
          action: upsert

    # Remove broker.id for cluster-level metrics
    transform/remove_broker_id:
      metric_statements:
        - context: datapoint
          statements:
            - delete_key(attributes, "broker.id")

    # Filter out scrape overhead metrics
    filter/scrape-overhead:
      metrics:
        exclude:
          match_type: regexp
          metric_names:
            - "^jmx_.*"
            - "^process_.*"
            - "^jvm_buffer_pool_.*"
            - "^jvm_threads_.*"
            - "^jvm_classes_.*"
            - "^jvm_memory_(heap|non_heap)_(committed|init|max|used)_bytes$"
            - "^jvm_compilation_.*"
            - "^jvm_(runtime|info).*"
            - "^jvm_memory_pool_(allocated_bytes_total|committed_bytes|init_bytes|collection_(committed|init|max)_bytes)$"

    # Include only cluster-level metrics for cluster pipeline
    filter/include_cluster_metrics:
      metrics:
        include:
          match_type: regexp
          metric_names:
            - "^kafka\\.partition\\.offline$"
            - "^kafka\\.(leader|unclean)\\.election\\.rate$"
            - "^kafka\\.partition\\.non_preferred_leader$"
            - "^kafka\\.broker\\.fenced\\.count$"
            - "^kafka\\.cluster\\.partition\\.count$"
            - "^kafka\\.cluster\\.topic\\.count$"

    # Exclude cluster-level metrics from broker pipeline
    filter/exclude_cluster_metrics:
      metrics:
        exclude:
          match_type: regexp
          metric_names:
            - "^kafka\\.partition\\.offline$"
            - "^kafka\\.(leader|unclean)\\.election\\.rate$"
            - "^kafka\\.partition\\.non_preferred_leader$"
            - "^kafka\\.broker\\.fenced\\.count$"
            - "^kafka\\.cluster\\.partition\\.count$"
            - "^kafka\\.cluster\\.topic\\.count$"

    # Remove unnecessary attributes
    transform/remove_attributes:
      metric_statements:
        - context: metric
          statements:
            - set(description, "") where description != ""
            - set(unit, "") where unit != ""
        - context: resource
          statements:
            - delete_key(attributes, "server.address")
            - delete_key(attributes, "server.port")
            - delete_key(attributes, "service.instance.id")
            - delete_key(attributes, "host.name")
            - delete_key(attributes, "k8s.pod.uid")
            - delete_key(attributes, "url.scheme")

    # Aggregate partition metrics to topic level
    metricstransform/topic-aggregation:
      transforms:
        - include: kafka.partition.replicas_in_sync
          action: insert
          new_name: kafka.partition.replicas_in_sync.total
          operations:
            - action: aggregate_labels
              label_set: [topic]
              aggregation_type: sum
        - include: kafka.partition.replicas
          action: insert
          new_name: kafka.partition.replicas.total
          operations:
            - action: aggregate_labels
              label_set: [topic]
              aggregation_type: sum

    # Filter out original partition replicas metric
    filter/exclude_partition_replicas_metric:
      metrics:
        exclude:
          match_type: strict
          metric_names:
            - kafka.partition.replicas_in_sync

    # Filter internal Kafka topics as a safety net (topic_match handles the receiver side)
    filter/internal_topics:
      metrics:
        datapoint:
          - 'attributes["topic"] != nil and IsMatch(attributes["topic"], "^__.*")'

    # Convert cumulative to delta metrics
    cumulativetodelta:

    groupbyattrs/cluster:
      keys: [kafka.cluster.name]

    metricstransform/cluster_max:
      transforms:
        - include: "kafka\\.partition\\.offline|kafka\\.leader\\.election\\.rate|kafka\\.unclean\\.election\\.rate|kafka\\.partition\\.non_preferred_leader|kafka\\.broker\\.fenced\\.count|kafka\\.cluster\\.partition\\.count|kafka\\.cluster\\.topic\\.count"
          match_type: regexp
          action: update
          operations:
            - action: aggregate_labels
              aggregation_type: max
              label_set: []

  service:
    pipelines:
      # Override default pipelines to only use custom Kafka metrics pipelines
      traces: null
      logs: null
      metrics: null

      # Broker-level metrics from Prometheus JMX scraping
      metrics/broker:
        receivers:
          - prometheus/kafka-jmx
        processors:
          - resource/cluster-name
          - filter/scrape-overhead
          - transform/metric-naming
          - transform/remove_attributes
          - filter/exclude_cluster_metrics
          - memory_limiter
          - cumulativetodelta
          - batch/export
        exporters:
          - otlp/backend

      # Cluster-level metrics from Prometheus JMX scraping
      metrics/cluster/prometheus:
        receivers:
          - prometheus/kafka-jmx
        processors:
          - resource/cluster-name
          - filter/scrape-overhead
          - transform/metric-naming
          - transform/remove_attributes
          - filter/include_cluster_metrics
          - transform/remove_broker_id
          - memory_limiter
          - cumulativetodelta
          - groupbyattrs/cluster
          - metricstransform/cluster_max
          - batch/export
        exporters:
          - otlp/backend

      # Cluster-level metrics from Kafka metrics receiver
      metrics/cluster/kafkametrics:
        receivers:
          - kafkametrics/cluster
        processors:
          - resource/cluster-name
          - filter/internal_topics
          - transform/remove_attributes
          - metricstransform/topic-aggregation
          - filter/exclude_partition_replicas_metric
          - memory_limiter
          - cumulativetodelta
          - batch/export
        exporters:
          - otlp/backend

Configuration parameters: Same parameters as the NRDOT option above. See the configuration parameters table for details, including resource limits.

Pour des options de configuration avancées, reportez-vous aux pages de documentation de ces récepteurs :

Documentation du récepteur Prometheus - Options de configuration supplémentaires du récepteur

Documentation du récepteur de métriques Kafka - Configuration de métriques Kafka supplémentaires

Step 3. Install OpenTelemetry Collector with Helm

Ajoutez le dépôt Helm et installez le collecteur OpenTelemetry à l'aide du fichier values.yaml :

bash

$helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
$helm upgrade kafka-monitoring open-telemetry/opentelemetry-collector \
>  --install \
>  --namespace newrelic \
>  --create-namespace \
>  -f values.yaml

Step 4. Verify the deployment:

bash

$# Check pod status
$kubectl get pods -n newrelic -l app.kubernetes.io/name=opentelemetry-collector
$
$# View logs to verify metrics collection
$kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector --tail=50

Vous devriez voir des logs indiquant un scraping réussi des brokers Kafka sur le port 9404.

La méthode d'installation par manifeste offre un contrôle direct sur les ressources Kubernetes sans utiliser Helm.

Step 1. Create New Relic credentials secret

Créez un secret Kubernetes contenant votre clé de licence New Relic et votre point de terminaison OTLP. Choisissez le point de terminaison pour votre région New Relic :

bash

$kubectl create secret generic newrelic-otlp-secret \
>  --namespace newrelic \
>  --from-literal=NEW_RELIC_LICENSE_KEY='your-license-key-here' \
>  --from-literal=NEW_RELIC_OTLP_ENDPOINT='https://otlp.nr-data.net:4317'

bash

$kubectl create secret generic newrelic-otlp-secret \
>  --namespace newrelic \
>  --from-literal=NEW_RELIC_LICENSE_KEY='your-license-key-here' \
>  --from-literal=NEW_RELIC_OTLP_ENDPOINT='https://otlp.eu01.nr-data.net:4317'

Conseil

Pour d'autres configurations de point de terminaison, consultez Configurer votre point de terminaison OTLP.

Step 2. Create manifest files

Créez les fichiers manifestes Kubernetes pour votre collecteur préféré. Les deux collecteurs utilisent une configuration identique - seule l'image diffère.

Choisissez votre option de collecteur et créez les trois fichiers requis :

Step 1. Create collector-rbac.yaml - RBAC configuration for Kubernetes API access:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector
  namespace: newrelic
  labels:
    app: otel-collector
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector
  labels:
    app: otel-collector
rules:
- apiGroups: [""]
  resources: ["pods", "nodes"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-collector
  labels:
    app: otel-collector
subjects:
- kind: ServiceAccount
  name: otel-collector
  namespace: newrelic
roleRef:
  kind: ClusterRole
  name: otel-collector
  apiGroup: rbac.authorization.k8s.io

Step 2. Create collector-configmap.yaml - OpenTelemetry Collector configuration:

---
apiVersion: v1
kind: ConfigMap
metadata:
name: otel-collector-config
namespace: newrelic
labels:
app: otel-collector
data:
otel-collector-config.yaml: |
receivers:
  kafkametrics/cluster:
    brokers:
      # TODO#1: Replace with your Kafka bootstrap service
      - "my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092"
    collection_interval: 30s
    protocol_version: 2.0.0
    scrapers:
      - brokers
      - topics
      - consumers
    # Exclude internal Kafka topics (prefixed with __) at the source
    topic_match: "^[^_].*$"
    metrics:
      kafka.topic.min_insync_replicas:
        enabled: true
      kafka.topic.replication_factor:
        enabled: true
      kafka.partition.replicas:
        enabled: false
      kafka.partition.oldest_offset:
        enabled: false
      kafka.partition.current_offset:
        enabled: false

  prometheus/kafka-jmx:
    config:
      scrape_configs:
        - job_name: 'kafka-jmx-metrics'
          scrape_interval: 30s
          kubernetes_sd_configs:
            - role: pod
              namespaces:
                names:
                  # TODO#2: Replace with the namespace where your Kafka cluster is deployed
                  - kafka
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_label_strimzi_io_name]
              action: keep
              # TODO#3: Replace with your Strimzi Kafka cluster name followed by '-kafka'
              regex: my-cluster-kafka
            - source_labels: [__meta_kubernetes_pod_label_strimzi_io_cluster]
              action: keep
              # TODO#4: Replace with your Strimzi Kafka cluster name
              regex: my-cluster
            - source_labels: [__meta_kubernetes_pod_name]
              target_label: broker.id
              regex: '.*-(\\d+)$'
              replacement: '$1'
            - source_labels: [__meta_kubernetes_pod_ip]
              target_label: __address__
              replacement: '$1:9404'

exporters:
  otlp/backend:
    endpoint: ${NEW_RELIC_OTLP_ENDPOINT}
    tls:
      insecure: false
    sending_queue:
      num_consumers: 12
      queue_size: 5000
    retry_on_failure:
      enabled: true
    headers:
      api-key: ${NEW_RELIC_LICENSE_KEY}

processors:
  batch/export:
    send_batch_size: 1024
    timeout: 30s
  memory_limiter:
    limit_percentage: 80
    spike_limit_percentage: 30
    check_interval: 1s
  transform/metric-naming:
    metric_statements:
    - context: metric
      statements:
      - replace_pattern(name, "_", ".")
      - replace_pattern(name, "\\.load\\.1", ".load_1")
      - replace_pattern(name, "\\.recent\\.util", ".recent_util")
      - replace_pattern(name, "file\\.descriptor\\.count", "file_descriptor.count")
      - replace_pattern(name, "\\.memory\\.pool\\.used\\.bytes$", ".memory.pool.used")
      - replace_pattern(name, "\\.memory\\.pool\\.max\\.bytes$", ".memory.pool.max")
      - replace_pattern(name, "\\.memory\\.pool\\.collection\\.used\\.bytes$", ".memory.pool.used_after_last_gc")
      - replace_pattern(name, "\\.non\\.preferred\\.leader", ".non_preferred_leader")
      - replace_pattern(name, "\\.under\\.min\\.isr", ".under_min_isr")
      - replace_pattern(name, "\\.under\\.replicated", ".under_replicated")
      - replace_pattern(name, "\\.total$", "") where name != "kafka.request.time.total"
    - context: datapoint
      statements:
      - set(attributes["name"], attributes["gc"]) where attributes["gc"] != nil
      - delete_key(attributes, "gc") where attributes["gc"] != nil
      - set(attributes["name"], attributes["pool"]) where attributes["pool"] != nil
      - delete_key(attributes, "pool") where attributes["pool"] != nil
  resource/cluster-name:
    attributes:
    - key: kafka.cluster.name
      # TODO#5: Replace with your Kafka cluster name (used to identify and filter metrics in New Relic)
      value: my-cluster
      action: upsert
  transform/remove_broker_id:
    metric_statements:
    - context: datapoint
      statements:
      - delete_key(attributes, "broker.id")
  filter/scrape-overhead:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
        - "^jmx_.*"
        - "^process_.*"
        - "^jvm_buffer_pool_.*"
        - "^jvm_threads_.*"
        - "^jvm_classes_.*"
        - "^jvm_memory_(heap|non_heap)_(committed|init|max|used)_bytes$"
        - "^jvm_compilation_.*"
        - "^jvm_(runtime|info).*"
        - "^jvm_memory_pool_(allocated_bytes_total|committed_bytes|init_bytes|collection_(committed|init|max)_bytes)$"
  filter/include_cluster_metrics:
    metrics:
      include:
        match_type: regexp
        metric_names:
        - "^kafka\\.partition\\.offline$"
        - "^kafka\\.(leader|unclean)\\.election\\.rate$"
        - "^kafka\\.partition\\.non_preferred_leader$"
        - "^kafka\\.broker\\.fenced\\.count$"
        - "^kafka\\.cluster\\.partition\\.count$"
        - "^kafka\\.cluster\\.topic\\.count$"
  filter/exclude_cluster_metrics:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
        - "^kafka\\.partition\\.offline$"
        - "^kafka\\.(leader|unclean)\\.election\\.rate$"
        - "^kafka\\.partition\\.non_preferred_leader$"
        - "^kafka\\.broker\\.fenced\\.count$"
        - "^kafka\\.cluster\\.partition\\.count$"
        - "^kafka\\.cluster\\.topic\\.count$"
  transform/remove_attributes:
    metric_statements:
    - context: metric
      statements:
      - set(description, "") where description != ""
      - set(unit, "") where unit != ""
    - context: resource
      statements:
      - delete_key(attributes, "server.address")
      - delete_key(attributes, "server.port")
      - delete_key(attributes, "service.instance.id")
      - delete_key(attributes, "host.name")
      - delete_key(attributes, "k8s.pod.uid")
      - delete_key(attributes, "url.scheme")
  metricstransform/topic-aggregation:
    transforms:
    - include: kafka.partition.replicas_in_sync
      action: insert
      new_name: kafka.partition.replicas_in_sync.total
      operations:
      - action: aggregate_labels
        label_set: [topic]
        aggregation_type: sum
    - include: kafka.partition.replicas
      action: insert
      new_name: kafka.partition.replicas.total
      operations:
      - action: aggregate_labels
        label_set: [topic]
        aggregation_type: sum
  filter/exclude_partition_replicas_metric:
    metrics:
      exclude:
        match_type: strict
        metric_names:
        - kafka.partition.replicas_in_sync
  # Filter internal Kafka topics as a safety net (topic_match handles the receiver side)
  filter/internal_topics:
    metrics:
      datapoint:
        - 'attributes["topic"] != nil and IsMatch(attributes["topic"], "^__.*")'
  cumulativetodelta:

  groupbyattrs/cluster:
    keys: [kafka.cluster.name]

  metricstransform/cluster_max:
    transforms:
      - include: "kafka\\.partition\\.offline|kafka\\.leader\\.election\\.rate|kafka\\.unclean\\.election\\.rate|kafka\\.partition\\.non_preferred_leader|kafka\\.broker\\.fenced\\.count|kafka\\.cluster\\.partition\\.count|kafka\\.cluster\\.topic\\.count"
        match_type: regexp
        action: update
        operations:
          - action: aggregate_labels
            aggregation_type: max
            label_set: []

service:
  pipelines:
    metrics/broker:
      receivers: [prometheus/kafka-jmx]
      processors:
        - resource/cluster-name
        - filter/scrape-overhead
        - transform/metric-naming
        - transform/remove_attributes
        - filter/exclude_cluster_metrics
        - memory_limiter
        - cumulativetodelta
        - batch/export
      exporters: [otlp/backend]
    metrics/cluster/prometheus:
      receivers: [prometheus/kafka-jmx]
      processors:
        - resource/cluster-name
        - filter/scrape-overhead
        - transform/metric-naming
        - transform/remove_attributes
        - filter/include_cluster_metrics
        - transform/remove_broker_id
        - memory_limiter
        - cumulativetodelta
        - groupbyattrs/cluster
        - metricstransform/cluster_max
        - batch/export
      exporters: [otlp/backend]
    metrics/cluster/kafkametrics:
      receivers: [kafkametrics/cluster]
      processors:
        - resource/cluster-name
        - filter/internal_topics
        - transform/remove_attributes
        - metricstransform/topic-aggregation
        - filter/exclude_partition_replicas_metric
        - memory_limiter
        - cumulativetodelta
        - batch/export
      exporters: [otlp/backend]

Step 3. Create collector-deployment.yaml - OpenTelemetry Collector deployment:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: newrelic
  labels:
    app: otel-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      serviceAccountName: otel-collector
      containers:
      - name: otel-collector
        image: newrelic/nrdot-collector:latest
        command:
        - "/nrdot-collector"
        - "--config=/conf/otel-collector-config.yaml"
        env:
        - name: NEW_RELIC_LICENSE_KEY
          valueFrom:
            secretKeyRef:
              name: newrelic-otlp-secret
              key: NEW_RELIC_LICENSE_KEY
        - name: NEW_RELIC_OTLP_ENDPOINT
          valueFrom:
            secretKeyRef:
              name: newrelic-otlp-secret
              key: NEW_RELIC_OTLP_ENDPOINT
        - name: GOGC
          value: "80"
        resources:
          limits:
            cpu: "1000m"
            memory: "1Gi"
          requests:
            cpu: "200m"
            memory: "512Mi"
        volumeMounts:
        - name: config
          mountPath: /conf
      volumes:
      - name: config
        configMap:
          name: otel-collector-config
          items:
          - key: otel-collector-config.yaml
            path: otel-collector-config.yaml

Paramètres de configuration

Le tableau suivant décrit les principaux paramètres de configuration :

paramètres	Description
`receivers.kafkametrics/cluster.brokers`	Replace with your Kafka bootstrap service
`receivers.prometheus/kafka-jmx.config.scrape_configs[0].kubernetes_sd_configs[0].namespaces.names`	Replace with the namespace where your Kafka cluster is deployed
`receivers.prometheus/kafka-jmx.config.scrape_configs[0].relabel_configs[strimzi_io_name].regex`	Replace with your Strimzi Kafka cluster name followed by `-kafka`
`receivers.prometheus/kafka-jmx.config.scrape_configs[0].relabel_configs[strimzi_io_cluster].regex`	Replace with your Strimzi Kafka cluster name
`processors.resource/cluster-name.attributes[kafka.cluster.name].value`	Replace with your Kafka cluster name (this will be used to identify and filter your metrics in New Relic)

Utilisez le OpenTelemetry Collector de la communauté pour un déploiement indépendant des fournisseurs.

Step 1. Create collector-rbac.yaml - Same as NRDOT option above (RBAC configuration is identical)

Step 2. Create collector-configmap.yaml - Same as NRDOT option above (configuration is identical)

Step 3. Create collector-deployment.yaml - OpenTelemetry Collector deployment (only the image differs):

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: newrelic
  labels:
    app: otel-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      serviceAccountName: otel-collector
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib:latest
        command:
        - "/otelcol-contrib"
        - "--config=/conf/otel-collector-config.yaml"
        env:
        - name: NEW_RELIC_LICENSE_KEY
          valueFrom:
            secretKeyRef:
              name: newrelic-otlp-secret
              key: NEW_RELIC_LICENSE_KEY
        - name: NEW_RELIC_OTLP_ENDPOINT
          valueFrom:
            secretKeyRef:
              name: newrelic-otlp-secret
              key: NEW_RELIC_OTLP_ENDPOINT
        - name: GOGC
          value: "80"
        resources:
          limits:
            cpu: "1000m"
            memory: "1Gi"
          requests:
            cpu: "200m"
            memory: "512Mi"
        volumeMounts:
        - name: config
          mountPath: /conf
      volumes:
      - name: config
        configMap:
          name: otel-collector-config
          items:
          - key: otel-collector-config.yaml
            path: otel-collector-config.yaml

Configuration parameters: Same parameters as the NRDOT option above. See the configuration parameters table for details.

Pour des options de configuration avancées, reportez-vous aux pages de documentation de ces récepteurs :

Documentation du récepteur Prometheus - Options de configuration supplémentaires du récepteur

Documentation du récepteur de métriques Kafka - Configuration de métriques Kafka supplémentaires

Step 3. Deploy the manifests

Appliquez les manifestes Kubernetes pour déployer le Collecteur OpenTelemetry :

bash

$# Create namespace if it doesn't exist
$kubectl create namespace newrelic --dry-run=client -o yaml | kubectl apply -f -
$
$# Apply RBAC configuration
$kubectl apply -f collector-rbac.yaml
$
$# Apply ConfigMap
$kubectl apply -f collector-configmap.yaml
$
$# Apply Deployment
$kubectl apply -f collector-deployment.yaml

Step 4. Verify the deployment:

bash

$# Check pod status
$kubectl get pods -n newrelic -l app=otel-collector
$
$# View logs to verify metrics collection
$kubectl logs -n newrelic -l app=otel-collector --tail=50

Vous devriez voir des logs indiquant un scraping réussi des brokers Kafka sur le port 9404.

(Facultatif) Instrumenter les applications productrices ou consommatrices

Important

Language support: Java applications support out-of-the-box Kafka client instrumentation using the OpenTelemetry Java agent.

To collect application-level telemetry from your Kafka producer and consumer applications, use the OpenTelemetry Java agent.

Instrumentez votre application Kafka

Use an init container to download the OpenTelemetry Java agent at runtime:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-producer-app
spec:
  template:
    spec:
      initContainers:
      - name: download-java-agent
        image: busybox:latest
        command:
        - sh
        - -c
        - |
          wget -O /otel-auto-instrumentation/opentelemetry-javaagent.jar \
          https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
        volumeMounts:
        - name: otel-auto-instrumentation
          mountPath: /otel-auto-instrumentation

      containers:
      - name: app
        image: your-kafka-app:latest
        env:
        - name: JAVA_TOOL_OPTIONS
          value: >-
            -javaagent:/otel-auto-instrumentation/opentelemetry-javaagent.jar
            -Dotel.service.name=order-process-service
            -Dotel.resource.attributes=kafka.cluster.name=my-cluster
            -Dotel.exporter.otlp.endpoint=http://localhost:4317
            -Dotel.exporter.otlp.protocol=grpc
            -Dotel.metrics.exporter=otlp
            -Dotel.traces.exporter=otlp
            -Dotel.logs.exporter=otlp
            -Dotel.instrumentation.kafka.experimental-span-attributes=true
            -Dotel.instrumentation.messaging.experimental.receive-telemetry.enabled=true
            -Dotel.instrumentation.kafka.producer-propagation.enabled=true
            -Dotel.instrumentation.kafka.enabled=true
        volumeMounts:
        - name: otel-auto-instrumentation
          mountPath: /otel-auto-instrumentation

      volumes:
      - name: otel-auto-instrumentation
        emptyDir: {}

Paramètres de configuration

Le tableau suivant décrit les principaux paramètres de configuration :

paramètres	Description
`service.name`	Remplacez `order-process-service` par un nom unique pour votre application producteur ou consommateur
`kafka.cluster.name`	Remplacez `my-cluster` par le même nom de cluster utilisé dans la configuration de votre collecteur.
`otlp.endpoint`	Le point de terminaison `http://localhost:4317` suppose que le collecteur s'exécute en tant que sidecar dans le même pod ou est accessible via localhost

Conseil

La configuration ci-dessus envoie la télémétrie à un Collecteur OpenTelemetry. Si vous devez envoyer des données de télémétrie au collecteur, déployez-le comme décrit à l'étape 3 avec cette configuration :

Ajoutez le récepteur OTLP et les pipelines à la configuration de votre collecteur pour recevoir la télémétrie provenant des applications instrumentées :

Step 1. Add to receivers section:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"

  # ... existing receivers (prometheus/kafka-jmx, kafkametrics/cluster)

Step 2. Add to exporters section:

exporters:
  otlp/backend:
    endpoint: ${NEW_RELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${NEW_RELIC_LICENSE_KEY}

  # ... existing exporters

Step 3. Add to service.pipelines section:

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/backend]

    metrics:
      receivers: [otlp]
      exporters: [otlp/backend]

    logs:
      receivers: [otlp]
      exporters: [otlp/backend]

    # ... existing pipelines (metrics/broker, metrics/cluster/prometheus, metrics/cluster/kafkametrics)

Cela permet au collecteur de recevoir les traces, métriques et logs d'application de vos clients Kafka instrumentés et de les transférer vers New Relic aux côtés des métriques de broker.

The Java agent provides out-of-the-box Kafka instrumentation with zero code changes, capturing:

Latences des requêtes
Métriques de débit
Taux d'erreur
traces distribuées

Pour une configuration avancée, consultez la documentation d'instrumentation Kafka.

(Facultatif) Transmettre les logs du broker Kafka

To collect Kafka broker logs and send them to New Relic, add a filelog receiver to your collector configuration.

Add the following to your values.yaml collector configuration:

Step 1. Add to receivers section:

receivers:
  # ... existing receivers ...

  # File log receiver for Kafka broker logs
  filelog/kafka_broker_0:
    include:
      - /var/log/kafka/server.log
    start_at: end
    multiline:
      line_start_pattern: '^\['
    resource:
      broker.id: "0"
      kafka.cluster.name: ${env:KAFKA_CLUSTER_NAME}

Step 2. Add logs pipeline to service section:

service:
  pipelines:
    # ... existing pipelines ...

    logs/broker:
      receivers: [filelog/kafka_broker_0]
      processors: [memory_limiter, batch/export]
      exporters: [otlp/backend]

Paramètres de configuration

Le tableau suivant décrit les principaux paramètres de configuration :

paramètres	Description
`filelog/kafka_broker_0.include`	Update `/var/log/kafka/server.log` to your actual Kafka log path inside the broker pod. For Strimzi, logs are typically written to stdout — use the Strimzi logging configuration to redirect them to a file, or use a sidecar log collector instead.
`filelog/kafka_broker_0.resource.broker.id`	L'attribut de ressource `broker.id` corrèle les logs avec des métriques et des entités de broker spécifiques.
Multiple broker receivers	Pour plusieurs brokers, créez des récepteurs `filelog` distincts (par ex. `filelog/kafka_broker_1`, `filelog/kafka_broker_2`) avec leurs identifiants de broker respectifs
`filelog/kafka_broker_0.multiline.line_start_pattern`	The `multiline` pattern assumes logs start with `[` — adjust if your log format differs
Log volume	Prenez en compte le volume de logs et les coûts de collecte avant d'activer le transfert de logs
Reference	Pour les options de configuration complètes, consultez la documentation du récepteur filelog

Step 3. Upgrade the Helm release:

bash

$helm upgrade kafka-otel-collector open-telemetry/opentelemetry-collector \
>  --namespace newrelic \
>  --values values.yaml

Vos logs de broker Kafka apparaîtront à deux endroits :

Entités de broker: Accédez à l'entité de broker Kafka dans New Relic pour voir les logs corrélés à ce broker spécifique
Interface utilisateur des logs: Interrogez tous les logs Kafka à l'aide de l'interface utilisateur des logs avec des filtres tels que kafka.cluster.name = 'my-cluster'
Vous pouvez également interroger vos logs avec NRQL :
```
FROM Log SELECT * WHERE kafka.cluster.name = 'my-kafka-cluster'
```

Trouvez vos données

After a few minutes, your Kafka data should appear in New Relic. See Find your data for detailed instructions on exploring your Kafka data across different views in the New Relic UI.

The following table summarizes where each signal type is stored. Replace my-kafka-cluster with your KAFKA_CLUSTER_NAME value in all queries below:

Signal	Type d'événement	Ce qui est inclus
Métriques	`Metric`	Broker, topic, partition, consumer group, and JVM metrics
Logs	`Log`	Logs from producer and consumer applications (via OTel Java agent) and broker logs collected via the optional log forwarding step
Traces	`Span`	Producer and consumer spans, including per-message `publish` and `receive` operations across topics

Métriques

Broker, topic, partition, consumer group, and JVM metrics are stored in the Metric event type. Replace my-kafka-cluster with your KAFKA_CLUSTER_NAME value:

FROM Metric SELECT * WHERE kafka.cluster.name = 'my-kafka-cluster' SINCE 30 minutes ago

Logs

Logs from producer and consumer applications instrumented with the OpenTelemetry Java agent, and broker logs collected via the optional log forwarding step, are stored in the Log event type:

FROM Log SELECT * WHERE kafka.cluster.name = 'my-kafka-cluster' SINCE 30 minutes ago

Traces

If you deploy producer or consumer applications instrumented with the OpenTelemetry Java agent, producer and consumer spans are stored in the Span event type:

FROM Span SELECT * WHERE kafka.cluster.name = 'my-kafka-cluster' SINCE 30 minutes ago

Exemple

A complete working example with Strimzi Kafka custom resources, JMX Exporter configuration, OTel Collector setup, and sample producer/consumer applications is available in the New Relic OpenTelemetry Examples repository.

Dépannage

1. Activer les logs de débogage du collecteur: ajoutez un logging détaillé pour dépanner les problèmes de configuration

Pour les déploiements Helm, mettez à jour votre values.yaml:

config:
  service:
    telemetry:
      logs:
        level: "debug"  # Enable detailed collector internal logs

Pour les déploiements de manifestes, modifiez la ConfigMap de votre collecteur :

bash

$kubectl edit configmap -n newrelic otel-collector-config

Ajoutez la section télémétrie sous service::

service:
  telemetry:
    logs:
      level: "debug"
  pipelines:
    # ... existing pipelines ...

2. Ajoutez l'exporteur de débogage: visualisez les métriques dans les logs du collecteur avant de les envoyer à New Relic

Ajoutez à votre configuration :

exporters:
  debug:
    verbosity: detailed

  otlp/backend:
    endpoint: ${NEW_RELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${NEW_RELIC_LICENSE_KEY}

service:
  pipelines:
    metrics/broker:
      receivers: [prometheus/kafka-jmx]
      processors: [resource/cluster-name, filter/scrape-overhead, transform/metric-naming, transform/remove_attributes, filter/exclude_cluster_metrics, memory_limiter, cumulativetodelta, batch/export]
      exporters: [debug, otlp/backend]  # Add debug exporter

Ensuite, redémarrez le collectteur et vérifiez les logs :

bash

$# Restart collector
$kubectl rollout restart deployment -n newrelic otel-collector
$
$# View logs with metric output
$kubectl logs -n newrelic -l app=otel-collector -f

Important: Supprimez l'exportateur de débogage en production pour éviter le débordement des logs.

1. Vérifier l'état et les événements du pod:

bash

$# Check pod status
$# For Helm:
$kubectl get pods -n newrelic -l app.kubernetes.io/name=opentelemetry-collector
$# For Manifest:
$kubectl get pods -n newrelic -l app=otel-collector
$
$# View detailed pod description
$# For Helm:
$kubectl describe pod -n newrelic -l app.kubernetes.io/name=opentelemetry-collector
$# For Manifest:
$kubectl describe pod -n newrelic -l app=otel-collector
$
$# Check recent logs
$# For Helm:
$kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector --previous --tail=50
$# For Manifest:
$kubectl logs -n newrelic -l app=otel-collector --previous --tail=50

Problèmes courants et solutions:

2. Configuration non valide: Validez la syntaxe YAML de la ConfigMap

bash

$# For manifest deployments, check ConfigMap
$kubectl get configmap -n newrelic otel-collector-config -o yaml
$
$# Validate YAML syntax
$kubectl get configmap -n newrelic otel-collector-config -o yaml | kubectl apply --dry-run=client -f -
$
$# For Helm deployments, check the values
$helm get values <release-name> -n newrelic

3. Permissions RBAC: Vérifiez que le ServiceAccount dispose des liaisons ClusterRole appropriées

bash

$# Check ServiceAccount
$kubectl get serviceaccount -n newrelic otel-collector
$
$# Check ClusterRole and ClusterRoleBinding
$kubectl get clusterrole otel-collector -o yaml
$kubectl get clusterrolebinding otel-collector -o yaml

4. Contraintes de ressources: Vérifiez si le pod a été OOMKilled ou s'il atteint les limites de ressources

bash

$# Check resource usage
$# For Helm:
$kubectl top pods -n newrelic -l app.kubernetes.io/name=opentelemetry-collector
$# For Manifest:
$kubectl top pods -n newrelic -l app=otel-collector
$
$# Check for resource limits
$# For Helm:
$kubectl describe pod -n newrelic -l app.kubernetes.io/name=opentelemetry-collector | grep -A 5 "Limits\|Requests"
$# For Manifest:
$kubectl describe pod -n newrelic -l app=otel-collector | grep -A 5 "Limits\|Requests"

5. Secret introuvable: Vérifiez que le secret des identifiants New Relic existe

bash

$# Check if secret exists
$kubectl get secret -n newrelic newrelic-otlp-secret
$
$# Verify secret has required keys
$kubectl get secret -n newrelic newrelic-otlp-secret -o jsonpath='{.data}' | jq 'keys'

1. Vérifiez que JMX Exporter est activé: Vérifiez que JMX Exporter est configuré dans votre ressource Strimzi Kafka

bash

$# Check Kafka resource configuration for JMX Exporter
$kubectl get kafka -n kafka -o yaml | grep -A 10 jmxPrometheusExporter
$
$# Should show something like:
$# jmxPrometheusExporter:
$#   lowercaseOutputName: true

2. Vérifier le port de JMX Exporter: Vérifiez que l'exportateur est en écoute sur le port 9404

bash

$# Check if port 9404 is exposed on Kafka pods
$kubectl get pods -n kafka -l strimzi.io/name=<cluster-name>-kafka -o yaml | grep -A 3 "containerPort: 9404"
$
$# Test connectivity from collector pod
$# For Helm:
$kubectl exec -it -n newrelic deployment/kafka-monitoring-opentelemetry-collector -- sh -c "curl -s http://<kafka-pod-ip>:9404/metrics" | head -10
$# For Manifest:
$kubectl exec -it -n newrelic deployment/otel-collector -- sh -c "curl -s http://<kafka-pod-ip>:9404/metrics" | head -10

3. Vérifier que le récepteur Prometheus peut collecter des métriques:

bash

$# Check collector logs for Prometheus scraping
$# For Helm:
$kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector | grep -i "prometheus\|scrape"
$# For Manifest:
$kubectl logs -n newrelic -l app=otel-collector | grep -i "prometheus\|scrape"
$
$# Look for successful scrape messages or errors
$# For Helm:
$kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector | grep "prometheus/kafka-jmx"
$# For Manifest:
$kubectl logs -n newrelic -l app=otel-collector | grep "prometheus/kafka-jmx"

4. Vérifiez la découverte de services Kubernetes: Assurez-vous que les labels des pods correspondent à relabel_configs

bash

$# Verify Kafka pod labels
$kubectl get pods -n kafka -l strimzi.io/name=<cluster-name>-kafka --show-labels
$
$# Should include labels like:
$# strimzi.io/cluster=<cluster-name>
$# strimzi.io/name=<cluster-name>-kafka

5. Tester la collecte manuelle: Vérifier que les métriques sont disponibles

bash

$# Get Kafka broker pod IP
$kubectl get pods -n kafka -o wide
$
$# Curl metrics endpoint
$# For Helm:
$kubectl exec -it -n newrelic deployment/kafka-monitoring-opentelemetry-collector -- curl http://<kafka-pod-ip>:9404/metrics
$# For Manifest:
$kubectl exec -it -n newrelic deployment/otel-collector -- curl http://<kafka-pod-ip>:9404/metrics

6. Vérifier les erreurs du récepteur kafkametrics:

bash

$# Look for kafkametrics connection issues
$# For Helm:
$kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector | grep "kafkametrics"
$# For Manifest:
$kubectl logs -n newrelic -l app=otel-collector | grep "kafkametrics"
$
$# Common errors:
$# - Connection refused: Check broker address is correct
$# - Timeout: Check network policies allow access
$# - Authentication failed: Remove TLS configuration if using plaintext

1. Monitor l'utilisation des ressources:

bash

$# Check current memory usage
$# For Helm:
$kubectl top pods -n newrelic -l app.kubernetes.io/name=opentelemetry-collector
$# For Manifest:
$kubectl top pods -n newrelic -l app=otel-collector
$
$# Watch memory usage over time
$# For Helm:
$watch kubectl top pods -n newrelic -l app.kubernetes.io/name=opentelemetry-collector
$# For Manifest:
$watch kubectl top pods -n newrelic -l app=otel-collector

2. Réduire les topics monitorés: limiter la collecte aux topics essentiels uniquement

# In your values.yaml (Helm) or ConfigMap (manifest), add topic filtering:
receivers:
  kafkametrics/cluster:
    brokers:
      - "my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092"
    collection_interval: 30s
    scrapers:
      - brokers
      - topics  # Consider removing if not needed
      - consumers  # Consider removing if not needed
    topic_match: "^(important-topic-1|important-topic-2)$"  # Filter specific topics

3. Réduire la fréquence de collecte: Augmenter les intervalles pour collecter moins souvent

receivers:
  kafkametrics/cluster:
    collection_interval: 60s  # Increase from 30s to 60s

  prometheus/kafka-jmx:
    config:
      scrape_configs:
        - job_name: 'kafka-jmx-metrics'
          scrape_interval: 60s  # Increase from 30s to 60s

4. Optimiser le traitement par lots: Ajuster les paramètres du processeur de lots

processors:
  batch/export:
    timeout: 60s  # Increase from 30s
    send_batch_size: 512  # Reduce from 1024

5. Ajuster le limiteur de mémoire: Régler le processeur de limitation de mémoire

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75  # Reduce from 80
    spike_limit_percentage: 20  # Reduce from 30

6. Mettre à jour les limites de ressources: pour les déploiements Helm, mettez à jour values.yaml

resources:
  requests:
    memory: 512Mi
    cpu: 250m
  limits:
    memory: 1Gi  # Adjust as needed
    cpu: 500m

Pour les déploiements de manifestes, mettez à jour le déploiement directement :

bash

$kubectl patch deployment -n newrelic otel-collector --patch '
$spec:
$  template:
$    spec:
$      containers:
$      - name: otel-collector
$        resources:
$          limits:
$            memory: "1Gi"
$          requests:
$            memory: "512Mi"
$'

7. Redémarrer le collecteur après les modifications:

bash

$# For Helm:
$kubectl rollout restart deployment -n newrelic kafka-monitoring-opentelemetry-collector
$# For Manifest:
$kubectl rollout restart deployment -n newrelic otel-collector

1. Vérifier la connectivité réseau: vérifier que le collecteur peut atteindre les pods du broker Kafka

bash

$# Get Kafka broker pod IPs
$kubectl get pods -n kafka -l strimzi.io/kind=Kafka -o wide
$
$# Test connectivity from collector pod
$# For Helm:
$kubectl exec -it -n newrelic deployment/kafka-monitoring-opentelemetry-collector -- curl -m 5 http://<kafka-pod-ip>:9404/metrics | head -20
$# For Manifest:
$kubectl exec -it -n newrelic deployment/otel-collector -- curl -m 5 http://<kafka-pod-ip>:9404/metrics | head -20

2. Vérifiez que JMX Exporter est en cours d'exécution: Vérifiez que JMX Exporter est activé sur les brokers Kafka

bash

$# Check Kafka pod for JMX Exporter container
$kubectl get pods -n kafka -o yaml | grep -A 5 "jmx-exporter"
$
$# Check if port 9404 is listening
$kubectl exec -n kafka <kafka-pod-name> -- netstat -tlnp | grep :9404
$
$# Or test from within the pod
$kubectl exec -n kafka <kafka-pod-name> -- curl -s localhost:9404/metrics | head

3. Vérifiez que les relabel_configs correspondent aux labels des pods: Assurez-vous que le récepteur Prometheus puisse découvrir les pods Kafka

bash

$# Verify pod labels match the relabel_configs in your configuration
$kubectl get pods -n kafka -l strimzi.io/kind=Kafka --show-labels
$
$# Should show labels like:
$# strimzi.io/cluster=my-cluster
$# strimzi.io/name=my-cluster-kafka

4. Vérifier la configuration de l'espace de noms (namespace): vérifiez que le récepteur Prometheus cherche dans le bon espace de noms

# In your configuration, verify namespace matches where Kafka is deployed
receivers:
  prometheus/kafka-jmx:
    config:
      scrape_configs:
        - job_name: 'kafka-jmx-metrics'
          kubernetes_sd_configs:
            - role: pod
              namespaces:
                names:
                  - kafka  # Must match your Kafka namespace

5. Vérifier les permissions RBAC: Vérifiez que le ClusterRole autorise la découverte des pods

bash

$# Check ClusterRole has pod list/watch permissions
$kubectl get clusterrole otel-collector -o yaml | grep -A 3 "pods"
$
$# Should include:
$# - apiGroups: [""]
$#   resources: ["pods", "nodes"]
$#   verbs: ["get", "list", "watch"]

6. Augmenter le délai d'expiration du scrape: Si le point de terminaison des métriques est lent à répondre

receivers:
  prometheus/kafka-jmx:
    config:
      scrape_configs:
        - job_name: 'kafka-jmx-metrics'
          scrape_interval: 30s
          scrape_timeout: 20s  # Increase from default 10s

7. Vérifier les logs du collecteur pour les détails du scrape:

bash

$# View Prometheus receiver logs
$# For Helm:
$kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector | grep "prometheus/kafka-jmx"
$# For Manifest:
$kubectl logs -n newrelic -l app=otel-collector | grep "prometheus/kafka-jmx"
$
$# Look for discovered targets
$# For Helm:
$kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector | grep -i "target\|scrape"
$# For Manifest:
$kubectl logs -n newrelic -l app=otel-collector | grep -i "target\|scrape"

1. Vérifier les processeurs de filtrage: vérifiez que vous ne filtrez pas de métriques par inadvertance

# Review your filter configurations:
processors:
  filter/scrape-overhead:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
          - "^jmx_.*"  # These are excluded
          - "^process_.*"
          - "^jvm_.*"

2. Les noms de métriques JMX Kafka sont transformés: les traits de soulignement sont remplacés par des points pour respecter les standards OpenTelemetry

La configuration transforme automatiquement les noms de métriques du JMX Exporter (défini à l'étape 1) pour s'aligner sur les conventions sémantiques d'OpenTelemetry. Par exemple :

kafka_topic_io devient kafka.topic.io
kafka_broker_leader_count devient kafka.broker.leader.count

Lorsque vous recherchez des métriques dans New Relic, utilisez les noms transformés avec des points au lieu de traits de soulignement :

# This transformation is applied automatically:
processors:
  transform/metric-naming:
    metric_statements:
      - context: metric
        statements:
          - replace_pattern(name, "_", ".")

3. Activer le logging détaillé: voir exactement quelles métriques sont traitées

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    metrics/broker:
      exporters: [debug, otlp/backend]

4. Interroger New Relic pour obtenir les noms de métriques: vérifiez quelles métriques sont effectivement reçues

FROM Metric SELECT uniques(metricName)
WHERE kafka.cluster.name = 'my-cluster'
SINCE 1 hour ago

La configuration comprend une section Additional metrics pour un monitoring complet. Sa suppression n'affecte pas les fonctionnalités principales de l'interface utilisateur New Relic : la santé du broker, le lag du consommateur, la vue d'ensemble du cluster et les dashboards JVM continuent tous de fonctionner.

1. Supprimez la section Additional metrics de la ConfigMap JMX

Dans votre ConfigMap kafka-jmx-config.yaml, supprimez tout ce qui suit ce commentaire (jusqu'à la fin de la liste rules:) :

# Additional metrics — remove this section to reduce data ingest

  # Request latency: total count, 50th percentile, and average (99p kept above)
  - pattern: 'kafka.network<type=RequestMetrics...'
  ...

Après avoir modifié le ConfigMap, appliquez-le et redémarrez les brokers Kafka pour prendre en compte la modification :

bash

$kubectl apply -f kafka-jmx-config.yaml
$kubectl rollout restart statefulset -n kafka <kafka-cluster-name>-kafka

2. Désactiver les métriques d'offset des consommateurs dans le récepteur kafkametrics

Dans la configuration de votre collecteur, ajoutez à la section metrics du récepteur kafkametrics/cluster :

receivers:
  kafkametrics/cluster:
    # ...existing config...
    metrics:
      kafka.consumer_group.offset:
        enabled: false
      kafka.consumer_group.offset_sum:
        enabled: false

Les métriques de retard du consommateur (kafka.consumer_group.lag, kafka.consumer_group.lag_sum) restent activées. C'est ce que l'interface utilisateur Kafka de New Relic utilise pour les vues de monitoring des consommateurs.

Prochaines étapes

Explorer les métriques Kafka - Consulter la référence complète des métriques
Créer des dashboards personnalisés - Créez des visualisations pour vos données Kafka
Set up alerts - Monitor critical metrics like consumer lag and under-replicated partitions

Cette traduction automatique est fournie pour votre commodité.

Monitorer Kafka sur Kubernetes (Strimzi) avec OpenTelemetry

Architecture .css-21sua1{background:none;border:none;width:0;padding:0;}

Étapes d'installation

Avant de commencer

Configurer le cluster Kafka pour les métriques JMX Kafka

Conseil

Important

Déployer le Collecteur OpenTelemetry

région de l'UE

Conseil

Utilisation de NRDOT Collector (recommandé)

Utiliser le collecteur OpenTelemetry

Région US

région de l'UE

Conseil

Utilisation de NRDOT Collector (recommandé)

Utiliser le collecteur OpenTelemetry

(Facultatif) Instrumenter les applications productrices ou consommatrices

Important

Instrumentez votre application Kafka

Conseil

Configuration du collecteur pour la télémétrie applicative

(Facultatif) Transmettre les logs du broker Kafka

Configurer la collecte des logs

Trouvez vos logs dans New Relic

Trouvez vos données

Métriques

Logs

Traces

Exemple

Dépannage

Activer la logging de débogage

Le pod du collecteur ne démarre pas

Aucune métrique Kafka collectée

Utilisation élevée de la mémoire

Erreurs de collecte Prometheus

Certaines métriques sont manquantes

Réduire l'ingestion de données

Prochaines étapes

Architecture