Monitor Kubernetes-hosted RabbitMQ with OpenTelemetry

Monitor RabbitMQ running in Kubernetes clusters using the OpenTelemetry Collector with automatic pod discovery. This guide walks you through deploying the collector via Helm with dynamic broker detection powered by the Kubernetes observer.

Dica

Running RabbitMQ on Linux hosts? See the self-hosted installation guide for virtual machine or bare metal deployments.

How the integration works

This Kubernetes-native deployment provides automatic discovery and monitoring:

Key components:

Kubernetes observer: Continuously watches for RabbitMQ pods based on label selectors
Receiver creator: Dynamically creates RabbitMQ receivers for each discovered pod
Resource attribution: Automatically enriches metrics with Kubernetes metadata:
- k8s.cluster.name - Your cluster identifier
- k8s.namespace.name - Pod namespace
- k8s.pod.name - Individual pod name
- rabbitmq.deployment.name - Derived from pod name

Deployment model: The collector runs as a Kubernetes deployment (single replica) with RBAC permissions to list and watch pods across namespaces. When new RabbitMQ pods are created or existing ones are removed, the observer automatically updates the collector's configuration.

Importante

The collector requires get, list, and watch permissions on pods to discover RabbitMQ instances automatically.

Installation steps

Follow these steps to deploy the OpenTelemetry Collector with RabbitMQ monitoring.

Before you begin

Ensure your environment meets these requirements:

Verify the management endpoint

Ensure RabbitMQ pods have the management plugin enabled:

bash

$kubectl exec deploy/<your-rabbitmq-deployment> -n <rabbitmq-namespace> -- curl -I -u admin:password http://localhost:15672/api/overview

You should see HTTP/1.1 200 OK in the response.

Create Helm values file

Create a file named otel-collector-values.yaml. This configuration uses the receiver_creator with k8s_observer to automatically discover RabbitMQ pods.

Update these placeholders:

my-rabbitmq-cluster: Your Kubernetes cluster name
rabbitmq: The value of the app label for your RabbitMQ pods
admin/password: Your RabbitMQ credentials
15672: The management API port

opentelemetry-collector:
  mode: deployment

  image:
    repository: otel/opentelemetry-collector-contrib
    pullPolicy: IfNotPresent

  command:
    name: otelcol-contrib

  resources:
    limits:
      cpu: 500m
      memory: 300Mi
    requests:
      cpu: 100m
      memory: 100Mi

  extraEnvs:
    - name: NEWRELIC_LICENSE_KEY
      valueFrom:
        secretKeyRef:
          name: newrelic-licenses
          key: NEWRELIC_LICENSE_KEY
    - name: NEWRELIC_OTLP_ENDPOINT
      valueFrom:
        secretKeyRef:
          name: newrelic-licenses
          key: NEWRELIC_OTLP_ENDPOINT
    - name: K8S_NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    - name: K8S_CLUSTER_NAME
      value: my-rabbitmq-cluster
    - name: RABBITMQ_USERNAME
      value: admin
    - name: RABBITMQ_PASSWORD
      value: password

  clusterRole:
    create: true
    rules:
      - apiGroups: [""]
        resources: ["pods", "nodes", "nodes/stats", "nodes/proxy"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["apps"]
        resources: ["replicasets", "deployments", "statefulsets"]
        verbs: ["get", "list", "watch"]

  clusterRoleBinding:
    name: ""

  config:
    extensions:
      health_check:
        endpoint: 0.0.0.0:13133

      k8s_observer:
        auth_type: serviceAccount
        observe_pods: true
        observe_nodes: false

    receivers:
      receiver_creator/rabbitmq:
        watch_observers: [k8s_observer]
        receivers:
          rabbitmq:
            # Discover pods with label "app=rabbitmq"
            # Adjust the label selector to match your RabbitMQ pods
            rule: type == "pod" && labels["app"] == "rabbitmq"
            config:
              endpoint: 'http://`endpoint`:15672'
              username: ${env:RABBITMQ_USERNAME}
              password: ${env:RABBITMQ_PASSWORD}
              collection_interval: 30s
              metrics:
                # Queue Metrics (essential for message flow and backlog)
                rabbitmq.consumer.count:
                  enabled: true
                rabbitmq.message.delivered:
                  enabled: true
                rabbitmq.message.published:
                  enabled: true
                rabbitmq.message.acknowledged:
                  enabled: true
                rabbitmq.message.dropped:
                  enabled: true
                rabbitmq.message.current:
                  enabled: true # Crucial for monitoring queue backlog, includes 'ready' and 'unacknowledged' states

                # Node Health Metrics (critical for server resource monitoring)
                rabbitmq.node.disk_free:
                  enabled: true
                rabbitmq.node.disk_free_limit:
                  enabled: true
                rabbitmq.node.disk_free_alarm:
                  enabled: true
                rabbitmq.node.mem_used:
                  enabled: true
                rabbitmq.node.mem_limit:
                  enabled: true
                rabbitmq.node.mem_alarm:
                  enabled: true
                rabbitmq.node.mem_used_details.rate:
                  enabled: true
                rabbitmq.node.fd_used:
                  enabled: true
                rabbitmq.node.fd_total:
                  enabled: true
                rabbitmq.node.sockets_used:
                  enabled: true
                rabbitmq.node.sockets_total:
                  enabled: true
                rabbitmq.node.proc_used:
                  enabled: true
                rabbitmq.node.proc_total:
                  enabled: true
                rabbitmq.node.uptime:
                  enabled: true
                rabbitmq.node.run_queue:
                  enabled: true
                rabbitmq.node.processors:
                  enabled: true
                rabbitmq.node.context_switches_details.rate:
                  enabled: true
                rabbitmq.node.gc_num_details.rate:
                  enabled: true
                rabbitmq.node.gc_bytes_reclaimed_details.rate:
                  enabled: true

                # I/O Metrics (important for understanding disk and network activity)
                rabbitmq.node.io_read_count_details.rate:
                  enabled: true
                rabbitmq.node.io_read_bytes_details.rate:
                  enabled: true
                rabbitmq.node.io_read_avg_time_details.rate:
                  enabled: true
                rabbitmq.node.io_write_count_details.rate:
                  enabled: true
                rabbitmq.node.io_write_bytes_details.rate:
                  enabled: true
                rabbitmq.node.io_write_avg_time_details.rate:
                  enabled: true
                rabbitmq.node.io_sync_count_details.rate:
                  enabled: true
                rabbitmq.node.io_sync_avg_time_details.rate:
                  enabled: true
                rabbitmq.node.io_seek_count_details.rate:
                  enabled: true
                rabbitmq.node.io_seek_avg_time_details.rate:
                  enabled: true
                rabbitmq.node.io_reopen_count_details.rate:
                  enabled: true

                # Mnesia and Store Metrics (for internal database and message storage)
                rabbitmq.node.mnesia_ram_tx_count_details.rate:
                  enabled: true
                rabbitmq.node.mnesia_disk_tx_count_details.rate:
                  enabled: true
                rabbitmq.node.msg_store_read_count_details.rate:
                  enabled: true
                rabbitmq.node.msg_store_write_count_details.rate:
                  enabled: true
                rabbitmq.node.queue_index_write_count_details.rate:
                  enabled: true
                rabbitmq.node.queue_index_read_count_details.rate:
                  enabled: true

                # Connection/Channel/Queue Lifecycle Metrics
                rabbitmq.node.connection_created_details.rate:
                  enabled: true
                rabbitmq.node.connection_closed_details.rate:
                  enabled: true
                rabbitmq.node.channel_created_details.rate:
                  enabled: true
                rabbitmq.node.channel_closed_details.rate:
                  enabled: true
                rabbitmq.node.queue_declared_details.rate:
                  enabled: true
                rabbitmq.node.queue_created_details.rate:
                  enabled: true
                rabbitmq.node.queue_deleted_details.rate:
                  enabled: true

              resource_attributes:
                rabbitmq.server.endpoint: 'http://`endpoint`:15672'
                rabbitmq.port: '15672'

    processors:
      batch:
        send_batch_size: 1024
        timeout: 30s

      resource/cluster:
        attributes:
          - key: k8s.cluster.name
            value: ${env:K8S_CLUSTER_NAME}
            action: upsert

      transform/rabbitmq:
        metric_statements:
          - context: resource
            statements:
              # Create a display name combining Kubernetes metadata
              - set(attributes["rabbitmq.display.name"], Concat([
                  "server",
                  "k8s",
                  attributes["k8s.cluster.name"],
                  attributes["k8s.namespace.name"],
                  "pod",
                  attributes["k8s.pod.name"],
                  "rabbitmq",
                  attributes["rabbitmq.port"]
                ], ":"))
              # Use pod name as deployment name
              - set(attributes["rabbitmq.deployment.name"], attributes["k8s.pod.name"])

    exporters:
      otlphttp:
        endpoint: "${NEWRELIC_OTLP_ENDPOINT}"
        headers:
          api-key: "${NEWRELIC_LICENSE_KEY}"
        compression: gzip

    service:
      extensions: [health_check, k8s_observer]
      pipelines:
        metrics/rabbitmq:
          receivers: [receiver_creator/rabbitmq]
          processors: [batch, resource/cluster, transform/rabbitmq]
          exporters: [otlphttp]

Dica

Label selector customization: If your RabbitMQ pods use different labels, update the rule line. For example:

StatefulSet pods: type == "pod" && labels["app.kubernetes.io/name"] == "rabbitmq"
Operator deployments: type == "pod" && labels["app.kubernetes.io/component"] == "rabbitmq"

Run kubectl get pods --show-labels -n <namespace> to view your pod labels.

Create Kubernetes secret

Store your New Relic credentials in a Kubernetes secret.

Select the region that matches your New Relic account (check your browser URL when logged into New Relic):

bash

$kubectl create secret generic newrelic-licenses \
>  --from-literal=NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY \
>  --from-literal=NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318 \
>  --namespace newrelic

Replace YOUR_LICENSE_KEY with your New Relic license key.

bash

$kubectl create secret generic newrelic-licenses \
>  --from-literal=NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY \
>  --from-literal=NEWRELIC_OTLP_ENDPOINT=https://otlp.eu01.nr-data.net:4318 \
>  --namespace newrelic

Replace YOUR_LICENSE_KEY with your New Relic license key.

Deploy with Helm

Add the OpenTelemetry Helm repository:

bash

$helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
$helm repo update

Install or upgrade the collector:

bash

$helm upgrade --install rabbitmq-otel-collector \
>  open-telemetry/opentelemetry-collector \
>  --namespace newrelic \
>  --create-namespace \
>  --values otel-collector-values.yaml

This creates:

A deployment named rabbitmq-otel-collector in the newrelic namespace
A Serviceaccount with RBAC permissions to watch pods
ConfigMaps containing the collector configuration

Verify the deployment and data flow

Check that the collector pod is running:

bash

$kubectl get pods -n newrelic -l app.kubernetes.io/name=opentelemetry-collector

Expected output:

NAME                                 READY   STATUS    RESTARTS   AGE
rabbitmq-otel-collector-6d8c5c5d8d-abc12   1/1     Running   0          2m

View collector logs to verify pod discovery:

bash

$kubectl logs deploy/rabbitmq-otel-collector -n newrelic --tail=50

Look for messages indicating successful pod discovery and metric collection:

INFO    k8sobserver/extension.go:150    Discovered pod {"kind": "pod", "name": "rabbitmq-0", "namespace": "default"}
INFO    RabbitmqReceiver                Successfully scraped rabbitmq metrics from pod rabbitmq-0

Verify your data in New Relic:

Wait 2-3 minutes for data to appear, then run this query in the query builder:

SELECT count(*)
FROM Metric
WHERE metricName LIKE 'rabbitmq.%'
  AND instrumentation.provider = 'opentelemetry'
  AND k8s.cluster.name = 'my-rabbitmq-cluster'
FACET k8s.pod.name, metricName
SINCE 10 minutes ago

You should see metrics from each RabbitMQ pod with Kubernetes attributes:

k8s.cluster.name - Your cluster identifier
k8s.namespace.name - Pod namespace
k8s.pod.name - Individual pod name
rabbitmq.deployment.name - Derived from pod name

Dica

If you don't see data after 5 minutes, check the troubleshooting section below.

Troubleshooting

If metrics aren't appearing in New Relic after deployment:

1. Verify RabbitMQ management plugin is enabled:

bash

$kubectl exec <rabbitmq-pod> -n <namespace> -- rabbitmq-plugins list | grep management

You should see [E*] rabbitmq_management indicating it's enabled.

2. Check collector pod logs:

bash

$kubectl logs deploy/rabbitmq-otel-collector -n newrelic --tail=100

Look for:

Pod discovery messages
Authentication errors (incorrect credentials)
Connection errors (network issues)
Parsing errors (configuration problems)

3. Verify RabbitMQ service is accessible:

bash

$kubectl get svc -n <rabbitmq-namespace>

Ensure there's a service exposing port 15672.

4. Test management API from within the cluster:

bash

$kubectl run test-pod --image=curlimages/curl:latest --rm -it --restart=Never -- \
>  curl -I -u admin:password http://<rabbitmq-service>.<namespace>.svc.cluster.local:15672/api/overview

Should return HTTP/1.1 200 OK.

5. Check RBAC permissions:

bash

$kubectl auth can-i list pods --as=system:serviceaccount:newrelic:rabbitmq-otel-collector -n <rabbitmq-namespace>

Should return yes. If not, verify the ClusterRole and ClusterRoleBinding.

If the collector isn't discovering RabbitMQ pods automatically:

1. Verify label selector matches your pods:

bash

$kubectl get pods --show-labels -n <rabbitmq-namespace>

Compare the labels with the rule in your Helm values file.

2. Check k8s_observer logs:

bash

$kubectl logs deploy/rabbitmq-otel-collector -n newrelic | grep k8sobserver

Look for messages about discovered pods.

3. Test the label selector:

bash

$kubectl get pods -l app=rabbitmq --all-namespaces

This should list your RabbitMQ pods. If empty, your label selector is incorrect.

4. Verify ClusterRole permissions:

bash

$kubectl describe clusterrole rabbitmq-otel-collector

Ensure it has get, list, and watch permissions on pods.

Common label patterns:

Standard deployments: app=rabbitmq
StatefulSets: app.kubernetes.io/name=rabbitmq
Operators: app.kubernetes.io/component=rabbitmq

If you see connection errors in the collector logs:

Check service endpoints:

bash

$kubectl get endpoints -n <rabbitmq-namespace>

Verify that endpoints exist for your RabbitMQ service.

Verify network policies: Ensure network policies allow traffic from the newrelic namespace to your RabbitMQ namespace.

Test connectivity:

bash

$kubectl run test-pod --image=curlimages/curl:latest --rm -it --restart=Never -n newrelic -- \
>  curl http://<rabbitmq-service>.<namespace>.svc.cluster.local:15672/api/overview

Check DNS resolution:

bash

$kubectl run test-pod --image=busybox:latest --rm -it --restart=Never -n newrelic -- \
>  nslookup <rabbitmq-service>.<namespace>.svc.cluster.local

If you see high cardinality warnings in New Relic:

Filter to specific queues: If monitoring many queues, consider filtering to only critical ones.

Increase collection interval:

receivers:
  receiver_creator/rabbitmq:
    receivers:
      rabbitmq:
        config:
          collection_interval: 60s # Increased from 30s

Use metric aggregation: Configure the batch processor to aggregate metrics before export.

Disable less critical metrics: Comment out metrics you don't need in the receiver configuration.

Additional optimization tips:

Use more specific label selectors to monitor only production pods
Filter specific queues by name instead of monitoring all queues
Increase the collection_interval to reduce data points
Consider using metric aggregation in the collector

If the collector pod is crashing:

Check resource limits:

bash

$kubectl describe pod -n newrelic -l app.kubernetes.io/name=opentelemetry-collector

Look for OOMKilled status. If so, increase memory limits:

resources:
  limits:
    memory: 512Mi # Increased from 300Mi

Review crash logs:

bash

$kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector --previous

Check configuration syntax: Ensure your YAML configuration is valid and properly indented.

If you see 401 Unauthorized errors:

Verify credentials in the Helm values match your RabbitMQ setup:

bash

$kubectl exec <rabbitmq-pod> -n <namespace> -- rabbitmqctl list_users

Check that the RabbitMQ user has administrator privileges and ensure credentials don't contain special characters that need escaping.

What's next?

Now that you have RabbitMQ monitoring set up, you can enhance your observability:

Explore your data:

Explore RabbitMQ metrics - Complete list of metrics with NRQL query examples

Enhance monitoring:

Create alerts - Set up alerts for queue depths and pod health
Build dashboards - Create custom dashboards to visualize your RabbitMQ metrics

Monitor Kubernetes-hosted RabbitMQ with OpenTelemetry

Dica

How the integration works

Importante

Installation steps

Before you begin

Kubernetes cluster requirements

RabbitMQ requirements

New Relic requirements

Verify the management endpoint

Create Helm values file

Dica

Create Kubernetes secret

Deploy with Helm

Verify the deployment and data flow

Dica

Troubleshooting

No data appearing in New Relic

Pod discovery not working

Connection errors

High cardinality warnings

Collector pod crashing or restarting

Authentication failures

What's next?

Monitor Kubernetes-hosted RabbitMQ with OpenTelemetry

Dica

How the integration works .css-21sua1{background:none;border:none;width:0;padding:0;}

Importante

Installation steps

Before you begin

RabbitMQ requirements

New Relic requirements

Verify the management endpoint

Create Helm values file

Dica

Create Kubernetes secret

Deploy with Helm

Verify the deployment and data flow

Dica

Troubleshooting

No data appearing in New Relic

Pod discovery not working

Connection errors

High cardinality warnings

Collector pod crashing or restarting

Authentication failures

What's next?

How the integration works