Monitor self-hosted RabbitMQ with OpenTelemetry

Monitor RabbitMQ running on Linux hosts by installing and configuring the OpenTelemetry Collector. This comprehensive guide walks you through installation, configuration, and verification for Debian, Ubuntu, RHEL, and CentOS systems.

Installation steps

Follow these steps in order to set up monitoring for your self-hosted RabbitMQ instance.

Before you begin

Make sure your environment meets these requirements:

Enable the RabbitMQ management plugin

The management plugin exposes the API that the OpenTelemetry Collector scrapes:

bash

$sudo rabbitmq-plugins enable rabbitmq_management

Verify it's working:

bash

$curl -I -u admin:password http://localhost:15672/api/overview

You should see HTTP/1.1 200 OK in the response.

Install the OpenTelemetry Collector

For all Linux distribution:

Download and install the OpenTelemetry Collector Contrib binary for your host operating system from the OpenTelemetry Collector releases.

Configure the collector

Create or replace /etc/otelcol-contrib/config.yaml:

receivers:
  rabbitmq:
    endpoint: http://localhost:15672
    username: admin
    password: password
    collection_interval: 30s
    metrics:
      # Queue Metrics (essential for message flow and backlog)
      rabbitmq.consumer.count:
        enabled: true
      rabbitmq.message.delivered:
        enabled: true
      rabbitmq.message.published:
        enabled: true
      rabbitmq.message.acknowledged:
        enabled: true
      rabbitmq.message.dropped:
        enabled: true
      rabbitmq.message.current:
        enabled: true # Crucial for monitoring queue backlog, includes 'ready' and 'unacknowledged' states

      # Node Health Metrics (critical for server resource monitoring)
      rabbitmq.node.disk_free:
        enabled: true
      rabbitmq.node.disk_free_limit:
        enabled: true
      rabbitmq.node.disk_free_alarm:
        enabled: true
      rabbitmq.node.mem_used:
        enabled: true
      rabbitmq.node.mem_limit:
        enabled: true
      rabbitmq.node.mem_alarm:
        enabled: true
      rabbitmq.node.mem_used_details.rate:
        enabled: true
      rabbitmq.node.fd_used:
        enabled: true
      rabbitmq.node.fd_total:
        enabled: true
      rabbitmq.node.sockets_used:
        enabled: true
      rabbitmq.node.sockets_total:
        enabled: true
      rabbitmq.node.proc_used:
        enabled: true
      rabbitmq.node.proc_total:
        enabled: true
      rabbitmq.node.uptime:
        enabled: true
      rabbitmq.node.run_queue:
        enabled: true
      rabbitmq.node.processors:
        enabled: true
      rabbitmq.node.context_switches_details.rate:
        enabled: true
      rabbitmq.node.gc_num_details.rate:
        enabled: true
      rabbitmq.node.gc_bytes_reclaimed_details.rate:
        enabled: true

      # I/O Metrics (important for understanding disk and network activity)
      rabbitmq.node.io_read_count_details.rate:
        enabled: true
      rabbitmq.node.io_read_bytes_details.rate:
        enabled: true
      rabbitmq.node.io_read_avg_time_details.rate:
        enabled: true
      rabbitmq.node.io_write_count_details.rate:
        enabled: true
      rabbitmq.node.io_write_bytes_details.rate:
        enabled: true
      rabbitmq.node.io_write_avg_time_details.rate:
        enabled: true
      rabbitmq.node.io_sync_count_details.rate:
        enabled: true
      rabbitmq.node.io_sync_avg_time_details.rate:
        enabled: true
      rabbitmq.node.io_seek_count_details.rate:
        enabled: true
      rabbitmq.node.io_seek_avg_time_details.rate:
        enabled: true
      rabbitmq.node.io_reopen_count_details.rate:
        enabled: true

      # Mnesia and Store Metrics (for internal database and message storage)
      rabbitmq.node.mnesia_ram_tx_count_details.rate:
        enabled: true
      rabbitmq.node.mnesia_disk_tx_count_details.rate:
        enabled: true
      rabbitmq.node.msg_store_read_count_details.rate:
        enabled: true
      rabbitmq.node.msg_store_write_count_details.rate:
        enabled: true
      rabbitmq.node.queue_index_write_count_details.rate:
        enabled: true
      rabbitmq.node.queue_index_read_count_details.rate:
        enabled: true

      # Connection/Channel/Queue Lifecycle Metrics
      rabbitmq.node.connection_created_details.rate:
        enabled: true
      rabbitmq.node.connection_closed_details.rate:
        enabled: true
      rabbitmq.node.channel_created_details.rate:
        enabled: true
      rabbitmq.node.channel_closed_details.rate:
        enabled: true
      rabbitmq.node.queue_declared_details.rate:
        enabled: true
      rabbitmq.node.queue_created_details.rate:
        enabled: true
      rabbitmq.node.queue_deleted_details.rate:
        enabled: true

# processors: Process data before exporting.
processors:
  resourcedetection:
    detectors: [system]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true
  resource:
    attributes:
      - key: instrumentation.provider
        value: opentelemetry
        action: upsert
      - key: rabbitmq.deployment.name
        value: my-rabbitmq-server  # Replace with your server name
        action: upsert
  batch:
    send_batch_size: 1024
    timeout: 30s

exporters:
  otlphttp/newrelic:
    endpoint: ${env:NEWRELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEWRELIC_LICENSE_KEY}
    compression: gzip

service:
  pipelines:
    metrics:
      receivers: [rabbitmq]
      processors: [resourcedetection, resource, batch]
      exporters: [otlphttp/newrelic]

Update these values in the configuration:

endpoint: Your RabbitMQ management API URL (default: http://localhost:15672)
username and password: Your RabbitMQ credentials
rabbitmq.deployment.name: A unique name for this RabbitMQ instance
collection_interval: How often to scrape metrics (default: 30 seconds)

Set environment variables

Create a systemd override file for environment variables:

bash

$sudo mkdir -p /etc/systemd/system/otelcol-contrib.service.d

Choose your New Relic region:

bash

$cat <<EOF | sudo tee /etc/systemd/system/otelcol-contrib.service.d/environment.conf
$[Service]
$Environment="NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318"
$Environment="NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY"
$EOF

Replace YOUR_LICENSE_KEY with your New Relic license key.

bash

$cat <<EOF | sudo tee /etc/systemd/system/otelcol-contrib.service.d/environment.conf
$[Service]
$Environment="NEWRELIC_OTLP_ENDPOINT=https://otlp.eu01.nr-data.net:4318"
$Environment="NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY"
$EOF

Replace YOUR_LICENSE_KEY with your New Relic license key.

Reload systemd and restart the collector:

bash

$sudo systemctl daemon-reload
$sudo systemctl restart otelcol-contrib

Verify your data in New Relic

Check the service status:

bash

$sudo systemctl status otelcol-contrib

You should see active (running) in the output.

View recent logs:

bash

$sudo journalctl -u otelcol-contrib -n 50 --no-pager

Look for messages indicating successful metric collection:

INFO    RabbitmqReceiver        Successfully scraped rabbitmq metrics

Verify metrics in New Relic:

Wait 2-3 minutes for data to appear, then run this query in the query builder:

SELECT count(*)
FROM Metric
WHERE metricName LIKE 'rabbitmq.%'
  AND instrumentation.provider = 'opentelemetry'
  AND rabbitmq.deployment.name = 'my-rabbitmq-server'
FACET metricName
SINCE 10 minutes ago

You should see metric names like:

rabbitmq.queue.count - Number of queues
rabbitmq.queue.message.count - Total messages across queues
rabbitmq.connection.count - Active connections
rabbitmq.consumer.count - Active consumers

Conseil

If you don't see data after 5 minutes, check the troubleshooting section below.

(Optional) Forward RabbitMQ logs to New Relic

In addition to metrics, you can forward RabbitMQ application logs to New Relic for comprehensive observability. This helps correlate issues across metrics and logs.

Important

Log forwarding requires additional configuration and system permissions. Ensure you have the necessary access before proceeding.

Update /etc/otelcol-contrib/config.yaml to add the filelog receiver:

receivers:
  rabbitmq:
    # ... existing rabbitmq config ...

  # Add filelog receiver for RabbitMQ logs
  filelog/rabbitmq:
    include:
      - /var/log/rabbitmq/*.log
      - /var/log/rabbitmq/**/*.log
    include_file_path: true
    include_file_name: false
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}) \[(?P<level>\w+)\] <(?P<pid>[^>]+)> (?P<message>.*)$'
        timestamp:
          parse_from: attributes.timestamp
          layout: '%Y-%m-%d %H:%M:%S.%L'
        severity:
          parse_from: attributes.severity

processors:
  # ... existing processors ...

  transform/rabbitmq_logs:
    log_statements:
      - context: resource
        statements:
          - set(attributes["rabbitmq.display.name"], Concat(["server", attributes["rabbitmq.deployment.name"]], ":"))

service:
  pipelines:
    metrics:
      receivers: [rabbitmq]
      processors: [resourcedetection, resource, batch]
      exporters: [otlphttp/newrelic]

    logs/rabbitmq:
      receivers: [filelog]
      processors: [batch, resource, transform/rabbitmq_logs]
      exporters: [otlphttp/newrelic]

Grant the collector permission to read logs. Choose one method:

Add to rabbitmq group:

bash

$sudo usermod -a -G rabbitmq otelcol-contrib

Or set file permissions:

bash

$sudo chmod 644 /var/log/rabbitmq/*.log

Restart the collector:

bash

$sudo systemctl restart otelcol-contrib

Verify logs are being collected in New Relic:

SELECT count(*)
FROM Log
WHERE service.name = 'rabbitmq'
  AND rabbitmq.deployment.name = 'my-rabbitmq-server'
SINCE 10 minutes ago

Find your data

After a few minutes, your RabbitMQ metrics should appear in New Relic. See Find and query your data for detailed instructions on exploring your RabbitMQ metrics across different views in the New Relic UI.

You can also query your data with NRQL:

FROM Metric SELECT * WHERE rabbitmq.deployment.name = 'my-rabbitmq-server'

Troubleshooting

If you don't see metrics in New Relic after installation:

1. Verify the management plugin is enabled:

bash

$sudo rabbitmq-plugins list | grep management

You should see [E*] rabbitmq_management indicating it's enabled and running.

2. Test the management API:

bash

$curl -u admin:password http://localhost:15672/api/overview

This should return JSON data about your RabbitMQ instance. If not, the management plugin isn't working correctly.

3. Check collector logs for errors:

bash

$sudo journalctl -u otelcol-contrib -f

Look for:

Authentication errors (incorrect username/password)
Connection timeouts (network issues)
Parsing errors (configuration syntax problems)

4. Verify the collector service is running:

bash

$sudo systemctl status otelcol-contrib

Should show active (running). If not, check logs for why it failed to start.

5. Check your license key:

bash

$sudo cat /etc/systemd/system/otelcol-contrib.service.d/environment.conf

Ensure NEWRELIC_LICENSE_KEY is set correctly and has no extra spaces or quotes.

If you see connection refused in the collector logs:

Verify RabbitMQ is listening on port 15672:

bash

$sudo netstat -tlnp | grep 15672

You should see RabbitMQ listening. If not, the management plugin isn't enabled or isn't binding to the expected port.

Check firewall rules:

bash

$sudo iptables -L -n | grep 15672

Ensure no firewall rules are blocking port 15672.

Verify endpoint configuration:
Ensure the endpoint in /etc/otelcol-contrib/config.yaml matches your RabbitMQ setup (hostname, port).

If you see 401 Unauthorized errors:

Verify credentials:
Check that the username and password in /etc/otelcol-contrib/config.yaml match your RabbitMQ user.

Ensure administrator privileges:

bash

$sudo rabbitmqctl list_users

Your user should have the [administrator] tag. If not, grant admin privileges:

bash

$sudo rabbitmqctl set_user_tags your-username administrator

If the collector consumes excessive resources:

Reduce batch size:
In the batch processor, reduce send_batch_size from 512 to 256 or lower.

Increase collection interval:
Change collection_interval from 30s to 60s or longer to scrape less frequently.

Disable unnecessary metrics:
Comment out metrics you don't need in the rabbitmq receiver configuration.

Example:

receivers:
  rabbitmq:
    collection_interval: 60s  # Increased
    metrics:
      rabbitmq.queue.message.count:
        enabled: true
      # Disable less critical metrics
      # rabbitmq.queue.count:
      #   enabled: false

If you see permission errors when collecting logs:

Test file access:

bash

$sudo -u otelcol-contrib cat /var/log/rabbitmq/rabbit@$(hostname).log

If this fails, the collector user can't read the log files.

Check file permissions:

bash

$ls -la /var/log/rabbitmq/

Fix permissions:
Add the collector user to the rabbitmq group:

bash

$sudo usermod -a -G rabbitmq otelcol-contrib
$sudo systemctl restart otelcol-contrib

Next steps

Now that you have RabbitMQ monitoring set up, you can enhance your observability:

Explore your data:

Find and query your data - Navigate New Relic UI and write NRQL queries
Explore RabbitMQ metrics - Complete metrics reference with alerting recommendations

Enhance monitoring:

Create alerts - Set up alerts for queue depths and message backlogs
Build dashboards - Create custom dashboards to visualize your RabbitMQ metrics

Monitor self-hosted RabbitMQ with OpenTelemetry

Installation steps

Before you begin

RabbitMQ requirements

System requirements

New Relic requirements

Software requirements

Enable the RabbitMQ management plugin

Install the OpenTelemetry Collector

Configure the collector

Set environment variables

Verify your data in New Relic

Conseil

(Optional) Forward RabbitMQ logs to New Relic

Important

Find your data

Troubleshooting

No data appearing in New Relic

Connection refused errors

Authentication failures (401 Unauthorized)

High memory or CPU usage

Permission denied for logs

Next steps

Monitor self-hosted RabbitMQ with OpenTelemetry

Installation steps .css-21sua1{background:none;border:none;width:0;padding:0;}

Before you begin

System requirements

New Relic requirements

Software requirements

Enable the RabbitMQ management plugin

Install the OpenTelemetry Collector

Configure the collector

Set environment variables

Verify your data in New Relic

Conseil

(Optional) Forward RabbitMQ logs to New Relic

Important

Find your data

Troubleshooting

No data appearing in New Relic

Connection refused errors

Authentication failures (401 Unauthorized)

High memory or CPU usage

Permission denied for logs

Next steps

Installation steps