Monitor self-hosted NGINX with OpenTelemetry

Add this configuration to your NGINX config file (/etc/nginx/nginx.conf):

server {
    listen 8080;
    server_name localhost;

    location /nginx_status {
        stub_status on;
        access_log off;        # Don't log monitoring requests
        allow 127.0.0.1;       # Only allow localhost access
        allow ::1;             # Allow IPv6 localhost
        deny all;              # Deny all other access
    }
}

1. Test your NGINX configuration:

   bash

   コピー

   $
   sudo nginx -t

2. If the test passes, reload NGINX:

   bash

   コピー

   $
   sudo nginx -s reload

Test the endpoint:

$
curl http://127.0.0.1:8080/nginx_status

Expected output:

Active connections: 1
server accepts handled requests
 16 16 16
Reading: 0 Writing: 1 Waiting: 0

Test the HTTP response:

$
curl -I http://127.0.0.1:8080/nginx_status

Expected output: Should start with HTTP/1.1 200 OK

The OpenTelemetry Collector uses three main components for NGINX monitoring:

• Receivers - Connect to your NGINX stub status endpoint to collect metrics
• Processors - Add server identification and batch metrics for efficient transmission
• Exporters - Send the processed metrics to your New Relic account via OTLP HTTP

Before editing the configuration, you'll need two key pieces of information:

1. Your stub status URL - From Step 1 (default: http://127.0.0.1:8080/nginx_status)

2. A unique deployment name - Choose a name that identifies this specific NGINX server (e.g., production-web-01, staging-api, prod-lb-01)

   The deployment name will appear in New Relic dashboards and helps you identify specific servers when you have multiple NGINX instances.

Edit your Collector configuration file (typically /etc/otelcol-contrib/config.yaml) and add the following sections. If you already have receivers, processors, or exporters sections, merge these with your existing configuration:

1. Configure the NGINX receiver:

receivers:
  nginx:
    endpoint: http://127.0.0.1:8080/nginx_status  # Replace with your stub status URL
    collection_interval: 30s                      # How often to collect metrics
    metrics:
      nginx.requests:
        enabled: true
      nginx.connections_accepted:
        enabled: true
      nginx.connections_handled:
        enabled: true
      nginx.connections_current:
        enabled: true

2. Configure processors for metadata and batching:

processors:
  # Detect system information (hostname, etc.)
  resourcedetection:
    detectors: [system]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true

  # Add NGINX-specific identification
  resource:
    attributes:
      - key: nginx.server.endpoint
        value: "http://127.0.0.1:8080/nginx_status"  # Replace with your endpoint
        action: upsert
      - key: nginx.deployment.name
        value: "production-web-01"                    # Replace with your server name
        action: upsert

  # Batch metrics for efficient sending
  batch:
    timeout: 30s
    send_batch_size: 1024

  # Transform metrics for better display in New Relic
  transform/nginx_metrics:
    metric_statements:
      - context: resource
        statements:
          # Customize the display name as needed for your New Relic dashboard
          - set(attributes["nginx.display.name"], Concat(["server", attributes["nginx.deployment.name"]], ":"))

3. Configure the New Relic exporter:

exporters:
  # Send metrics to New Relic via OTLP
  otlphttp/newrelic:
    endpoint: ${env:NEWRELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEWRELIC_LICENSE_KEY}

4. Connect everything with a pipeline:

service:
  pipelines:
    metrics:
      receivers: [nginx]
      processors: [resourcedetection, resource, batch, transform/nginx_metrics]
      exporters: [otlphttp/newrelic]

Replace these values in the configuration above:

1. Endpoint URL: Change http://127.0.0.1:8080/nginx_status to match your stub status configuration from Step 1

2. Deployment name: Replace production-web-01 with a unique identifier for this NGINX server

   Choosing a good deployment name:

• Use descriptive names like production-web-01, staging-api, or prod-lb-01
• Must be unique across all NGINX servers within your New Relic account
• This name will appear in dashboards and help you identify specific servers
• Keep it short but meaningful for easy filtering and alerting

Before configuring authentication, gather these two pieces of information:

1. Your License Key:

   • Go to one.newrelic.com → API Keys
   • Find "Ingest - License" section → Click Show key
   • Copy the license key (starts with "NRAK-...")

2. Your OTLP Endpoint: Use the appropriate endpoint for your New Relic region. See Configure endpoint, port, and protocol for the complete list of endpoints and supported ports for your region. For example:

   • US region: https://otlp.nr-data.net:4318
   • EU region: https://otlp.eu01.nr-data.net:4318

1. Create a systemd override directory:

   bash

   コピー

   $
   sudo mkdir -p /etc/systemd/system/otelcol-contrib.service.d

2. Create the environment configuration file:

   bash

   コピー

   $
   cat <<EOF | sudo tee /etc/systemd/system/otelcol-contrib.service.d/environment.conf
   $
   [Service]
   $
   Environment="NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318"
   $
   Environment="NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY_HERE"
   $
   EOF

3. Update the configuration with your credentials:

   • Replace https://otlp.nr-data.net:4318 with your region's endpoint
   • Replace YOUR_LICENSE_KEY_HERE with your actual license key from above

1. Apply the configuration changes:

   bash

   コピー

   $
   sudo systemctl daemon-reload
   $
   sudo systemctl restart otelcol-contrib.service

2. Verify the service is running:

   bash

   コピー

   $
   sudo systemctl status otelcol-contrib.service

   Expected output: Active: active (running) with no recent errors

3. Check startup logs:

   bash

   コピー

   $
   sudo journalctl -u otelcol-contrib.service -n 20

   Look for these success indicators:

   • ✅ "Everything is ready. Begin running and processing data."
   • ✅ "Scraping metrics" - Successfully collecting from NGINX
   • ✅ "Exporting metrics" - Successfully sending to New Relic

4. Generate test traffic (to create metrics):

   bash

   コピー

   $
   # Make a few requests to your NGINX server
   $
   curl http://localhost

5. Allow time for initial data to appear in New Relic, then access your NGINX dashboard to verify data collection.

First, configure NGINX to output JSON-formatted logs that are easier to parse and query.

Add this to your NGINX configuration (/etc/nginx/nginx.conf):

http {
    # JSON log format for better parsing
    log_format json_combined escape=json
    '{'
      '"time":"$time_local",'
      '"remote_addr":"$remote_addr",'
      '"request":"$request",'
      '"status":$status,'
      '"bytes_sent":$body_bytes_sent,'
      '"request_time":$request_time,'
      '"referer":"$http_referer",'
      '"user_agent":"$http_user_agent"'
    '}';

    # Use the JSON format for access logs
    access_log /var/log/nginx/access.log json_combined;
    error_log /var/log/nginx/error.log warn;

    # Your existing configuration...
}

Reload NGINX to apply the changes:

$
sudo nginx -t && sudo nginx -s reload

Add these sections to your /etc/otelcol-contrib/config.yaml:

receivers:
  # Your existing nginx receiver...
  nginx:
    # existing configuration...

  # Add log receivers
  filelog/nginx_access:
    include:
      - /var/log/nginx/access.log

  filelog/nginx_error:
    include:
      - /var/log/nginx/error.log

processors:
  # Your existing processors...

  # Add log processor
  transform/nginx_access_logs:
    log_statements:
    - context: resource
      statements:
        - set(attributes["nginx.display.name"], Concat(["server", attributes["nginx.deployment.name"]], ":"))
        - set(attributes["logtype"], "nginx")
  transform/nginx_error_logs:
    log_statements:
    - context: resource
      statements:
        - set(attributes["nginx.display.name"], Concat(["server", attributes["nginx.deployment.name"]], ":"))
        - set(attributes["logtype"], "nginx-error")

service:
  pipelines:
    # Your existing metrics pipeline...
    metrics:
      receivers: [nginx]
      processors: [resourcedetection, resource, batch, transform/nginx_metrics]
      exporters: [otlphttp/newrelic]

    # Add log pipelines
    logs/nginx-access:
      receivers: [filelog/nginx_access]
      processors: [resource, batch, transform/nginx_access_logs]
      exporters: [otlphttp/newrelic]

    logs/nginx-error:
      receivers: [filelog/nginx_error]
      processors: [resource, batch, transform/nginx_error_logs]
      exporters: [otlphttp/newrelic]

Allow the Collector to read NGINX log files:

$
# Add collector user to adm group (has read access to logs)
$
sudo usermod -a -G adm otelcol-contrib
$

$
# Ensure log files are readable
$
sudo chmod 644 /var/log/nginx/access.log
$
sudo chmod 644 /var/log/nginx/error.log

Restart the Collector:

$
sudo systemctl restart otelcol-contrib.service

Getting 404 Not Found:

$
curl http://127.0.0.1:8080/nginx_status

Solutions:

• Check that the location path matches your request URL
• Verify the configuration was added to the correct server block
• Run sudo nginx -T | grep -A5 nginx_status to confirm configuration is loaded

Getting 403 Forbidden:

• Ensure you're testing from localhost (127.0.0.1)
• Check your allow/deny directives in the NGINX configuration
• Verify no other access restrictions are blocking the request

Connection refused:

• Check if NGINX is running: sudo systemctl status nginx
• Verify port 8080 isn't blocked: sudo netstat -tlnp | grep :8080
• Check firewall rules if applicable

Check service status:

$
sudo systemctl status otelcol-contrib.service
$
sudo journalctl -u otelcol-contrib.service -n 50

Common causes and fixes:

• YAML syntax errors - Fix indentation and syntax issues
• Missing environment variables - Verify credentials are set in environment file
• File permission issues - Run sudo chown otelcol-contrib:otelcol-contrib /etc/otelcol-contrib/config.yaml
• Invalid endpoint URLs - Check NGINX endpoint and New Relic OTLP endpoint
• Missing component configurations - Ensure all receivers, processors, and exporters referenced in the service pipelines section are actually defined in their respective configuration sections above

Restart after fixes:

$
sudo systemctl restart otelcol-contrib.service

Step-by-step diagnosis:

1. Verify NGINX stub status works:

   bash

   コピー

   $
   curl http://127.0.0.1:8080/nginx_status

   Should return connection statistics.

2. Check Collector is running and healthy:

   bash

   コピー

   $
   sudo systemctl status otelcol-contrib.service
   $
   sudo journalctl -u otelcol-contrib -n 20

   The logs will provide detailed context if there are any configuration or runtime issues with the OpenTelemetry Collector.

3. Check for data with NRQL:

   FROM Metric SELECT * WHERE nginx.deployment.name LIKE '%production%' LIMIT 1

   コピー

No logs appearing in New Relic:

$
# Check Collector logs for file errors
$
sudo journalctl -u otelcol-contrib -f | grep -i "filelog\|error"

Common issues:

• File permission errors - Add collector to adm group: sudo usermod -a -G adm otelcol-contrib
• Wrong file paths - Verify log file locations in your config