Install Elasticsearch OpenTelemetry integration

Install the New Relic Elasticsearch OpenTelemetry integration to monitor your Elasticsearch clusters with industry-standard protocols. This guide walks you through configuring the OpenTelemetry Collector to collect metrics and logs from your Elasticsearch infrastructure and send them to New Relic.

To install the integration, complete the following steps:

Before you begin - Check requirements and prerequisites
Configure the OpenTelemetry Collector - Set up data collection
Set environment variables - Configure authentication
Find and use data - View your Elasticsearch data in New Relic
Set up alerts - Configure proactive monitoring

Step 1: Before you begin

Ensure you have:

Required access privileges - Elasticsearch cluster admin privileges and New Relic account with access
Elasticsearch version 7.16 or higher - This integration requires a modern Elasticsearch cluster
Monitor or manage cluster privileges - If security is enabled, you need either monitor or manage cluster privilege. See the Elasticsearch security privileges documentation for more details
Network connectivity - Outbound HTTPS connectivity (port 443) to New Relic's OTLP ingest endpoint
OpenTelemetry Collector - OpenTelemetry Collector Contrib installed and running on your host. Install via an official package (.deb or .rpm) to ensure the systemd service unit is created correctly
Configuration values ready - You'll need two key values for the configuration:
- Elasticsearch endpoint - Your actual Elasticsearch URL (replace https://localhost:9200)
- Cluster name - A unique name to identify your cluster in New Relic

Step 2: Configure the OpenTelemetry Collector

Configure the OpenTelemetry Collector to collect metrics and logs from your Elasticsearch cluster. Create or update your configuration file at /etc/otelcol-contrib/config.yaml.

The configuration varies based on your Elasticsearch setup and monitoring requirements. Choose the appropriate configuration below:

Start here if you have: An unsecured Elasticsearch cluster without authentication or SSL.

This configuration collects comprehensive metrics from Elasticsearch and host system without authentication:

Importante

Replace the endpoint value with your Elasticsearch cluster endpoint and update elasticsearch.cluster.name in the processor block with a unique name to uniquely identify your cluster in New Relic.

# =================================================================================================
# OpenTelemetry Collector Configuration for Elasticsearch and Host
# This configuration collects metrics and logs for a complete observability solution.
# =================================================================================================
# -------------------------------------------------------------------------------------------------
# Receivers
# Receivers define how data gets into the Collector. This config uses four receivers:
# - elasticsearch: to scrape metrics from the Elasticsearch API
# - hostmetrics: to collect system-level metrics from the host itself
# - filelog: to tail Elasticsearch log files
# - otlp: to accept data from other OpenTelemetry-instrumented services
# -------------------------------------------------------------------------------------------------
receivers:
  elasticsearch:
    endpoint: "http://localhost:9200"
    collection_interval: 15s
    metrics:
      elasticsearch.breaker.tripped:
        enabled: true
      elasticsearch.cluster.data_nodes:
        enabled: true
      elasticsearch.cluster.health:
        enabled: true
      elasticsearch.cluster.in_flight_fetch:
        enabled: true
      elasticsearch.cluster.nodes:
        enabled: true
      elasticsearch.cluster.pending_tasks:
        enabled: true
      elasticsearch.cluster.shards:
        enabled: true
      elasticsearch.cluster.state_update.time:
        enabled: true
      elasticsearch.index.documents:
        enabled: true
      elasticsearch.index.operations.merge.current:
        enabled: true
      elasticsearch.index.operations.time:
        enabled: true
      elasticsearch.indexing_pressure.memory.total.primary_rejections:
        enabled: true
      elasticsearch.node.cache.count:
        enabled: true
      elasticsearch.node.cache.evictions:
        enabled: true
      elasticsearch.node.cache.memory.usage:
        enabled: true
      elasticsearch.node.cluster.io:
        enabled: true
      elasticsearch.node.documents:
        enabled: true
      elasticsearch.node.disk.io.read:
        enabled: true
      elasticsearch.node.disk.io.write:
        enabled: true
      elasticsearch.node.fs.disk.available:
        enabled: true
      elasticsearch.node.fs.disk.total:
        enabled: true
      elasticsearch.node.http.connections:
        enabled: true
      elasticsearch.node.ingest.documents.current:
        enabled: true
      elasticsearch.node.ingest.operations.failed:
        enabled: true
      elasticsearch.node.open_files:
        enabled: true
      elasticsearch.node.operations.completed:
        enabled: true
      elasticsearch.node.operations.current:
        enabled: true
      elasticsearch.node.operations.get.completed:
        enabled: true
      elasticsearch.node.operations.get.time:
        enabled: true
      elasticsearch.node.operations.time:
        enabled: true
      elasticsearch.node.shards.reserved.size:
        enabled: true
      elasticsearch.node.thread_pool.tasks.finished:
        enabled: true
      elasticsearch.os.cpu.load_avg.1m:
        enabled: true
      elasticsearch.os.cpu.load_avg.5m:
        enabled: true
      elasticsearch.os.cpu.load_avg.15m:
        enabled: true
      elasticsearch.os.memory:
        enabled: true
      jvm.gc.collections.count:
        enabled: true
      jvm.gc.collections.elapsed:
        enabled: true
      jvm.memory.heap.max:
        enabled: true
      jvm.memory.heap.used:
        enabled: true
      jvm.memory.heap.utilization:
        enabled: true
      jvm.threads.count:
        enabled: true
  hostmetrics:
    collection_interval: 60s # Recommended for cost savings and stability
    scrapers:
      cpu:
        metrics:
          # CPU Utilization and Time are the core metrics
          system.cpu.utilization: {enabled: true}
          system.cpu.time: {enabled: true}
      load:
        metrics:
          # Load Averages (used for system health dashboards)
          system.cpu.load_average.1m: {enabled: true}
          system.cpu.load_average.5m: {enabled: true}
          system.cpu.load_average.15m: {enabled: true}
      memory:
        metrics:
          # Memory Usage and Utilization
          system.memory.usage: {enabled: true}
          system.memory.utilization: {enabled: true}
      disk:
        metrics:
          # Disk I/O operations (throughput)
          system.disk.io: {enabled: true}
          system.disk.operations: {enabled: true}
      filesystem:
        metrics:
          # Filesystem usage (disk space capacity)
          system.filesystem.usage: {enabled: true}
          system.filesystem.utilization: {enabled: true} 
      network:
        # Since this was already working, keeping it simple is best.
        # But for completeness:
        metrics:
          system.network.io: {enabled: true}
          system.network.packets: {enabled: true}
      process:
           metrics:
             process.cpu.utilization:
               enabled: true
# -------------------------------------------------------------------------------------------------
# Processors
# -------------------------------------------------------------------------------------------------
processors:
  cumulativetodelta: {}
  resource/cluster_name_override:
    attributes:
      # Use the actual cluster name defined in your Elasticsearch config
      - key: elasticsearch.cluster.name
        value: "<elasticsearch-cluster-name>" # <-- REPLACE THIS WITH A UNIQUE CLUSTER NAME TO UNIQUELY IDENTIFY YOUR CLUSTER IN NEW RELIC 
        action: upsert
  # This processor adds resource attributes to all telemetry data.
  # 'service.name' is crucial for creating an entity in New Relic.
  resourcedetection:
    detectors: [ system ]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true
        os.type:
          enabled: true 
  # This processor batches data for more efficient sending.
  batch:
    timeout: 10s
    send_batch_size: 1024
  # 1. CARDINALITY REDUCTION: Drops volatile or redundant attributes
  attributes/cardinality_reduction:
    actions:
      # Filter out VOLATILE PROCESS IDS (High churn)
      - key: process.pid
        action: delete
      - key: process.parent_pid
        action: delete
      # Filter out REDUNDANT/STATIC METADATA (Already known at the Resource level)
      - key: elasticsearch.node.version
        action: delete
      - key: os.type
        action: delete
  transform/metadata_nullify:
    # We use 'metric_statements' to run OTTL logic on the metric signal
    metric_statements:
      - context: metric  # <-- Targets the high-level Metric structure itself
        statements:
          # Sets the 'description' field to an empty string ("")
          - set(description, "")
          # Sets the 'unit' field to an empty string ("")
          - set(unit, "")      
exporters:
  # This exporter sends all data to New Relic via OTLP/HTTP.
  otlphttp:
    endpoint: ${env:NEWRELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEWRELIC_LICENSE_KEY}
# -------------------------------------------------------------------------------------------------
# Service
# The service block defines the pipelines.
# -------------------------------------------------------------------------------------------------
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [resourcedetection, resource/cluster_name_override, attributes/cardinality_reduction, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]
    metrics/host:
      receivers: [hostmetrics]
      processors: [resourcedetection,batch]
      exporters: [otlphttp]

Use this if you have: A secured Elasticsearch cluster with authentication and/or SSL certificates.

Add authentication credentials and SSL configuration to the basic configuration above:

receivers:
  elasticsearch:
    endpoint: "https://localhost:9200"
    username: "elastic"
    password: "your_password"
    tls:
      ca_file: "/etc/elasticsearch/certs/http_ca.crt"
      insecure_skip_verify: false
    collection_interval: 15s

Optional: Include this if you want to send Elasticsearch log files to New Relic in addition to metrics.

Add the filelog receiver configuration to collect and forward Elasticsearch logs. Ensure the otelcol-contrib user has read access to the log files.

If running Elasticsearch on Linux (Host):

receivers:
  filelog:
    include:
      - /var/log/elasticsearch/elasticsearch.log #Replace with path of the elasticsearch log file.
      - /var/log/elasticsearch/*.log             #We can send multiple log files using regex.

If running Elasticsearch in Docker:

receivers:
  filelog:
    include:
      - /var/lib/docker/containers/*/*.log       # Replace with the container log file path. 
    operators:
      - type: move
        from: attributes.log
        to: body

Add filelog receiver in the service pipeline:

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [resource/cluster_name_override]
      exporters: [otlphttp]

Grant permission by adding the user to the group:

bash

$sudo usermod -a -G elasticsearch otelcol-contrib

Optional: Include this if you want to tag your data with custom attributes like environment, team, or region.

Use the resource/static_override processor to add custom metadata tags to all your metrics:

processors:
  resource/static_override:
    attributes:
      - key: env
        value: "production"
        action: upsert
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [resourcedetection, resource/cluster_name_override, resource/static_override, attributes/cardinality_reduction, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]
    metrics/host:
      receivers: [hostmetrics]
      processors: [resourcedetection, resource/static_override, batch]
      exporters: [otlphttp]

Dica

Correlate APM with Elasticsearch: To connect your APM application and Elasticsearch cluster, include the resource attribute es.cluster.name="your-cluster-name" in your APM metrics. This enables cross-service visibility and faster troubleshooting within New Relic.

Step 3: Set environment variables

Configure authentication by adding your New Relic and OTLP endpoint to the collector service.

Create a systemd override directory:

bash

$sudo mkdir -p /etc/systemd/system/otelcol-contrib.service.d

Write environment.conf with your OTLP endpoint. Replace YOUR_LICENSE_KEY with the New Relic license key and YOUR_OTLP_ENDPOINT with the appropriate endpoint for your region. Refer to the OTLP endpoint configuration documentation to select the right endpoint.
bash
```
$cat <<EOF | sudo tee /etc/systemd/system/otelcol-contrib.service.d/environment.conf
$[Service]
$Environment="NEWRELIC_OTLP_ENDPOINT=YOUR_OTLP_ENDPOINT"
$Environment="NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY"
$EOF
```

Reload systemd and restart the collector:

bash

$sudo systemctl daemon-reload
$sudo systemctl restart otelcol-contrib.service

Step 4: View your Elasticsearch data

Once the collector is running and sending data, you can view your Elasticsearch metrics in New Relic:

Go to one.newrelic.com > Integrations & Agents
Search for Elasticsearch (OpenTelemetry)
Under Dashboards, click Elasticsearch OpenTelemetry Dashboard
Select your account and click View dashboard

You should see dashboards showing cluster health, performance metrics, and resource usage.

Dica

Not seeing data? It may take a few minutes for data to appear. If you don't see metrics after 10 minutes, check our troubleshooting guide.

Next steps with your data:

Explore metrics: All Elasticsearch metrics are stored as Metric event types
Create custom queries: Use NRQL to build custom charts and dashboards
Set up alerts: Continue to Step 5 to configure proactive monitoring

Step 5: Set up alerts

Proactive monitoring with alerts helps you catch issues before they impact your users. To create alert conditions in New Relic:

Go to one.newrelic.com > Alerts > Alert Conditions.
Click Create condition.
Configure the alert using either Guided mode or the NRQL query builder.

The alert configurations below are recommended for robust Elasticsearch monitoring:

Essential Alerts (High Priority)

These alerts monitor critical cluster health issues that can cause data loss or service outages:

Alert Name	Threshold Rationale (Example Condition)
Unassigned Shards Alert	Metric `elasticsearch.cluster.shards` (where `state = 'unassigned'`) is above 0 for at least 5 minutes.
Healthy Data Nodes Alert	Metric `elasticsearch.cluster.data_nodes` is below your minimum required node count for at least 5 minutes.
Heap Usage Too High Alert	Heap usage percentage (Used/Max) is above 90% for at least 5 minutes.
Pending Tasks Alert	Metric `elasticsearch.cluster.pending_tasks` is above 5 for at least 5 minutes.

Additional Monitoring Alerts

These alerts help monitor performance and operational issues:

Alert Name	Threshold Rationale (Example Condition)
Query Time Slow Alert	95th percentile of `elasticsearch.node.operations.time` is above 5ms for at least 2 minutes.
Initializing Shards Too Long	Metric `elasticsearch.cluster.shards` (where `state = 'initializing'`) is above 0 for at least 5 minutes.
Relocating Shards Too Long	Metric `elasticsearch.cluster.shards` (where `state = 'relocating'`) is above 0 for at least 5 minutes.

Troubleshooting

If you encounter issues during installation or don't see data in New Relic, see our comprehensive troubleshooting guide for step-by-step solutions to common problems.

Install Elasticsearch OpenTelemetry integration

Step 1: Before you begin

Step 2: Configure the OpenTelemetry Collector

Basic metrics configuration

Importante

Authentication & SSL configuration

Enable logs (filelog receiver)

If running Elasticsearch on Linux (Host):

If running Elasticsearch in Docker:

Add filelog receiver in the service pipeline:

Add custom metadata

Dica

Step 3: Set environment variables

Step 4: View your Elasticsearch data

Dica

Step 5: Set up alerts

Essential Alerts (High Priority)

Additional Monitoring Alerts

Troubleshooting

Install Elasticsearch OpenTelemetry integration

Step 1: Before you begin .css-21sua1{background:none;border:none;width:0;padding:0;}

Step 2: Configure the OpenTelemetry Collector

Authentication & SSL configuration

Enable logs (filelog receiver)

Add custom metadata

Dica

Step 3: Set environment variables

Step 4: View your Elasticsearch data

Dica

Step 5: Set up alerts

Essential Alerts (High Priority)

Additional Monitoring Alerts

Troubleshooting

Step 1: Before you begin