Elasticsearch monitoring integration

Gain deep insights into Elasticsearch performance with seamless data integration into New Relic. Monitor key metrics at the cluster, node, and index level - all within our powerful platform. Create alerts to stay ahead of performance anomalies, build custom dashboards for tailored views, and proactively optimize your Elasticsearch monitoring.

To get the most out of this page, select the installation method that fits your environment. You need a New Relic account before starting the installation process.

Tip

Use guided install to quickly see your data in the UI

The guided install is a single CLI command you can run to monitor your Elasticsearch instance. It's a good option for small organizations, or for anyone who wants to test out New Relic.

Guided install

EU guided install

For a more permanent and scalable solution, we recommend the standard manual install of the agent: keep reading for how to do that.

Elasticsearch-config.yml sample files

Here are some example basic and advanced sample config files:

This is the basic configuration used to collect metrics and inventory from your localhost:

integrations:
  - name: nri-elasticsearch
    env:
      HOSTNAME: localhost
      PORT: 9200
      USERNAME: elasticsearch_user
      PASSWORD: elasticsearch_password
      CONFIG_PATH: /etc/elasticsearch/elasticsearch.yml
    interval: 15s
    labels:
      environment: production
    inventory_source: config/elasticsearch

This configuration collects metrics every 15 seconds and inventory every 60 seconds:

integrations:
  - name: nri-elasticsearch
    env:
      METRICS: true
      HOSTNAME: localhost
      PORT: 9200
      USERNAME: elasticsearch_user
      PASSWORD: elasticsearch_password
      REMOTE_MONITORING: true
    interval: 15s
    labels:
      environment: production

  - name: nri-elasticsearch
    env:
      INVENTORY: true
      HOSTNAME: localhost
      PORT: 9200
      USERNAME: elasticsearch_user
      PASSWORD: elasticsearch_password
      CONFIG_PATH: /etc/elasticsearch/elasticsearch.yml
    interval: 60s
    labels:
      environment: production
    inventory_source: config/elasticsearch

In this configuration we are using the environment variable ELASTIC_HOST to populate the HOSTNAME setting of the integration:

integrations:
  - name: nri-elasticsearch
    env:
      METRICS: "true"
      HOSTNAME: {{ELASTIC_HOST}}
      PORT: 9200
      USERNAME: elasticsearch_user
      PASSWORD: elasticsearch_password
    interval: 15s
    labels:
      env: production
      role: load_balancer

The integration is capable of collecting all metrics from the cluster just connecting to one of the nodes, using a basic configuration showed before all the cluster metrics can be scraped.

Another way to configure the integration is to configure the scrape of all instances but activating the MASTER_ONLY parameter which makes the integration only collect metrics if the scraped instance is master. This allows to share the same config for all the nodes and avoid data duplication.
This is how the config file of each of the nodes should look like:

integrations:
  - name: nri-elasticsearch
    env:
      HOSTNAME: localhost
      PORT: 9200
      USERNAME: elasticsearch_user
      PASSWORD: elasticsearch_password
      MASTER_ONLY: "true"
      CONFIG_PATH: /etc/elasticsearch/elasticsearch.yml
    interval: 15s
    labels:
      environment: production
    inventory_source: config/elasticsearch

Elasticsearch config settings

You can use the env section of the elasticsearch-config.yml file, to define specific settings related to Elasticsearch. These settings control the connection to your Elasticsearch instance, as well as other security settings and features.

The Elasticsearch integration collects both Metrics and Inventory information. In the table, use the Applies to column for the settings available to each collection:

Setting	Description	Default	Applies to
`HOSTNAME`	Hostname or IP where Elasticsearch is running.	`localhost`	M/I
`LOCAL_HOSTNAME`	Hostname or IP of the Elasticsearch node from which to collect inventory data. Should only be set if you do not wish to collect inventory data against localhost.	`localhost`	M
`PORT`	Port on which Elasticsearch is listening.	`9200`	M/I
`USERNAME`	Username for accessing Elasticsearch Node.	N/A	M/I
`PASSWORD`	Password for the given user.	N/A	M/I
`CLUSTER_ENVIRONMENT`	A way to further specify which cluster we are gathering data, for example, 'staging'.	N/A	M/I
`USE_SSL`	Use SSL when communicating with the Elasticsearch node.	`false`	M/I
`CA_BUNDLE_FILE`	Location of SSL certificate on the host. Only required if `USE_SSL` is true.	N/A	M/I
`CA_BUNDLE_DIR`	Alternative Certificate Authority bundle directory.	N/A	M/I
`SSL_ALTERNATIVE_HOSTNAME`	Alternative server hostname that the integration will accept as valid for the purposes of SSL negotiation.	N/A	M/I
`TLS_INSECURE_SKIP_VERIFY`	Skip the verification of the server's certificate chain and hostname.	`false`	M/I
`CONFIG_PATH`	Path to the Elasticsearch configuration file.	`/etc/elasticsearch/elasticsearch.yml`	I
`TIMEOUT`	Timeout for API requests, in seconds.	`30`	M/I
`COLLECT_INDICES`	Signals whether to collect indices metrics or not.	`true`	M
`COLLECT_PRIMARIES`	Signals whether to collect primaries metrics or not.	`true`	M
`INDICES_REGEX`	Filter which indices are collected.	N/A	M
`MASTER_ONLY`	Collect cluster metrics on the elected master only.	`false`	M
`METRICS`	Set to `true` to enable metrics-only collection.	`false`
`INVENTORY`	Set to `true` to enable inventory-only collection.	`false`

Metric data

The Elasticsearch integration collects the following metrics. Each metric name is prefixed with a category indicator and a period, such as cluster. or shards..

Metric	Description
`cluster.dataNodes`	The number of data nodes in the cluster.
`cluster.nodes`	The number of nodes in the cluster.
`cluster.status`	The Elasticsearch cluster health: `red`, `yellow`, or `green`.
`shards.active`	The number of active shards in the cluster.
`shards.initializing`	The number of shards that are currently initializing.
`shards.primaryActive`	The number of active primary shards in the cluster.
`shards.relocating`	The number of shards that are relocating from one node to another.
`shards.unassigned`	The number of shards that are unassigned to a node.

Metric	Description
`activeSearches`	The number of active searches.
`activeSearchesInMilliseconds`	The time spent on the search fetch.
`breakers.estimatedSizeFieldDataCircuitBreakerInBytes`	The estimated size of the field data circuit breaker, in bytes.
`breakers.estimatedSizeParentCircuitBreakerInBytes`	The estimated size of the parent circuit breaker, in bytes.
`breakers.estimatedSizeRequestCircuitBreakerInBytes`	The estimated size of the request circuit breaker, in bytes.
`breakers.fieldDataCircuitBreakerTripped`	The number of times the field data circuit breaker has tripped.
`breakers.parentCircuitBreakerTripped`	The number of times the parent circuit breaker has tripped.
`breakers.requestCircuitBreakerTripped`	The number of times the request circuit breaker has tripped.
`cache.cacheSizeIDInBytes`	The size of the id cache, in bytes.
`flush.indexFlushDisk`	The number of index flushes to disk since start.
`flush.timeFlushIndexDiskInSeconds`	The time spent flushing the index to disk.
`fs.bytesAvailableJVMInBytes`	Bytes available to this Java virtual machine on this file store, in bytes.
`fs.bytesReadsInBytes`	The total bytes read from the file store, in bytes.
`fs.bytesUserIoOperationsInBytes`	The total bytes used for all I/O operations on the file store, in bytes.
`fs.iOOperations`	The total I/O operations on the file store.
`fs.reads`	The total number of reads from the file store.
`fs.totalSizeInBytes`	The total size of the file store, in bytes.
`fs.unallocatedBytesInBytes`	The total number of unallocated bytes in the file store, in bytes.
`fs.writes`	The total number of writes to the file store.
`fs.writesInBytes`	The total bytes written to the file store, in bytes.
`get.currentRequestsRunning`	The number of get requests currently running.
`get.requestsDocumentExists`	The number of get requests where the document existed.
`get.requestsDocumentExistsInMilliseconds`	The time spent on get requests where the document existed.
`get.requestsDocumentMissing`	The number of get requests where the document was missing.
`get.requestsDocumentMissingInMilliseconds`	The time spent on get requests where the document was missing.
`get.timeGetRequestsInMilliseconds`	The time spent on get requests.
`get.totalGetRequests`	The number of get requests.
`http.currentOpenConnections`	The number of current open HTTP connections.
`http.openedConnections`	The number of opened HTTP connections.
`indexing.docsCurrentlyDeleted`	The number of documents currently being deleted from an index.
`indexing.documentsCurrentlyIndexing`	The number of documents currently being indexed to an index.
`indexing.documentsIndexed`	The number of documents indexed to an index.
`indexing.timeDeletingDocumentsInMilliseconds`	The time spent deleting documents from an index.
`indexing.timeIndexingDocumentsInMilliseconds`	The time spent indexing documents to an index.
`indexing.totalDocumentsDeleted`	The number of documents deleted from an index.
`indices.indexingOperationsFailed`	The number of failed indexing operations.
`indices.indexingWaitedThrottlingInMilliseconds`	The time indexing waited due to throttling.
`indices.memoryQueryCacheInBytes`	The memory used by the query cache, in bytes.
`indices.numberIndices`	The number of documents across all primary shards assigned to the node.
`indices.queryCacheEvictions`	The number of query cache evictions.
`indices.queryCacheHits`	The number of query cache hits.
`indices.queryCacheMisses`	The number of query cache misses.
`indices.recoveryOngoingShardSource`	The number of ongoing recoveries for which a shard serves as a source.
`indices.recoveryOngoingShardTarget`	The number of ongoing recoveries for which a shard serves as a target.
`indices.recoveryWaitedThrottlingInMilliseconds`	The total time recoveries waited due to throttling.
`indices.requestCacheEvictions`	The number of request cache evictions.
`indices.requestCacheHits`	The number of request cache hits.
`indices.requestCacheMemoryInBytes`	The memory used by the request cache, in bytes.
`indices.requestCacheMisses`	The number of request cache misses.
`indices.segmentsIndexShard`	The number of segments in an index shard.
`indices.segmentsMaxMemoryIndexWriterInBytes`	The maximum memory used by the index writer, in bytes.
`indices.segmentsMemoryUsedDocValuesInBytes`	The memory used by doc values, in bytes.
`indices.segmentsMemoryUsedFixedBitSetInBytes`	The memory used by fixed bit set, in bytes.
`indices.segmentsMemoryUsedIndexSegmentsInBytes`	The memory used by index segments, in bytes.
`indices.segmentsMemoryUsedIndexWriterInBytes`	The memory used by the index writer, in bytes.
`indices.segmentsMemoryUsedNormsInBytes`	The memory used by norm, in bytes.
`indices.segmentsMemoryUsedSegmentVersionMapInBytes`	The memory used by the segment version map, in bytes.
`indices.segmentsMemoryUsedStoredFieldsInBytes`	The memory used by stored fields, in bytes.
`indices.segmentsMemoryUsedTermsInBytes`	The memory used by terms, in bytes.
`indices.segmentsMemoryUsedTermVectorsInBytes`	The memory used by term vectors, in bytes.
`indices.translogOperations`	The number of operations in the transaction log.
`indices.translogOperationsInBytes`	The size of the transaction log, in bytes.
`jvm.gc.collections`	The number of garbage collections run by the JVM.
`jvm.gc.collectionsInMilliseconds`	The time spent on garbage collection in the JVM.
`jvm.gc.concurrentMarkSweep`	The number of concurrent mark & sweep GCs in the JVM.
`jvm.gc.concurrentMarkSweepInMilliseconds`	The time spent on concurrent mark & sweep GCs in the JVM.
`jvm.gc.majorCollectionsOldGenerationObjects`	The number of major GCs in the JVM that collect old generation objects.
`jvm.gc.majorCollectionsOldGenerationObjectsInMilliseconds`	The time spent in major GCs in the JVM that collect old generation objects.
`jvm.gc.minorCollectionsYoungGenerationObjects`	The number of minor GCs in the JVM that collects young generation objects.
`jvm.gc.minorCollectionsYoungGenerationObjectsInMilliseconds`	The time spent in minor GCs in the JVM that collects young generation objects.
`jvm.gc.parallelNewCollections`	The number of parallel new GCs in the JVM.
`jvm.gc.parallelNewCollectionsInMilliseconds`	The time spent on parallel new GCs in the JVM.
`jvm.mem.heapCommittedInBytes`	The amount of memory guaranteed to be available to the JVM heap, in bytes.
`jvm.mem.heapMaxInBytes`	The maximum amount of memory that can be used by the JVM heap, in bytes.
`jvm.mem.heapUsed`	The percentage of memory currently used by the JVM heap as a value between `0` and `1`.
`jvm.mem.heapUsedInBytes`	The amount of memory currently used by the JVM heap, in bytes.
`jvm.mem.maxOldGenerationHeapInBytes`	The maximum amount of memory that can be used by the old generation heap, in bytes.
`jvm.mem.maxSurvivorSpaceInBytes`	The maximum amount of memory that can be used by the survivor space, in bytes.
`jvm.mem.maxYoungGenerationHeapInBytes`	The maximum amount of memory that can be used by the young generation heap, in bytes.
`jvm.mem.nonHeapCommittedInBytes`	The amount of memory guaranteed to be available to JVM non-heap, in bytes.
`jvm.mem.nonHeapUsedInBytes`	The amount of memory currently used by the JVM non-heap, in bytes.
`jvm.mem.usedOldGenerationHeapInBytes`	The amount of memory currently used by the old generation heap, in bytes.
`jvm.mem.usedSurvivorSpaceInBytes`	The amount of memory currently used by the survivor space, in bytes.
`jvm.mem.usedYoungGenerationHeapInBytes`	The amount of memory currently used by the young generation heap, in bytes.
`jvm.ThreadsActive`	The number of active threads in the JVM.
`jvm.ThreadsPeak`	The peak number of threads used by the JVM.
`merges.currentActive`	The number of currently active segment merges.
`merges.docsSegmentsMerging`	The number of documents across segments currently being merged.
`merges.docsSegmentMerges`	The number of documents across all merged segments.
`merges.mergedSegmentsInBytes`	The size of all merged segments, in bytes.
`merges.segmentMerges`	The number of segment merges.
`merges.sizeSegmentsMergingInBytes`	The size of the segments currently being merged, in bytes.
`merges.totalSegmentMergingInMilliseconds`	The time spent on segment merging.
`openFD`	The number of opened file descriptors associated with the current process, or`-1` if not supported.
`queriesTotal`	The number of queries.
`refresh.total`	The number of index refreshes.
`refresh.totalInMilliseconds`	The time spent on index refreshes.
`searchFetchCurrentlyRunning`	The number of search fetches currently running.
`searchFetches`	The number of search fetches.
`sizeStoreInBytes`	The size of the store, in bytes.
`threadpool.bulk.Queue`	The number of queued threads in the bulk pool.
`threadpool.bulkActive`	The number of active threads in the bulk pool.
`threadpool.bulkRejected`	The number of rejected threads in the bulk pool.
`threadpool.bulkThreads`	The number of threads in the bulk pool.
`threadpool.fetchShardStartedQueue`	The number of queued threads in the fetch shard started pool.
`threadpool.fetchShardStartedRejected`	The number of rejected threads in the fetch shard started pool.
`threadpool.fetchShardStartedThreads`	The number of threads in the fetch shard started pool.
`threadpool.fetchShardStoreActive`	The number of active threads in the fetch shard store pool.
`threadpool.fetchShardStoreQueue`	The number of queued threads in the fetch shard store pool.
`threadpool.fetchShardStoreRejected`	The number of rejected threads in the fetch shard store pool.
`threadpool.fetchShardStoreThreads`	The number of threads in the fetch shard store pool.
`threadpool.flushActive`	The number of active threads in the flush queue.
`threadpool.flushQueue`	The number of queued threads in the flush pool.
`threadpool.flushRejected`	The number of rejected threads in the flush pool.
`threadpool.flushThreads`	The number of threads in the flush pool.
`threadpool.forceMergeActive`	The number of active threads for force merge operations.
`threadpool.forceMergeQueue`	The number of queued threads for force merge operations.
`threadpool.forceMergeRejected`	The number of rejected threads for force merge operations.
`threadpool.forceMergeThreads`	The number of threads for force merge operations.
`threadpool.genericActive`	The number of active threads in the generic pool.
`threadpool.genericQueue`	The number of queued threads in the generic pool.
`threadpool.genericRejected`	The number of rejected threads in the generic pool.
`threadpool.genericThreads`	The number of threads in the generic pool.
`threadpool.getActive`	The number of active threads in the get pool.
`threadpool.getQueue`	The number of queued threads in the get pool.
`threadpool.getRejected`	The number of rejected threads in the get pool.
`threadpool.getThreads`	The number of threads in the get pool.
`threadpool.listenerActive`	The number of active threads in the listener pool. (Deprecated on Elasticsearch 8)
`threadpool.listenerQueue`	The number of queued threads in the listener pool. (Deprecated on Elasticsearch 8)
`threadpool.listenerRejected`	The number of rejected threads in the listener pool. (Deprecated on Elasticsearch 8)
`threadpool.listenerThreads`	The number of threads in the listener pool. (Deprecated on Elasticsearch 8)
`threadpool.managementActive`	The number of active threads in the management pool.
`threadpool.managementQueue`	The number of queued threads in the management pool.
`threadpool.managementRejected`	The number of rejected threads in the management pool.
`threadpool.managementThreads`	The number of threads in the management pool.
`threadpool.mergeActive`	The number of active threads in the merge pool.
`threadpool.mergeQueue`	The number of queued threads in the merge pool.
`threadpool.mergeRejected`	The number of rejected threads in the merge pool.
`threadpool.mergeThreads`	The number of threads in the merge pool.
`threadpool.percolateActive`	The number of active threads in the percolate pool.
`threadpool.percolateQueue`	The number of queued threads in the percolate pool.
`threadpool.percolateRejected`	The number of rejected threads in the percolate pool.
`threadpool.percolateThreads`	The number of threads in the percolate pool.
`threadpool.refreshActive`	The number of active threads in the refresh pool.
`threadpool.refreshQueue`	The number of queued threads in the refresh pool.
`threadpool.refreshRejected`	The number of rejected threads in the refresh pool.
`threadpool.refreshThreads`	The number of threads in the refresh pool.
`threadpool.searchActive`	The number of active threads in the search pool.
`threadpool.searchQueue`	The number of queued threads in the search pool.
`threadpool.searchRejected`	The number of rejected threads in the search pool.
`threadpool.searchThreads`	The number of threads in the search pool.
`threadpool.snapshotActive`	The number of active threads in the snapshot pool.
`threadpool.snapshotQueue`	The number of queued threads in the snapshot pool.
`threadpool.snapshotRejected`	The number of rejected threads in the snapshot pool.
`threadpool.snapshotThreads`	The number of threads in the snapshot pool.
`threadpool.activeFetchShardStarted`	The number of active threads in the fetch shard started pool.
`transport.connectionsOpened`	The number of connections opened for cluster communication.
`transport.packetsReceived`	The number of packets received in cluster communication.
`transport.packetsReceivedInBytes`	The size of data received in cluster communication, in bytes.
`transport.packetsSent`	The number of packets sent in cluster communication.
`transport.packetsSentInBytes`	The size of data sent in cluster communication, in bytes.

Metric	Description
`primaries.docsDeleted`	The number of documents deleted from the primary shards.
`primaries.docsnumber`	The number of documents in the primary shards.
`primaries.flushesTotal`	The number of index flushes to disk from the primary shards since start.
`primaries.flushTotalTimeInMilliseconds`	The time spent flushing the index to disk from the primary shards.
`primaries.get.documentsExist`	The number of get requests on primary shards where the document existed.
`primaries.get.documentsExistInMilliseconds`	The time spent on get requests from the primary shards where the document existed.
`primaries.get.documentsMissing`	The number of get requests from the primary shards where the document was missing.
`primaries.get.documentsMissingInMilliseconds`	The time spent on get requests from the primary shards where the document was missing.
`primaries.get.requests`	The number of get requests from the primary shards.
`primaries.get.requestsCurrent`	The number of get requests currently running on the primary shards.
`primaries.get.requestsInMilliseconds`	The time spent on get requests from the primary shards.
`primaries.index.docsCurrentlyDeleted`	The number of documents currently being deleted from an index on the primary shards.
`primaries.index.docsCurrentlyDeletedInMilliseconds`	The time spent deleting documents from an index on the primary shards.
`primaries.index.docsCurrentlyIndexing`	The number of documents currently being indexed to an index on the primary shards.
`primaries.index.docsCurrentlyIndexingInMilliseconds`	The time spent indexing documents to an index on the primary shards.
`primaries.index.docsDeleted`	The number of documents deleted from an index on the primary shards.
`primaries.index.docsTotal`	The number of documents indexed to an index on the primary shards.
`primaries.indexRefreshesTotal`	The number of index refreshes on the primary shards.
`primaries.indexRefreshesTotalInMilliseconds`	The time spent on index refreshes on the primary shards.
`primaries.merges.current`	The number of currently active segment merges on the primary shards.
`primaries.merges.docsSegmentsCurrentlyMerged`	The number of documents across segments currently being merged on the primary shards.
`primaries.merges.docsTotal`	The number of documents across all merged segments on the primary shards.
`primaries.merges.SegmentsCurrentlyMergedInBytes`	The size of the segments currently being merged on the primary shards, in bytes.
`primaries.merges.SegmentsTotal`	The number of segment merges on the primary shards.
`primaries.merges.segmentsTotalInBytes`	The size of all merged segments on the primary shards, in bytes.
`primaries.merges.segmentsTotalInMilliseconds`	The time spent on segment merging on the primary shards.
`primaries.queriesInMilliseconds`	The time spent querying on the primary shards.
`primaries.queriesTotal`	The number of queries to the primary shards.
`primaries.queryActive`	The number of currently active queries on the primary shards.
`primaries.queryFetches`	The number of query fetches currently running on the primary shards.
`primaries.queryFetchesInMilliseconds`	The time spent on query fetches on the primary shards.
`primaries.queryFetchesTotal`	The number of query fetches on the primary shards.
`primaries.sizeInBytes`	The size of all the primary shards, in bytes.

Metric	Description
`index.docs`	The number of documents in the index.
`index.docsDeleted`	The number of deleted documents in the index.
`index.health`	The status of the index: `red`, `yellow`, or `green`.
`index.primaryShards`	The number of primary shards in the index.
`index.primaryStoreSizeInBytes`	The store size of primary shards in the index.
`index.replicaShards`	The number of replica shards in the index.
`index.storeSizeInBytes`	The store size of primary and replica shards in the index, in bytes.

Elasticsearch monitoring integration

Tip

Elasticsearch-config.yml sample files

Basic configuration

Basic configuration with different metric and inventory intervals

Environment variables replacement

Cluster monitoring

Elasticsearch config settings

Metric data

ElasticsearchClusterSample event

ElasticsearchNodeSample event

ElasticsearchCommonSample event

ElasticsearchIndexSample event

Elasticsearch monitoring integration

Tip

Elasticsearch-config.yml sample files .css-21sua1{background:none;border:none;width:0;padding:0;}

Basic configuration with different metric and inventory intervals

Environment variables replacement

Cluster monitoring

Elasticsearch config settings

Metric data

ElasticsearchClusterSample event

ElasticsearchNodeSample event

ElasticsearchCommonSample event

ElasticsearchIndexSample event

Elasticsearch-config.yml sample files