AWS Elasticsearch monitoring integration

Access to this feature depends on your subscription level. Requires Infrastructure Pro.

New Relic Infrastructure's integrations include an integration for reporting Amazon Elasticsearch data to New Relic products. This document explains the integration's features, how to activate it, and what data can be reported.

Features

Amazon Elasticsearch Service is a fully managed service that delivers Elasticsearch’s easy-to-use APIs and real-time capabilities along with the availability, scalability, and security required by production workloads. New Relic's Elasticsearch monitoring integration allows you to track cluster status, CPU utilization, read/write latency, throughput, and other metrics, at specific points in time. Elasticsearch data is also available for querying, data analysis, and chart creation in New Relic Insights.

Activate integration

To enable this integration:

  1. Make sure you have installed the Infrastructure agent before you activate AWS integrations from your Infrastructure account.
  2. Follow standard procedures to Connect AWS services to Infrastructure.

Configuration and polling

You can change the polling frequency and filter data using configuration options.

Default polling information for the AWS Elasticsearch integration:

  • New Relic polling interval: 5 minutes
  • Amazon CloudWatch data interval: 1 minute

Find and use data

To find this integration's data in Infrastructure, go to infrastructure.newrelic.com > Integrations > Amazon Web Services and select one of the Elasticsearch integration links.

In New Relic Insights, data is attached to the DatastoreSample event type, with a provider value of ElasticsearchCluster.

For more on how to find and use integration data, see Understand integration data.

Metric data

The Elasticsearch integration collects these metrics:

Name Relevant statistics Description
ClusterStatus.green Minimum, Maximum

Indicates that all index shards are allocated to nodes in the cluster.

ClusterStatus.yellow Minimum, Maximum

Indicates that the primary shards for all indices are allocated to nodes in a cluster, but the replica shards for at least one index are not. Note that single node clusters always initialize with this cluster status because there is no second node to which a replica can be assigned. You can either increase your node count to obtain a green cluster status, or you can use the Amazon ES API to set the number_of_replicas setting for your index to 0. For more information, see Amazon'd documentation for Updating Indices Settings.

ClusterStatus.red Minimum, Maximum

Indicates that the primary and replica shards of at least one index are not allocated to nodes in a cluster. For more information, see Amazon's documentation on Red Cluster Status.

Nodes Minimum, Maximum, Average The number of nodes in the Amazon ES cluster.
SearchableDocuments Minimum, Maximum, Average

The total number of searchable documents across all indices in the cluster.

DeletedDocuments Minimum, Maximum, Average

The total number of deleted documents across all indices in the cluster.

CPUUtilization Minimum, Maximum, Average

The maximum percentage of CPU resources used for data nodes in the cluster.

FreeStorageSpace Minimum

The free space, in megabytes, for all data nodes in the cluster.

ClusterUsedSpace Minimum, Maximum

The total used space, in megabytes, for a cluster. You can view this metric in the Amazon CloudWatch console, but not in the Amazon ES console.

JVMMemoryPressure Maximum

The maximum percentage of the Java heap used for all data nodes in the cluster.

AutomatedSnapshotFailure Minimum, Maximum

The number of failed automated snapshots for the cluster. A value of 1 indicates that no automated snapshot was taken for the domain in the previous 36 hours.

CPUCreditBalance Minimum

The remaining CPU credits available for data nodes in the cluster. A CPU credit provides the performance of a full CPU core for one minute. This metrics is available only for the t2.micro.elasticsearch, t2.small.elasticsearch, and t2.medium.elasticsearch instance types.

MasterCPUUtilization Average

The maximum percentage of CPU resources used by the dedicated master nodes. We recommend increasing the size of the instance type when this metric reaches 60 percent.

MasterFreeStorageSpace N/A This metric is not relevant and can be ignored. The service does not use master nodes as data nodes.
MasterJVMMemoryPressure Maximum

The maximum percentage of the Java heap used for all dedicated master nodes in the cluster. We recommend moving to a larger instance type when this metric reaches 85 percent.

MasterCPUCreditBalance Minimum

The remaining CPU credits available for dedicated master nodes in the cluster. A CPU credit provides the performance of a full CPU core for one minute. This metric is available only for the t2.micro.elasticsearch, t2.small.elasticsearch, and t2.medium.elasticsearch instance types.

ReadLatency Minimum, Maximum, Average

The latency, in seconds, for read operations on EBS volumes.

WriteLatency Minimum, Maximum, Average

The latency, in seconds, for write operations on EBS volumes.

ReadThroughput Minimum, Maximum, Average

The throughput, in bytes per second, for read operations on EBS volumes.

WriteThroughput Minimum, Maximum, Average

The throughput, in bytes per second, for write operations on EBS volumes.

DiskQueueDepth Minimum, Maximum, Average

The number of pending input and output (I/O) requests for an EBS volume.

ReadIOPS Minimum, Maximum, Average

The number of input and output (I/O) operations per second for read operations on EBS volumes.

WriteIOPS Minimum, Maximum, Average

The number of input and output (I/O) operations per second for write operations on EBS volumes.

Inventory data

The integration collects this ElasticSearch data as inventory data. For more about inventory data, see Understand and use data.

Configuration

The integration collects this data from aws/elasticsearch/cluster/config:

Inventory Description
aRN The Amazon resource name (ARN) of an Elasticsearch domain.
accessPolicies The IAM access policy.
created The domain creation status. True if the creation of an Elasticsearch domain is complete. False if domain creation is still in progress.
deleted The domain deletion status. True if a delete request has been received for the domain but resource cleanup is still in progress. False if the domain has not been deleted.
domainId The unique identifier for the specified Elasticsearch domain.
endpoint The Elasticsearch domain endpoint that you use to submit index and search requests.
processing The status of the Elasticsearch domain configuration. True if Amazon Elasticsearch Service is processing configuration changes. False if the configuration is active.

eBSOptions

The integration collects this data from aws/elasticsearch/cluster/config/eBSOptions:

Name Description
eBSEnabled Specifies whether EBS-based storage is enabled.
iops Specifies the IOPD for a Provisioned IOPS EBS volume (SSD).
volumeSize Integer to specify the size of an EBS volume.
volumeType Specifies the volume type for EBS-based storage.

snapshotOptions

The integration collects this data from aws/elasticsearch/cluster/config/snapshotOptions:

Name Description
automatedSnapshotStartHour Specifies the time, in UTC format, when the service takes a daily automated snapshot of the specified Elasticsearch domain.

elasticsearchClusterConfig

The integration collects this data from aws/elasticsearch/cluster/config/elasticsearchClusterConfig:

Name Description
dedicatedMasterCount Total number of dedicated master nodes, active and on standby, for the cluster.
dedicatedMasterEnabled A boolean value to indicate whether a dedicated master node is enabled.
dedicatedMasterType The instance type for a dedicated master node.
instanceCount The number of instances in the specified domain cluster.
instanceType The instance type for an Elasticsearch cluster.
zoneAwarenessEnabled A boolean value to indicate whether zone awareness is enabled.

For more help

Recommendations for learning more: