Etcd is a strongly consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. It gracefully handles leader elections during network partitions and can tolerate machine failure, even in the leader node.
Use New Relic to display a curated dashboard based on Prometheus metrics which helps you understand the health of your Etcd cluster. With New Relic you can monitor:
- Leader changes
- Server proposals failures and committed
- gRPC stats
- Disk write latency
Follow these steps to enable the integration.
Follow the Etcd documentation for Prometheus to discover the metrics endpoints.
Set up Prometheus monitoring. Prometheus metrics needs to be integrated with New Relic, you can use the Prometheus Agent or the Remote Write integration, see how to send Prometheus metrics.
Use the following query to confirm metrics are being ingested as expected:FROM Metric SELECT count(*) WHERE metricName LIKE 'etcd_%' FACET metricName LIMIT MAX
Once you imported, you can edit or clone the assets to adapt them to your specific requirements.
Some charts of the dashboard include queries with conditions that require the identification of your pod or endpoint with one of the these labels
k8s-appcontaining the string
The different sets of metrics exposed by this integration are defined in the Etcd documentation.
Use the following NRQL queries to understand the metrics being ingested in New Relic:
List unique metric names:FROM Metric SELECT uniques(metricName) WHERE metricName LIKE 'etcd_%' LIMIT MAX
Count number of metric updates:FROM Metric SELECT datapointcount() WHERE metricName LIKE 'etcd_%' LIMIT MAX
Estimate data ingestion (daily ingest, in bytes):FROM Metric SELECT bytecountestimate() WHERE metricName LIKE 'etcd_%' SINCE1 day ago
Follow the troubleshooting tips from Etcd documentation to make sure that metrics are configured as expected on your cluster.
You can also check the specific troubleshooting guidelines for Prometheus integrations.