HCP Consul monitoring (via StatsD)

HCP (Hashicorp Cloud Platform) Consul is a version of Consul in which the control plane is managed by HashiCorp Cloud Platform. HCP Consul is a service mesh and service discovery solution provided by Hashicorp. It enables platform operators to quickly deploy a fully managed, secure-by-default service mesh, helping developers discover and securely connect any application on any runtime, including Kubernetes, Nomad, and Amazon ECS.

Keeping your Consul clusters healthy can help with optimizing SQL query times, troubleshooting slow HTTP response times, and real-time alerting on down-time for your Hashicorp clusters. Monitoring your Hashicorp cluster metrics is pivotal for keeping your Hashicorp mesh services operating at full capacity.

Monitor HCP Consul metrics with New Relic

There are a number of ways and environments in which HCP Consul is installed. In the examples below, we will work with HCP's Quick Start installed via Terraform.

New Relic supports monitoring of HCP Consul and Envoy metrics through the StatsD plugin integration. The StatsD plugin aggregates all the metrics exposed by HCP Consul and pushes them to New Relic for data visualization and alerting.

Generate the Terraform deploy file

Generate the Terraform deployment file from your HCP Portal account's Overview page.

Deploy the Consul cluster

Deploy the Consul cluster following the steps for the appropriate environment in the end-to-end overview. Important: Ensure you follow the Consul CLI configuration at the bottom of the Deploy page.

gostatsd installation

Copy deploy-gostatsd.yaml and rbac-gostatsd.yaml to your local Kubernetes client.

Edit deploy-gstatsd.yaml

1. Replace YOUR_NEW_RELIC_ACCOUNT_ID with your New Relic account ID.
2. Replace YOUR_NEW_RELIC_LICENSE_KEY with your New Relic license key.
3. Adjust URLs for EU or FedRAMP accounts if necessary (see comments in file).

Deploy gostatsd

Run the following commands in your local Kubernetes client:

$
kubectl apply -f deploy-gostatsd.yaml
$
kubectl apply -f rbac-gostatsd.yaml

Configure the Consul clients to report metrics

Edit consul-client-config Map

Add Consule telemetry configuration by editing the Consule Client's ConfigMap:

$
kubectl edit cm consul-client-config

Add the following under data:

telemetry-config.json: |-
  {   
    "telemetry": {
      "disable_hostname": false,
      "disable_compat_1.9": true,
      "dogstatsd_addr": "gostatsd.default.svc.cluster.local:8125",
      "dogstatsd_tags": ["consul.source.datacenter:YOUR_CONSUL_DATACENTER_NAME_HERE"]
    }
  }

The result should look something like this:

apiVersion: v1
data:
  central-config.json: |-
    {   
      "enable_central_service_config": true
    }   
  config.json: |-
    {   
      "check_update_interval": "0s"
    }   
  extra-from-values.json: '{}'
  telemetry-config.json: |-
    {   
      "telemetry": {
        "disable_hostname": false,
        "disable_compat_1.9": true,
        "dogstatsd_addr": "gostatsd.default.svc.cluster.local:8125",
        "dogstatsd_tags": ["consul.source.datacenter:YOUR_CONSUL_DATACENTER_NAME_HERE"]
      }   
    }

Notes:

1. We use the dogstatsd_addr here so we can add the consul.source.datacenter tag to each metric. This lets you filter your dashboards by Consul Datacenter.
2. Be sure to replace YOUR_CONSUL_DATACENTER_NAME_HERE with your Consul Datacenter name.
3. See Consul telemetry configuration options.

Edit consul-connect-injector deployment

Enable Consule metrics in the deployment by editing the deployment:

$
kubectl edit deploy consul-connect-injector

Search for default-enable-metrics and set it to true:

$
-default-enable-metrics=true \

Restart Consul client pods

The Consul clients may, or may not, restart after the configuration changes. If they do not, restart each Consul client pod:

$
kubectl delete pod consul-client-<POD_ID> -n <NAMESPACE>

Browse Consul metrics in New Relic

To see your HCP Consul data in New Relic, navigate to query builder.

Then, use the query below to see HCP Consul metrics:

SELECT * FROM Metric WHERE metricName LIKE 'consul.%' SINCE 1 minute ago

Install the dashboard

Get started in minutes with a pre-built dashboard to see key metrics from HCP Consul about your clients, networks, runtime, serf, and Envoy.

Go to the HCP Consul quickstart in New Relic Instant Observability, and click + Install now.

Troubleshooting

For a pod that crashes at init:

$
kubectl logs PODNAME -c consul-connect-inject-init

For a running pod:

$
kubectl logs PODNAME -c DEPLOYMENTNAME

Useful kubectl snippets

Delete all pods:

$
kubectl get pods | grep PODNAMEPREFIX | cut -f 1 -d ' ' | xargs kubectl delete pods

Delete a deployment and pods it generates:

$
kubectl delete deployment DEPLOYMENTNAME -n default