OpenTelemetry for Kubernetes

OpenTelemetry observability for Kubernetes provides complete, open-source setup paired with a top-notch Kubernetes UI that is already compatible with our proprietary Kubernetes instrumentation. Our K8s UIs are designed to be provider agnostic, allowing you to select either OpenTelemetry or New Relic instrumentation based on your needs.

This document outlines the process for monitoring a Kubernetes cluster using OpenTelemetry. It involves the installation of the nr-k8s-otel-collector Helm chart within the cluster and the deployment of the necessary collectors to enable first-class observability.

By integrating Kubernetes components into the OpenTelemetry collector, we can transmit metrics, events, and logs directly to New Relic. These telemetry signals automatically enhance our out-of-the-box experiences such as the Kubernetes navigator, overview dashboard, Kubernetes events, or Kubernetes APM summary page.

How it works?

The nr-k8s-otel-collector Helm chart deploys these OpenTelemetry collectors:

• Deamonset collector: Deployed on each worker node and responsible for gathering metrics from the underlying host in the node, the cAdvisor, the Kubelet, and collecting logs from the containers.

• Deployment collector: Deployed on the master node and responsible for gathering metrics of Kube state metrics and Kubernetes cluster events.

  

Requirements

To send Kubernetes telemetry data to New Relic, we need an OpenTelemetry collector. Our New Relic distribution of OpenTelemetry (NRDOT) is already set up to automatically monitor a Kubernetes cluster. It does this by deploying all the necessary components through our nr-k8s-otel-collector Helm chart.

If you switch to a different OpenTelemetry collector, make sure it has all the key components you need:

• Attributes processor
• Filter processor
• Filelog receiver
• GroupByAttrs processor
• Hostmetrics receiver
• K8sAttributes processor
• K8sevents receiver
• Kubelet receiver
• MetricsTransform processor
• Prometheus receiver
• ResourceDetection processor
• Resource processor
• Transform processor

Install your Kubernetes cluster with OpenTelemetry

To get OpenTelemetry up and running in your cluster, follow these steps:

1. Download the Helm chart values file adapt it to meet your specific requirements.

   • Cluster name and license key are mandatory.

   • Check the entire list of configuration parameters.

2. Install the Helm chart together with the values file.

   bash

   Copy

   $
   helm repo add newrelic https://helm-charts.newrelic.com
   $
   helm upgrade nr-k8s-otel-collector newrelic/nr-k8s-otel-collector -f your-custom-values.yaml -n newrelic --create-namespace --install

3. Ensure the pods have been successfully spun up.

   bash

   Copy

   $
   kubectl get pods -n newrelic --watch

4. Make sure New Relic is getting the data it needs, including metrics, events, and logs, by running the right queries. See Introduction to the query builder for more information.

   FROM Metric SELECT * WHERE k8s.cluster.name='<CLUSTER_NAME>'
   FROM InfrastructureEvent SELECT * WHERE k8s.cluster.name='<CLUSTER_NAME>'
   FROM Log SELECT * WHERE k8s.cluster.name='<CLUSTER_NAME>'

   Copy

5. If you're using a GKE AutoPilot cluster, it's necessary to apply the following configuration in your values.yaml file to ensure compatibility and proper functionality of the OpenTelemetry collectors.

   privileged: false
   receivers:
       filelog:
           enabled: false
   daemonset:
       containerSecurityContext:
           privileged: false

   Copy

Uninstall your Kubernetes cluster with OpenTelemetry

To stop monitoring a Kubernetes cluster with OpenTelemetry, run this command:

$
helm uninstall nr-k8s-otel-collector -n newrelic

Reduce data ingest

The LowDataMode option is enabled by default to ingest only the metrics required by our Kubernetes UIs.

If you need to cut down even more on data ingestion, increase the scrape interval in the nr-k8s-otel-collector chart values for each deployed component.

Metrics

• Metrics - Full list

• Metrics - LowDataMode list

Find and use data

Check out these documents to learn more on how to find data:

• Explore your Kubernetes cluster to know the status of your cluster, from the control plane to nodes and pods.

• Kubernetes APM summary page which offers insights into your Kubernetes integration alongside your monitored applications.

Troubleshooting

Check out the logs of the collector pod that's experiencing issues. Run this command:

$
kubectl logs <otel-pod-name> -n newrelic

You can also set the verboseLog parameter to true in the nr-k8s-otel-collector Helm chart.

Common errors

Check out the Common errors section in our GitHub repository.

Support

If you have issues with the OpenTelemetry observability for Kubernetes:

• Have a look at the issues section on GitHub for any similar problems or consider opening a new issue.

• Reach out to the New Relic support for assistance.