---
title: Link OpenTelemetry-instrumented applications to Kubernetes
source: https://docs.newrelic.com/docs/kubernetes-pixie/kubernetes-integration/advanced-configuration/link-otel-applications-kubernetes
---

If you're a developer running applications in Kubernetes, you can use New Relic to understand how the Kubernetes infrastructure affects your OpenTelemetry instrumented applications.

After completing the steps below, you can use the New Relic UI to correlate application-level metrics from OpenTelemetry with Kubernetes infrastructure metrics. This allows you to see the entire landscape of your telemetry data, and collaborate across teams to achieve faster mean time to resolution (MTTR) for issues in your Kubernetes environment.

## How we correlate the data [#correlate]

This document walks you through enabling your application to inject infrastructure-specific metadata into the telemetry data. The result is that the New Relic UI is populated with actionable information. Here are the steps you'll take to get started:

-   In each application container, define an environment variable to send telemetry data to the Collector.

-   Deploy the OpenTelemetry Collector as a `DaemonSet` in [agent mode](https://opentelemetry.io/docs/collector/getting-started/#agent) with `resourcedetection`, `resource`, `batch`, and `k8sattributes` processors to inject relevant metadata (cluster, deployment, and namespace names).

## Before you begin [#prereqs]

To successfully complete the steps below, you should already be familiar with OpenTelemetry and Kubernetes, and have done the following:

-   Created these environment variables:

    -   `OTEL_EXPORTER_OTLP_ENDPOINT`: See [New Relic OTLP endpoint](https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-otlp/) for more info.
    -   `NEW_RELIC_API_KEY`: See [New Relic API keys](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/) for more info.

-   Installed your [Kubernetes cluster with OpenTelemetry](https://docs.newrelic.com/docs/kubernetes-pixie/kubernetes-integration/installation/k8s-otel/#install).

-   Instrumented your applications with [OpenTelemetry](https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/opentelemetry-setup/), and successfully sent data to New Relic via OpenTelemetry Protocol (OTLP).

If you have general questions about using Collectors with New Relic, see our [Introduction to OpenTelemetry Collector with New Relic](https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/collector/opentelemetry-collector-intro).

## Configure your application to send telemetry data to the OpenTelemetry Collector [#instrument]

To set this up, you need to add a custom snippet to the `env` section of your Kubernetes YAML file. The example below shows the snippet for a sample frontend microservice (`Frontend.yaml`). The snippet includes 2 sections that do the following:

-   **Section 1:** Ensure that the telemetry data is sent to the Collector. This sets the environment variable `OTEL_EXPORTER_OTLP_ENDPOINT` with the host IP. It does this by calling the downward API to pull the host IP.

-   **Section 2:** Attach infrastructure-specific metadata. To do this, we capture `metadata.uid` using the downward API and add it to the `OTEL_RESOURCE_ATTRIBUTES` environment variable. This environment variable is used by the OpenTelemetry Collector's `resourcedetection` and `k8sattributes` processors to add additional infrastructure-specific context to telemetry data.

For each microservice instrumented with OpenTelemetry, add the highlighted lines below to your manifest's `env` section:

```yaml
# Frontend.yaml
apiVersion: apps/v1
kind: Deployment

# ...
spec:
  containers:
    - name: yourfrontendservice
      image: yourfrontendservice-beta
  env:
    # Section 1: Ensure that telemetry data is sent to the collector
    - name: HOST_IP
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
    
    # This is picked up by the opentelemetry sdks
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: "http://$(HOST_IP):55680"
    
    # Section 2: Attach infrastructure-specific metadata
    # Get pod ip so that k8sattributes can tag resources
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    
    - name: POD_UID
      valueFrom:
        fieldRef:
          fieldPath: metadata.uid
    
    # This is picked up by the resource detector
    - name: OTEL_RESOURCE_ATTRIBUTES
      value: "service.instance.id=$(POD_NAME),k8s.pod.uid=$(POD_UID)"
```

## Configure and deploy the OpenTelemetry Collector [#configure-otel-collector]

We recommend you deploy the [Collector as an agent](https://opentelemetry.io/docs/collector/getting-started/#agent) on every node within a Kubernetes cluster. The agent can receive telemetry data, and enrich telemetry data with metadata. For example, the Collector can add custom attributes or infrastructure information through processors, as well as handle batching, retry, compression and additional advanced features that are handled less efficiently at the client instrumentation level.

You can choose one of these options to monitor your cluster:

-   **(Recommended) [Install your Kubernetes cluster using OpenTelemetry](https://docs.newrelic.com/docs/kubernetes-pixie/kubernetes-integration/installation/k8s-otel/#install)**: This option automatically deploys the Collector as an agent. Everything will work right out of the box, you'll have the Kubernetes metadata in the APM telemetry and the Kubernetes UIs.

-   **Manual configuration and deployment**: If you prefer to configure it manually, follow these steps:

    ### Configure the OTLP exporter

    Add an OTLP exporter to your [OpenTelemetry Collector configuration YAML file](https://opentelemetry.io/docs/collector/configuration/) along with your New Relic license key as a header.

    ````yaml
    exporters:
      otlp:
        endpoint: $OTEL_EXPORTER_OTLP_ENDPOINT
        headers: api-key: $NEW_RELIC_API_KEY
    ```
      

    ````

    ### Configure the batch processor

    The batch processor accepts spans, metrics, or logs and places them in batches. This makes it easier to compress data and reduce outbound requests from the Collector.

    ````yaml
    processors:
      batch:
    ```


    ````

    ### Configure the resource detection processor

    The `resourcedetection` processor gets host-specific information to add additional context to the telemetry data being processed through the Collector. In this example, we use Google Kubernetes Engine (GKE) and Google Compute Engine (GCE) to get Google Cloud-specific metadata, including:

    -   `cloud.provider` ("gcp")
    -   `cloud.platform` ("`gcp_compute_engine`")
    -   `cloud.account.id`
    -   `cloud.region`
    -   `cloud.availability_zone`
    -   `host.id`
    -   `host.image.id`
    -   `host.type`

        ```yaml
        processors:
          resourcedetection:
            detectors: [gke, gce]
        ```

    ### Configure the Kubernetes attributes processor (general)

    When we run the `k8sattributes` processor as part of the OpenTelemetry Collector running as an agent, it detects the IP addresses of pods sending telemetry data to the OpenTelemetry Collector agent, using them to extract pod metadata. Below is a basic Kubernetes manifest example with only a processors section. To deploy the OpenTelemetry Collector as a `DaemonSet`, read this [comprehensive manifest example](https://github.com/newrelic-forks/microservices-demo/tree/main/src/otel-collector-agent).

    ````yaml
    processors:
      k8sattributes:
        auth_type: "serviceAccount"
        passthrough: false
        filter:
          node_from_env_var: KUBE_NODE_NAME
        extract:
          metadata:
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.deployment.name
            - k8s.cluster.name
            - k8s.namespace.name
            - k8s.node.name
            - k8s.pod.start_time
        pod_association:
          - from: resource_attribute
            name: k8s.pod.uid
    ```


    ````

    ### Configure the Kubernetes attributes processor (RBAC)

    You need to add configurations for role-based access control (RBAC). The `k8sattributes` processor needs `get`, `watch`, and `list` permissions for pods and namespaces resources included in the configured filters. This [example](https://github.com/newrelic-forks/microservices-demo/blob/main/otel-kubernetes-manifests/otel-collector-agent.yaml#L43-L69) shows how to configure role-based access control (RBAC) for `ClusterRole` to give a `ServiceAccount` the necessary permissions for all pods and namespaces in the cluster.

    ### Configure the Kubernetes attributes processor (discovery filter)

    When running the Collector as an agent, you should apply a discovery filter so that the processor only discovers pods from the same host that it's running on. If you don't use a filter, resource usage can be unnecessarily high, especially on very large clusters. Once the filter is applied, each processor will only query the Kubernetes API for pods running on its own node.

    To set the filter, use the downward API to inject the node name as an environment variable in the pod `env` section of the OpenTelemetry Collector agent configuration YAML file. For an example, see the [`otel-collector-config.yml`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/examples/kubernetes/otel-collector-config.yml) file on GitHub. This will inject a new environment variable to the OpenTelemetry Collector agent's container. The value will be the name of the node the pod was scheduled to run on.

    ````yaml
    spec:
      containers:
        - env:
          - name: KUBE_NODE_NAME
            valueFrom:
                fieldRef:
                apiVersion: v1
                fieldPath: spec.nodeName
    ```

    Then, you can filter by the node with the `k8sattributes`:

    ```yaml
    k8sattributes:
      filter:
        node_from_env_var: KUBE_NODE_NAME
    ```

    ````

## Validate that your configurations are working [#validate]

You should be able to verify that your configurations are working once you have successfully linked your OpenTelemetry data with your Kubernetes data.

> #### ⚠️ CAUTION
>
> The [Kubernetes summary page](https://docs.newrelic.com/docs/apm/apm-ui-pages/monitoring/kubernetes-summary-page) only displays data from applications monitored by New Relic agents or OpenTelemetry. If your environment uses a mix of different instrumentation providers, you may not see complete data on this page.

1.  Go to **[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > APM & Services** and select your application inside **Services - OpenTelemetry**.

2.  Click **Kubernetes** on the left navigation pane.

![This is an image of the Kubernetes APM page](https://docs.newrelic.com/images/apm_screenshot-crop_k8-apm-ui.webp "Kubernetes page")

Go to **[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > APM & Services > (selected app) > Kubernetes** to see the Kubernetes summary page.

## Choose your next step [#next]

[See our best practices guide](https://docs.newrelic.com/docs/integrations/open-source-telemetry-integrations/opentelemetry/opentelemetry-concepts)

Learn how to improve your use of OpenTelemetry and New Relic.

[Check out this blog post](https://newrelic.com/blog/how-to-relic/k8s-with-otel)

Correlate OpenTelemetry traces, metrics, and logs with Kubernetes performance data