Kubernetes APM metadata injection

With the injection of Kubernetes metadata for New Relic APM agents, we are able to provide Kubernetes object information in the context of a specific application distributed trace, transaction trace or error trace.

Our Kubernetes metadata injection project is open source. Here's the code to link APM and Infrastructure and the code to automatically manage certificates.

By default, we recommend using the Kubernetes certificates API to automatically manage the certificates required for the metadata injection. To proceed with this option:

  1. Download the manifest file:
    curl -O http://download.newrelic.com/infrastructure_agent/integrations/kubernetes/k8s-metadata-injection-latest.yaml 
  2. Then edit this file, replacing <YOUR_CLUSTER_NAME> with the name of your cluster.
  3. Next, apply the manifest to your Kubernetes cluster:
    kubectl apply -f k8s-metadata-injection-latest.yaml 

By default, all the pods you create will have the correct environment variables set and the injection will apply to all the cluster.

Optionally, you have the possibility to enable the injection only on specific namespaces. Using the label-based injection option, the injection is only applied to namespaces that have the newrelic-metadata-injection label set to enabled:

kubectl label <YOUR_NAMESPACE> newrelic-metadata-injection=enabled

Validate your installation

In order to validate that the webhook got installed correctly, you can deploy a new pod and check if the New Relic environment variables were injected.

  1. Create a dummy pod containing busybox by running:
    kubectl create -f https://git.io/vPieo
  2. Check if New Relic environment variables were injected:
    kubectl exec busybox0 -- env | grep NEW_RELIC_METADATA_KUBERNETES
    
    NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME=fsi
    NEW_RELIC_METADATA_KUBERNETES_NODE_NAME=nodea
    NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME=default
    NEW_RELIC_METADATA_KUBERNETES_POD_NAME=busybox0
    NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME=busybox
    

Certificate rotation

Certificates signed by Kubernetes have an expiration of 1 year. (See Kubernetes source code).

Future releases of New Relic Kubernetes metadata inection will implement certificate rotation.

Your cluster needs to have the MutatingAdmissionWebhook controller enabled. This feature requires Kubernetes 1.9 or higher and might not be enabled by default. Verify that your cluster is compatible by running the following command:

kubectl api-versions | grep admissionregistration.k8s.io/v1beta1 

admissionregistration.k8s.io/v1beta1 

If you see a different result, follow the official Kubernetes documentation to learn how to enable admission control in your cluster.

Note: The Kubernetes metadata for New Relic APM agents is not compatible with OpenShift due to differences in certificate management compared to upstream Kubernetes.

Troubleshooting

No Kubernetes metadata in APM or distributed tracing transactions

Problem

You look at the transactions in APM or distributed tracing and there is no Kubernetes metadata included in the transactions' attributes.

Solution

  1. Verify that the environment variables are being correctly injected by following the instructions described in the Validate your installation step.
  2. In case they are not present, get the name of the metadata injection pod by running kubectl get pods | grep newrelic-metadata-injection-deployment and then run kubectl logs -f pod/<POD_NAME>.
  3. In another terminal, create a new pod (you can use the example provided in the Validate your installation step) and inspect the logs of the metadata injection deployment for errors
  4. Ensure the metadata injection setup job ran successfully by inspecting the output of kubectl get job newrelic-metadata-setup.
  5. In case the job's COMPLETION is not 1/1, investigate the logs of the setup job: kubectl logs job/newrelic-metadata-setup.
  6. Ensure the CertificateSigningRequest is approved and issued by running kubectl get csr newrelic-metadata-injection-svc.default.
  7. Ensure the TLS secret is present by running kubectl get secret newrelic-metadata-injection-secret.

For more help

Other Kubernetes integration resources:

  • For discussions about the Kubernetes integration, visit New Relic's Explorers Hub.
  • For integration version changes, see the release notes.