Link APM-instrumented applications to Kubernetes

You can surface Kubernetes metadata and link it to your APM agents as distributed traces to explore performance issues and troubleshoot transaction errors. For more information, see this blog post about monitoring app performance via Kubernetes.

The metadata injection product uses a MutatingAdmissionWebhook to add the following environment variables to pods:

NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME
NEW_RELIC_METADATA_KUBERNETES_NODE_NAME
NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME
NEW_RELIC_METADATA_KUBERNETES_DEPLOYMENT_NAME
NEW_RELIC_METADATA_KUBERNETES_POD_NAME
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_IMAGE_NAME

Tip

Our Kubernetes metadata injection project is open source. Here's the code to link APM and infrastructure data.

Compatibility and requirements

To connect your applications to Kubernetes, you need to be able to deploy `MutatingWebhookConfiguration' to your Kubernetes cluster.

To verify that you have the required permissions, run this command:

bash

$kubectl auth can-i create mutatingwebhookconfigurations.admissionregistration.k8s.io -A

The output for the command above should be something similar to:

bash

$yes

If you see a different result, follow the Kubernetes documentation to enable admission control in your cluster.

Network requirements

For Kubernetes to talk to our MutatingAdmissionWebhook, the master node (or API server container, depending on how the cluster is set up) should allow egress for HTTPS traffic on port 443 to pods in all other nodes in the cluster.

This may require specific configuration depending on how your infrastructure is set up (on-premises, AWS, Google Cloud, etc.).

APM agent compatibility

The following New Relic agents collect Kubernetes metadata:

Set up the injection of metadata

When you install our integration using Helm, it includes metadata injection. When configuring the nri-bundle chart, make sure you enable the metadata injection webhook as follows.

nri-metadata-injection:
    enabled: true

After deploying the webhook, restart your application pods. They need to pick up the required environment variables.

By default, all the pods you create that include APM agents have the correct environment variables set and the metadata injection applies to the entire cluster. To check that the environment variables have been set, any running container must be stopped and a new instance started. See Validate the injection of metadata for more info.

This default configuration also uses the Kubernetes certificates API to automatically manage the certificates required for the injection. If needed, you can limit the injection of metadata to specific namespaces in your cluster or self-manage your certificates.

Custom configuration

Limit namespaces subject to injection

You can limit the injection of metadata only to specific namespaces by using labels.

To enable this feature, add the following to values-newrelic.yaml file:

nri-metadata-injection:
    injectOnlyLabeledNamespaces: true

With this option, injection is only applied to those namespaces that have the newrelic-metadata-injection label set to enabled:

bash

$kubectl label namespace YOUR_NAMESPACE newrelic-metadata-injection=enabled

Use cert-manager to generate certs

By default, our chart uses kube-webhook-certgen to automatically generate the required certificates for the webhook to run.

However, if you've cert-manager installed, you can configure our chart to use it instead, which can make deploying much easier:

nri-metadata-injection:
  certManager:
    enabled: true

Manage custom certificates

Tip

Manually managing webhook certificates is recommended for advanced users only. New Relic support team might not be able to help troubleshooting this configuration.

To use custom certificates you need to disable the automatic installation of certificates when you are installing using Helm.

To disable the installation for certificates just modify nri-bundle Helm values.yaml like this:

nri-metadata-injection:
  customTLSCertificate: true

Now you can proceed with the custom certificate management option. You need your certificate, server key, and Certification Authority (CA) bundle encoded in PEM format.

If you've them in the standard certificate format (X.509), install openssl, and run the following:

bash

$openssl x509 -in YOUR_CERTIFICATE_FILENAME -outform PEM -out YOUR_CERTIFICATE_FILENAME.pem
$openssl x509 -in YOUR_SERVER_KEY_FILENAME -outform PEM -out YOUR_SERVER_KEY_FILENAME.pem 
$openssl x509 -in YOUR_CA_BUNDLE_FILENAME -outform PEM -out YOUR_BUNDLE_FILENAME.pem

If your certificate and key pair are in another format, see the Digicert knowledgebase for more help.

Create the TLS secret with the signed certificate/key pair, and patch the mutating webhook configuration with the CA using the following commands:

bash

$kubectl create secret tls YOUR_NEWRELIC_METADATA_INJECTION_ADMISSION \
>  --key=YOUR_PEM_ENCODED_SERVER_KEY \
>  --cert=YOUR_PEM_ENCODED_CERTIFICATE \
>  --dry-run -o yaml |
$kubectl -n newrelic apply -f -
$
$caBundle=$(cat YOUR_PEM_ENCODED_CA_BUNDLE | base64 | td -d $'\n')
$kubectl patch mutatingwebhookconfiguration newrelic-metadata-injection-cfg --type='json' -p "[{'op': 'replace', 'path': '/webhooks/0/clientConfig/caBundle', 'value':'${caBundle}'}]"

Important

Certificates signed by Kubernetes have an expiration of one year. For more information, see the Kubernetes source code in GitHub.

Validate the injection of metadata

Deploy a new pod and check for the New Relic environment variables to verify the correct installation of the webhook responsible for injecting the metadata.

Create a dummy nginx pod by running:

bash

$kubectl run test-nginx --image nginx -n newrelic

Check if New Relic environment variables were injected:

bash

$kubectl exec -n newrelic test-nginx -- env | grep NEW_RELIC_METADATA_KUBERNETES

The expected output would be something like the following:

NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME=THE_CLUSTER_NAME
NEW_RELIC_METADATA_KUBERNETES_NODE_NAME=nodea
NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME=newrelic
NEW_RELIC_METADATA_KUBERNETES_POD_NAME=test-nginx
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME=nginx

Disable the injection of metadata

To uninstall the injection of metadata, change your values-newrelic.yaml file as follows:

webhook:
    enabled: false

After that, re-run the installation command.

Troubleshooting

Follow these troubleshooting tips as needed.

Problem

The creation of the secret by the k8s-webhook-cert-manager job used to fail due to the kubectl version used by the image when running in Kubernetes version 1.19.x, The new version 1.3.2 fixes this issue, therefore it is enough to run again the job using an update version of the image to fix the issue.

Solution

Update the image k8s-webhook-cert-manager to a version >= 1.3.2 and re-run the job.
Check that the secret was created correctly and that the k8s-metadata-injection pod starts.
Note that the new version of the manifest and of the nri-bundle are already updated with the correct version of the image.

Problem

There is no Kubernetes metadata included in the transaction attributes of your APM agent or in distributed tracing.

Solution

Verify that the environment variables are set correctly injected by following the instructions described in the Validate the injection of metadata section.

If they do not exist, get the name of the metadata injection pod by running:

bash

$kubectl get pods | grep newrelic-metadata-injection-deployment
$kubectl logs -f pod/my-pod

In another terminal, create a new pod and inspect the logs of the metadata injection deployment for errors. See Validate the injection of metadata section to create a new pod. For each pod created, there should be a set of 4 new entries in the logs, such as:

{"level":"info","ts":"2020-04-09T12:55:32.107Z","caller":"server/main.go:139","msg":"POST https://newrelic-metadata-injection-svc.default.svc:443/mutate?timeout=30s HTTP/2.0\" from 10.11.49.2:32836"}
{"level":"info","ts":"2020-04-09T12:55:32.110Z","caller":"server/webhook.go:168","msg":"received admission review","kind":"/v1, Kind=Pod","namespace":"default","name":"","pod":"busybox1","UID":"6577519b-7a61-11ea-965e-0e46d1c9335c","operation":"CREATE","userinfo":{"username":"admin","uid":"admin","groups":["system:masters","system:authenticated"]}}
{"level":"info","ts":"2020-04-09T12:55:32.111Z","caller":"server/webhook.go:182","msg":"admission response created","response":"[{\"op\":\"add\",\"path\":\"/spec/containers/0/env\",\"value\":[{\"name\":\"NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME\",\"value\":\"adn_kops\"}]},{\"op\":\"add\",\"path\":\"/spec/containers/0/env/-\",\"value\":{\"name\":\"NEW_RELIC_METADATA_KUBERNETES_NODE_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"spec.nodeName\"}}}},{\"op\":\"add\",\"path\":\"/spec/containers/0/env/-\",\"value\":{\"name\":\"NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}}},{\"op\":\"add\",\"path\":\"/spec/containers/0/env/-\",\"value\":{\"name\":\"NEW_RELIC_METADATA_KUBERNETES_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}}},{\"op\":\"add\",\"path\":\"/spec/containers/0/env/-\",\"value\":{\"name\":\"NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME\",\"value\":\"busybox\"}},{\"op\":\"add\",\"path\":\"/spec/containers/0/env/-\",\"value\":{\"name\":\"NEW_RELIC_METADATA_KUBERNETES_CONTAINER_IMAGE_NAME\",\"value\":\"busybox\"}}]"}
{"level":"info","ts":"2020-04-09T12:55:32.111Z","caller":"server/webhook.go:257","msg":"writing response"}

If there are no new entries in the logs, it means that the API server can't communicate with the webhook service, this could be due to network rules or security groups rejecting the communication.

To check if the API server is not being able to communicate with the webhook you should inspect the API server logs for errors like:
bash
```
$failed calling webhook "metadata-injection.newrelic.com": THE_ERROR_REASON
```

To get the API server logs:

Start a proxy to the Kubernetes API server by the executing this command in a terminal window and keep it running.
bash
```
$kubectl proxy --port=8001
```
Create a new pod in your cluster, this will make the API server try to communicate with the webhook. This command will create a busybox.
bash
```
$kubectl create -f https://git.io/vPieo
```

Retrieve the API server logs.

bash

$curl localhost:8001/logs/kube-apiserver.log > apiserver.log

Delete the busybox container.
bash
```
$kubectl delete -f https://git.io/vPieo
```
Inspect the logs for errors.
bash
```
$grep -E 'failed calling webhook' apiserver.log
```
Tip
One of the requirements for the metadata injection is that the API server must be allowed egress to the pods running on the cluster. If you encounter errors regarding connection timeouts or failed connections, make sure to check the cluster's security groups and firewall rules.
If there are no log entries in either the API server logs or the metadata injection deployment, it means that the webhook was not registered properly.
Ensure the metadata injection setup job ran successfully by inspecting the output of this command:
bash
```
$kubectl get job newrelic-metadata-setup
```
If the job isn't completed, investigate the logs of the setup job:
bash
```
$kubectl logs job/newrelic-metadata-setup
```
Ensure the CertificateSigningRequest is approved and issued by running this command:
bash
```
$kubectl get csr newrelic-metadata-injection-svc.default
```
Ensure the TLS secret is present by running this command:
bash
```
$kubectl get secret newrelic-metadata-injection-secret
```

Ensure the CA bundle is present in the mutating webhook configuration:

bash

$kubectl get mutatingwebhookconfiguration newrelic-metadata-injection-cfg -o json

Ensure the TargetPort of the Service resource matches the Port of the Deployment's container:

bash

$kubectl describe service/newrelic-metadata-injection-svc
$kubectl describe deployment/newrelic-metadata-injection-deployment

Link APM-instrumented applications to Kubernetes

Tip

Compatibility and requirements

Network requirements

APM agent compatibility

Set up the injection of metadata

Custom configuration

Limit namespaces subject to injection

Use cert-manager to generate certs

Manage custom certificates

Tip

Important

Validate the injection of metadata

Disable the injection of metadata

Troubleshooting

No Kubernetes metadata in APM or distributed tracing transactions

Problem

Solution

No Kubernetes metadata in the transaction attributes

Problem

Solution

Tip

Link APM-instrumented applications to Kubernetes

Tip

Compatibility and requirements .css-21sua1{background:none;border:none;width:0;padding:0;}

Network requirements

APM agent compatibility

Set up the injection of metadata

Custom configuration

Limit namespaces subject to injection

Use cert-manager to generate certs

Manage custom certificates

Tip

Important

Validate the injection of metadata

Disable the injection of metadata

Troubleshooting

No Kubernetes metadata in the transaction attributes

Compatibility and requirements