• EnglishEspañol日本語한국어Português
  • Log inStart now

Link APM-instrumented applications to Kubernetes

You can surface Kubernetes metadata and link it to your APM agents as distributed traces to explore performance issues and troubleshoot transaction errors. For more information, see this blog post about monitoring app performance via Kubernetes.

The metadata injection product uses a MutatingAdmissionWebhook to add the following environment variables to pods:

NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME
NEW_RELIC_METADATA_KUBERNETES_NODE_NAME
NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME
NEW_RELIC_METADATA_KUBERNETES_DEPLOYMENT_NAME
NEW_RELIC_METADATA_KUBERNETES_POD_NAME
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_IMAGE_NAME

Tip

Our Kubernetes metadata injection project is open source. Here's the code to link APM and infrastructure data.

Compatibility and requirements

To link your applications and Kubernetes, you must be able to deploy MutatingWebhookConfigurations to your Kubernetes cluster.

To verify that you have the required permissions, you can run the following command:

bash
$
kubectl auth can-i create mutatingwebhookconfigurations.admissionregistration.k8s.io -A

The output for the command above should be something similar to:

yes

If you see a different result, follow the Kubernetes documentation to enable admission control in your cluster.

Network requirements

For Kubernetes to speak to our MutatingAdmissionWebhook, the master node (or the API server container, depending on how the cluster is set up) should be allowed egress for HTTPS traffic on port 443 to pods in all of the other nodes in the cluster.

This might require specific configuration depending on how the infrastructure is set up (on-premises, AWS, Google Cloud, etc).

APM agent compatibility

The following New Relic agents collect Kubernetes metadata:

Set up the injection of metadata

Injection of metadata is included when you install our integration using Helm. Just be sure that, when you are configuring the nri-bundle chart, the webhook that injects the metadata is enabled:

nri-metadata-injection:
enabled: true

Your application pods will need to be restarted after the webhook is deployed, so that they can pick up the necessary environment variables.

By default, all the pods you create that include APM agents have the correct environment variables set and the metadata injection applies to the entire cluster. To check that the environment variables have been set, any container that is running must be stopped, and a new instance started (see Validate the injection of metadata).

This default configuration also uses the Kubernetes certificates API to automatically manage the certificates required for the injection. If needed, you can limit the injection of metadata to specific namespaces in your cluster or self-manage your certificates.

Custom configuration

Limit namespaces subject to injection

You can limit the injection of metadata only to specific namespaces by using labels.

To enable this feature, add the following to values-newrelic.yaml file:

nri-metadata-injection:
injectOnlyLabeledNamespaces: true

With this option, injection is only applied to those namespaces that have the newrelic-metadata-injection label set to enabled:

bash
$
kubectl label namespace YOUR_NAMESPACE newrelic-metadata-injection=enabled

Use cert-manager to generate certs

By default, our chart uses kube-webhook-certgen to automatically generate the required certificates for the webhook to run.

However, if you have cert-manager installed, you can configure our chart to use cert-manager instead, which can significantly simplify the deployment process:

nri-metadata-injection:
certManager:
enabled: true

Manage custom certificates

Tip

Manually managing webhook certificates is recommended for advanced users only. New Relic support team might not be able to help troubleshooting this configuration.

To use custom certificates you need to disable the automatic installation of certificates when you are installing using Helm.

To disable the installation for certificates just modify nri-bundle Helm values.yaml like this:

nri-metadata-injection:
customTLSCertificate: true

Now you can proceed with the custom certificate management option.

You need your certificate, server key, and Certification Authority (CA) bundle encoded in PEM format.

  • If you have them in the standard certificate format (X.509), install openssl, and run the following:
bash
$
openssl x509 -in YOUR_CERTIFICATE_FILENAME -outform PEM -out YOUR_CERTIFICATE_FILENAME.pem
$
openssl x509 -in YOUR_SERVER_KEY_FILENAME -outform PEM -out YOUR_SERVER_KEY_FILENAME.pem
$
openssl x509 -in YOUR_CA_BUNDLE_FILENAME -outform PEM -out YOUR_BUNDLE_FILENAME.pem

Create the TLS secret with the signed certificate/key pair, and patch the mutating webhook configuration with the CA using the following commands:

bash
$
kubectl create secret tls YOUR_NEWRELIC_METADATA_INJECTION_ADMISSION \
>
--key=YOUR_PEM_ENCODED_SERVER_KEY \
>
--cert=YOUR_PEM_ENCODED_CERTIFICATE \
>
--dry-run -o yaml |
$
kubectl -n newrelic apply -f -
$
$
caBundle=$(cat YOUR_PEM_ENCODED_CA_BUNDLE | base64 | td -d $'\n')
$
kubectl patch mutatingwebhookconfiguration newrelic-metadata-injection-cfg --type='json' -p "[{'op': 'replace', 'path': '/webhooks/0/clientConfig/caBundle', 'value':'${caBundle}'}]"

Important

Certificates signed by Kubernetes have an expiration of one year. For more information, see the Kubernetes source code in GitHub.

Validate the injection of metadata

In order to validate that the webhook (responsible for injecting the metadata) was installed correctly, deploy a new pod and check for the New Relic environment variables.

  1. Create a dummy nginx pod by running:
bash
$
kubectl run test-nginx --image nginx -n newrelic
  1. Check if New Relic environment variables were injected:
bash
$
kubectl exec -n newrelic test-nginx -- env | grep NEW_RELIC_METADATA_KUBERNETES

The expected output would be something like the following:

NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME=THE_CLUSTER_NAME
NEW_RELIC_METADATA_KUBERNETES_NODE_NAME=nodea
NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME=newrelic
NEW_RELIC_METADATA_KUBERNETES_POD_NAME=test-nginx
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME=nginx

Disable the injection of metadata

To disable/uninstall the injection of metadata, change your values-newrelic.yaml file as follows:

webhook:
enabled: false

And re-run the installation command.

Troubleshooting

Follow these troubleshooting tips as needed.

Copyright © 2024 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.