• Log inFree account

Link APM-instrumented applications to Kubernetes

You can surface Kubernetes metadata and link it to your APM agents as distributed traces to explore performance issues and troubleshoot transaction errors. For more information, see this blog post about monitoring app performance via Kubernetes.

Additionally, you can quickly start monitoring Kubernetes clusters using our Auto-telemetry with Pixie, which is currently a beta release. This Pixie integration into New Relic does not require a language agent. Learn more about Auto-telemetry with Pixie here.

The metadata injection product uses a MutatingAdmissionWebhook to add the following environment variables to pods:

NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME
NEW_RELIC_METADATA_KUBERNETES_NODE_NAME
NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME
NEW_RELIC_METADATA_KUBERNETES_DEPLOYMENT_NAME
NEW_RELIC_METADATA_KUBERNETES_POD_NAME
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_IMAGE_NAME

Tip

Our Kubernetes metadata injection project is open source. Here's the code to link APM and infrastructure data.

Compatibility and requirements

To link your applications and Kubernetes, you must be able to deploy MutatingWebhookConfigurations to your Kubernetes cluster.

To verify that you have the required permissions, you can run the following command:

bash
$
kubectl auth can-i create mutatingwebhookconfigurations.admissionregistration.k8s.io -A

The output for the command above should be something similar to:

yes

If you see a different result, follow the Kubernetes documentation to enable admission control in your cluster.

Network requirements

For Kubernetes to speak to our MutatingAdmissionWebhook, the master node (or the API server container, depending on how the cluster is set up) should be allowed egress for HTTPS traffic on port 443 to pods in all of the other nodes in the cluster.

This might require specific configuration depending on how the infrastructure is set up (on-premises, AWS, Google Cloud, etc).

APM agent compatibility

The following New Relic agents collect Kubernetes metadata:

Openshift requirements

To link Openshift and Kubernetes you must enable mutating admission webhooks, which requires Openshift 3.9 or higher.

  1. During the process, install a resource that requires admin permissions to the cluster. Run this to log in as admin:

    bash
    $
    oc login -u system:admin
  2. Check that webhooks are correctly configured. If they are not, update the master-config.yaml file.

    admissionConfig:
    pluginConfig:
    MutatingAdmissionWebhook:
    configuration:
    apiVersion: apiserver.config.k8s.io/v1alpha1
    kubeConfigFile: /dev/null
    kind: WebhookAdmission
    ValidatingAdmissionWebhook:
    configuration:
    apiVersion: apiserver.config.k8s.io/v1alpha1
    kubeConfigFile: /dev/null
    kind: WebhookAdmission
    location: ""

    Important

    Add kubeConfigFile: /dev/null to address some issues in Openshift.

  3. Enable certificate signing by editing the YAML file and updating your configuration:

    kubernetesMasterConfig:
    controllerArguments:
    cluster-signing-cert-file:
    - "/etc/origin/master/ca.crt"
    cluster-signing-key-file:
    - "/etc/origin/master/ca.key"
  4. Restart the Openshift services in the master node.

Important

Openshift 4.0 or higher requires Dynamic Admission Control to enable the admission webhooks, since the master-config.yaml file isn't supported in those versions.

Set up the injection of metadata

Injection of metadata is included when you install our integration using Helm. Just be sure that, when you are configuring the chart, the webhook that inject the metadata is enabled:

webhook:
enabled: true

By default, all the pods you create that include APM agents have the correct environment variables set and the metadata injection applies to the entire cluster. To check that the environment variables have been set, any container that is running must be stopped, and a new instance started (see Validate the injection of metadata).

This default configuration also uses the Kubernetes certificates API to automatically manage the certificates required for the injection. If needed, you can limit the injection of metadata to specific namespaces in your cluster or self-manage your certificates.

Custom configuration

Limit namespaces subject to injection

You can limit the injection of metadata only to specific namespaces by using labels.

To enable this feature, add the following to values-newrelic.yaml file:

nri-metadata-injection:
injectOnlyLabeledNamespaces: true

With this option, injection is only applied to those namespaces that have the newrelic-metadata-injection label set to enabled:

bash
$
kubectl label namespace YOUR_NAMESPACE newrelic-metadata-injection=enabled

Use cert-manager to generate certs

By default, our chart uses kube-webhook-certgen to automatically generate the required certificates for the webhook to run.

However, if you have cert-manager installed, you can configure our chart to use cert-manager instead, which can significantly simplify the deployment process:

nri-metadata-injection:
certManager:
enabled: true

Manage custom certificates

Tip

Manually managing webhook certificates is recommended for advanced users only. New Relic support team might not be able to help troubleshooting this configuration.

To use custom certificates you need to disable the automatic installation of certificates when you are installing using Helm.

To disable the installation for certificates just modify nri-bundle Helm values.yaml like this:

nri-metadata-injection:
customTLSCertificate: true

Now you can proceed with the custom certificate management option.

You need your certificate, server key, and Certification Authority (CA) bundle encoded in PEM format.

  • If you have them in the standard certificate format (X.509), install openssl, and run the following:
openssl x509 -in CERTIFICATE_FILENAME -outform PEM -out CERTIFICATE_FILENAME.pem
openssl x509 -in SERVER_KEY_FILENAME -outform PEM -out SERVER_KEY_FILENAME.pem
openssl x509 -in CA_BUNDLE_FILENAME -outform PEM -out BUNDLE_FILENAME.pem

Create the TLS secret with the signed certificate/key pair, and patch the mutating webhook configuration with the CA using the following commands:

kubectl create secret tls newrelic-metadata-injection-admission \
--key=PEM_ENCODED_SERVER_KEY \
--cert=PEM_ENCODED_CERTIFICATE \
--dry-run -o yaml |
kubectl -n newrelic apply -f -

caBundle=$(cat PEM_ENCODED_CA_BUNDLE | base64 | td -d $'\n')
kubectl patch mutatingwebhookconfiguration newrelic-metadata-injection-cfg --type='json' -p "[{'op': 'replace', 'path': '/webhooks/0/clientConfig/caBundle', 'value':'${caBundle}'}]"

Important

Certificates signed by Kubernetes have an expiration of one year. For more information, see the Kubernetes source code in GitHub.

Validate the injection of metadata

In order to validate that the webhook (responsible for injecting the metadata) was installed correctly, deploy a new pod and check for the New Relic environment variables.

  1. Create a dummy nginx pod by running:
bash
$
kubectl run test-nginx --image nginx -n newrelic
  1. Check if New Relic environment variables were injected:
bash
$
kubectl exec -n newrelic test-nginx -- env | grep NEW_RELIC_METADATA_KUBERNETES

The expected output would be something like the following:

NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME=cluster-name
NEW_RELIC_METADATA_KUBERNETES_NODE_NAME=nodea
NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME=newrelic
NEW_RELIC_METADATA_KUBERNETES_POD_NAME=test-nginx
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME=nginx

Disable the injection of metadata

To disable/uninstall the injection of metadata, change your values-newrelic.yaml file as follows:

webhook:
enabled: false

And re-run the installation command.

Troubleshooting

Follow these troubleshooting tips as needed.

Copyright © 2022 New Relic Inc.