Link your applications in Kubernetes

BETA

You can surface Kubernetes metadata and link it to your APM agents as transaction traces in order to explore performance issues and troubleshoot application transaction errors. For more information, see this blog post.

Our Kubernetes metadata injection project is open source. Here's the code to link APM and Infrastructure and the code to automatically manage certificates.

This document covers:

  • Compatibility and requirements
  • Configuration to report Kubernetes metadata to APM agents
  • Troubleshooting
  • Uninstall instructions

Compatibility and requirements

To be able to link your applications and Kubernetes, your cluster needs to have the MutatingAdmissionWebhook controller enabled, which requires Kubernetes 1.9 or higher, and might not be enabled by default. Verify that your cluster is compatible by running the following command:

kubectl api-versions | grep admissionregistration.k8s.io/v1beta1 

admissionregistration.k8s.io/v1beta1 

If you see a different result, follow the Kubernetes documentation to learn how to enable admission control in your cluster.

The following New Relic agents are able to collect Kubernetes metadata:

To link Openshift and Kubernetes you need to enable mutating admission webhooks, which requires Openshift 3.9 and higher.

  1. During the process you will install a resource that requires admin permissions to the cluster. Run this to login as admin:
    oc login -u system:admin
  2. Check that webhooks are correctly configured. If they are not, update the master-config.yaml file:
              admissionConfig:
               pluginConfig:
                 MutatingAdmissionWebhook:
                   configuration:
                     apiVersion:
    apiserver.config.k8s.io/v1alpha1
                     kubeConfigFile: /dev/null
                     kind: WebhookAdmission
                 ValidatingAdmissionWebhook:
                   configuration:
                     apiVersion:
    apiserver.config.k8s.io/v1alpha1
                     kubeConfigFile: /dev/null
                     kind: WebhookAdmission
                     location: ""

    The code line kubeConfigFile: /dev/null needs to be added to address some issues in Openshift.

  3. Enable certificate signing by editing the provided yaml file and updating your configuration if necessary:
    kubernetesMasterConfig:
       controllerArguments:
          cluster-signing-cert-file:
          - "/etc/origin/master/ca.crt"
          cluster-signing-key-file:
          - "/etc/origin/master/ca.key"
  4. Restart the Openshift services in the master node.

Configure the injection of metadata

By default, all the pods you create will have the correct environment variables set and the metadata injection will apply to the entire cluster. This default configuration also uses the Kubernetes certificates API to automatically manage the certificates required for the injection.

Optionally, you can limit the injection of metadata to specific namespaces in your cluster, or self-manage your certificates.

To proceed with the default injection of metadata:

  1. Download the yaml file:
    curl -O http://download.newrelic.com/infrastructure_agent/integrations/kubernetes/k8s-metadata-injection-latest.yaml 
  2. Edit this file, replacing <YOUR_CLUSTER_NAME> with the name of your cluster.
  3. Apply the yaml file to your Kubernetes cluster:
    kubectl apply -f k8s-metadata-injection-latest.yaml 

To activate any of the options, follow these instructions:

Restrict metadata injection to specific namespaces

You can limit the injection of metadata only to specific namespaces by using labels.

To enable this feature, edit your yaml file by finding the following lines and replacing the # with an empty space:

  # namespaceSelector:
  #   matchLabels:
  #     newrelic-metadata-injection: enabled

Using this option, the injection is only applied to those namespaces that have the newrelic-metadata-injection label set to enabled:

kubectl label ns namespace <YOUR_NAMESPACE> newrelic-metadata-injection=enabled
Manage custom certificates

To use the custom certificate management option, you will need your certificate, server key, and Certification Authority (CA) bundle encoded in PEM format.

  • If you have them in the standard certificate format (X.509), install openssl and run the following command:
openssl x509 -in <CERTIFICATE_FILENAME> -outform PEM -out <CERTIFICATE_FILENAME>.pem 
openssl x509 -in <SERVER_KEY_FILENAME> -outform PEM -out <SERVER_KEY_FILENAME>.pem 
openssl x509 -in <CA_BUNDLE_FILENAME> -outform PEM -out <BUNDLE_FILENAME>.pem

Create the TLS secret with the signed certificate/key pair, and patch the mutating webhook configuration with the CA using the commands below:

kubectl create secret tls newrelic-metadata-injection-secret \
--key=<PEM_ENCODED_SERVER_KEY> \
--cert=<PEM_ENCODED_CERTIFICATE> \
--dry-run -o yaml |
kubectl -n default apply -f -

caBundle=$(cat <PEM_ENCODED_CA_BUNDLE> | base64 | td -d '\n')
kubectl patch mutatingwebhookconfiguration newrelic-metadata-injection-cfg --type='json' -p "[{'op': 'replace', 'path': '/webhooks/0/clientConfig/caBundle', 'value':'$

{caBundle}

'}]"

Validate the injection of metadata

In order to validate that the webhook (responsible for injecting the metadata) was installed correctly, you can deploy a new pod and check for the New Relic environment variables.

  1. Create a dummy pod containing Busybox by running:
    kubectl create -f https://git.io/vPieo
  2. Check if New Relic environment variables were injected:
    kubectl exec busybox0 -- env | grep NEW_RELIC_METADATA_KUBERNETES
    
    NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME=fsi
    NEW_RELIC_METADATA_KUBERNETES_NODE_NAME=nodea
    NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME=default
    NEW_RELIC_METADATA_KUBERNETES_POD_NAME=busybox0
    NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME=busybox
    

Certificate rotation

Certificates signed by Kubernetes have an expiration of one year - for more information, see the Kubernetes source code.

Troubleshooting

No Kubernetes metadata in APM or distributed tracing transactions

Problem

There is no Kubernetes metadata included in the transactions' attributes of your APM agent or in distributed tracing.

Solution

  1. Verify that the environment variables are being correctly injected by following the instructions described in the Validate your installation step.
  2. If they are not present, get the name of the metadata injection pod by running
    kubectl get pods | grep newrelic-metadata-injection-deployment
    kubectl logs -f pod/
    
  3. In another terminal, create a new pod (you can use the example provided in the Validate your installation step) and inspect the logs of the metadata injection deployment for errors.
  4. Ensure the metadata injection setup job ran successfully by inspecting the output of
    kubectl get job newrelic-metadata-setup
  5. In case the job is not completed, investigate the logs of the setup job:
    kubectl logs job/newrelic-metadata-setup
  6. Ensure the CertificateSigningRequest is approved and issued by running
    kubectl get csr newrelic-metadata-injection-svc.default
  7. Ensure the TLS secret is present by running
    kubectl get secret newrelic-metadata-injection-secret

Uninstall

To uninstall the injection of metadata, use the following commands:

  1. Delete the kubernetes objects using the yaml file:
    kubectl delete -f k8s-metadata-injection-latest.yaml 
  2. Delete the TLS secret containing the certificate/key pair:
    kubectl delete secret/newrelic-metadata-injection-secret

For more help