Link your applications in Kubernetes

You can surface Kubernetes metadata and link it to your APM agents as transaction traces in order to explore performance issues and troubleshoot application transaction errors. For more information, see this New Relic blog post.

Our Kubernetes metadata injection project is open source. Here's the code to link APM and Infrastructure and the code to automatically manage certificates.

Compatibility and requirements

To be able to link your applications and Kubernetes, your cluster must have the MutatingAdmissionWebhook controller enabled, which requires Kubernetes 1.9 or higher, and may not be enabled by default. To verify that your cluster is compatible, run the following command:

kubectl api-versions | grep admissionregistration.k8s.io/v1beta1 
admissionregistration.k8s.io/v1beta1 

If you see a different result, follow the Kubernetes documentation to enable admission control in your cluster.

The following New Relic agents are able to collect Kubernetes metadata:

To link Openshift and Kubernetes you must enable mutating admission webhooks, which requires Openshift 3.9 or higher.

  1. During the process, install a resource that requires admin permissions to the cluster. Run this to log in as admin:

    oc login -u system:admin
  2. Check that webhooks are correctly configured. If they are not, update the master-config.yaml file.

    Be sure to add the code line kubeConfigFile: /dev/null to address some issues in Openshift.

              admissionConfig:
               pluginConfig:
                 MutatingAdmissionWebhook:
                   configuration:
                     apiVersion:
    apiserver.config.k8s.io/v1alpha1
                     kubeConfigFile: /dev/null
                     kind: WebhookAdmission
                 ValidatingAdmissionWebhook:
                   configuration:
                     apiVersion:
    apiserver.config.k8s.io/v1alpha1
                     kubeConfigFile: /dev/null
                     kind: WebhookAdmission
                     location: ""
  3. Enable certificate signing by editing the provided yaml file and updating your configuration if necessary:

    kubernetesMasterConfig:
       controllerArguments:
          cluster-signing-cert-file:
          - "/etc/origin/master/ca.crt"
          cluster-signing-key-file:
          - "/etc/origin/master/ca.key"
  4. Restart the Openshift services in the master node.

If your application is running with automated certificate management, complete the following steps to make the CA bundle available from inside the Kubernetes cluster:

  1. Follow New Relic's standard procedures to Configure the injection of metadata.
  2. Go to the Amazon EKS page of your cluster and copy the value in the Certificate authority field.
  3. In a terminal, run the following commands:
    $ caBundle=<PASTE_COPIED_CERTIFICATE_AUTHORITY>
    $ kubectl patch mutatingwebhookconfiguration newrelic-metadata-injection-cfg --type='json' -p "[{'op': 'replace', 'path': '/webhooks/0/clientConfig/caBundle', 'value':'${caBundle}'}]"
    

Configure the injection of metadata

By default, all the pods you create will have the correct environment variables set and the metadata injection will apply to the entire cluster. This default configuration also uses the Kubernetes certificates API to automatically manage the certificates required for the injection.

Optional: You can limit the injection of metadata to specific namespaces in your cluster, or self-manage your certificates.

To proceed with the default injection of metadata:

  1. Download the yaml file:

    curl -O http://download.newrelic.com/infrastructure_agent/integrations/kubernetes/k8s-metadata-injection-latest.yaml 
  2. Edit this file, replacing <YOUR_CLUSTER_NAME> with the name of your cluster.

  3. Apply the yaml file to your Kubernetes cluster:

    kubectl apply -f k8s-metadata-injection-latest.yaml 

Activate options

To activate any of the options, follow these instructions:

Restrict metadata injection to specific namespaces

You can limit the injection of metadata only to specific namespaces by using labels.

To enable this feature, edit your yaml file by finding the following lines and replacing the # with an empty space:

  # namespaceSelector:
  #   matchLabels:
  #     newrelic-metadata-injection: enabled

Using this option, the injection is only applied to those namespaces that have the newrelic-metadata-injection label set to enabled:

kubectl label namespace <YOUR_NAMESPACE> newrelic-metadata-injection=enabled
Manage custom certificates

To use custom certificates you need a specific yaml file:

  1. Download the yaml file without the automatic certificate management:

    curl -O http://download.newrelic.com/infrastructure_agent/integrations/kubernetes/k8s-metadata-injection-custom-certs-latest.yaml
  2. Edit this file, replacing <YOUR_CLUSTER_NAME> with the name of your cluster.

  3. Apply the yaml file to your Kubernetes cluster:

    kubectl apply -f k8s-metadata-injection-custom-certs-latest.yaml 

Once you have the correct yaml file, you can proceed with the custom certificate management option. You will need your certificate, server key, and Certification Authority (CA) bundle encoded in PEM format.

  • If you have them in the standard certificate format (X.509), install openssl, and run the following command:

    openssl x509 -in <CERTIFICATE_FILENAME> -outform PEM -out <CERTIFICATE_FILENAME>.pem 
    openssl x509 -in <SERVER_KEY_FILENAME> -outform PEM -out <SERVER_KEY_FILENAME>.pem 
    openssl x509 -in <CA_BUNDLE_FILENAME> -outform PEM -out <BUNDLE_FILENAME>.pem
  • If your certificate/key pair are in another format, see the Digicert knowledgebase for more help.

Create the TLS secret with the signed certificate/key pair, and patch the mutating webhook configuration with the CA using the following commands:

kubectl create secret tls newrelic-metadata-injection-secret \
--key=<PEM_ENCODED_SERVER_KEY> \
--cert=<PEM_ENCODED_CERTIFICATE> \
--dry-run -o yaml |
kubectl -n default apply -f -

caBundle=$(cat <PEM_ENCODED_CA_BUNDLE> | base64 | td -d '\n')
kubectl patch mutatingwebhookconfiguration newrelic-metadata-injection-cfg --type='json' -p "[{'op': 'replace', 'path': '/webhooks/0/clientConfig/caBundle', 'value':'${caBundle}'}]"

Validate the injection of metadata

In order to validate that the webhook (responsible for injecting the metadata) was installed correctly, deploy a new pod and check for the New Relic environment variables.

  1. Create a dummy pod containing Busybox by running:

    kubectl create -f https://git.io/vPieo
  2. Check if New Relic environment variables were injected:

    kubectl exec busybox0 -- env | grep NEW_RELIC_METADATA_KUBERNETES
    
    NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME=fsi
    NEW_RELIC_METADATA_KUBERNETES_NODE_NAME=nodea
    NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME=default
    NEW_RELIC_METADATA_KUBERNETES_POD_NAME=busybox0
    NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME=busybox
    

Certificate rotation

Certificates signed by Kubernetes have an expiration of one year. For more information, see the Kubernetes source code in GitHub.

Troubleshooting

Follow these troubleshooting tips as needed.

No Kubernetes metadata in APM or distributed tracing transactions

Problem

There is no Kubernetes metadata included in the transactions' attributes of your APM agent or in distributed tracing.

Solution

  1. Verify that the environment variables are being correctly injected by following the instructions described in the Validate your installation step.
  2. If they are not present, get the name of the metadata injection pod by running:

    kubectl get pods | grep newrelic-metadata-injection-deployment
    kubectl logs -f pod/podname
  3. In another terminal, create a new pod (for example, see Validate your installation), and inspect the logs of the metadata injection deployment for errors.

  4. Ensure the metadata injection setup job ran successfully by inspecting the output of:

    kubectl get job newrelic-metadata-setup
  5. If the job is not completed, investigate the logs of the setup job:

    kubectl logs job/newrelic-metadata-setup
  6. Ensure the CertificateSigningRequest is approved and issued by running:

    kubectl get csr newrelic-metadata-injection-svc.default
  7. Ensure the TLS secret is present by running:

    kubectl get secret newrelic-metadata-injection-secret
  8. Ensure the CA bundle is present in the mutating webhook configuration:

    kubectl get mutatingwebhookconfiguration newrelic-metadata-injection-cfg -o json
  9. Ensure the TargetPort of the Service resource matches the Port of the Deployment's container:
    kubectl describe service/newrelic-metadata-injection-svc
    kubectl describe deployment/newrelic-metadata-injection-deployment

Uninstall

To uninstall the injection of metadata, use the following commands:

  1. Delete the Kubernetes objects using the yaml file:

    kubectl delete -f k8s-metadata-injection-latest.yaml 
  2. Delete the TLS secret containing the certificate/key pair:

    kubectl delete secret/newrelic-metadata-injection-secret

For more help