Kubernetes logs

The Kubernetes integration does not produce debug level logs. If you are doing a more in-depth investigation on your own or with New Relic Support, you can enable verbose mode.

• To get verbose logging details using Helm and the nri-bundle chart, add the following configuration to your values.yaml file. You can also enable verbose logging for all subcharts:

  # To enable verboseLog for newrelic-infrastructure only
  # newrelic-infrastructure:
  #   verboseLog: true
  
  # To enable verboseLog for all sub-charts
  # global:
  #   verboseLog: true

  Copy

  Once you have added the desired configuration, upgrade the solution with the following command:

  bash

  Copy

  $
  helm upgrade <RELEASE_NAME> newrelic/nri-bundle \
  >
  --namespace <NEWRELIC_NAMESPACE> \
  >
  -f values-newrelic.yaml \
  >
  [--version fixed-chart-version]

  After collecting your verbose logs, revert the change in your values.yaml file, and run the upgrade command again.

• To get verbose logging details using a manifest file, you need to set two different environment variables:

  1. NRIA_VERBOSE="1" for all agent and forwarder containers; this enables agent verbose logs. You can set this below NRIA_LICENSE_KEY.

  2. set NRI_KUBERNETES_VERBOSE=true" in all components of the integration; this enables verbose logs for the integration.

     env:

     Copy
  • name: NRIA_LICENSE_KEY valueFrom: secretKeyRef: key: licenseKey name: newrelic-bundle-newrelic-infrastructure-license
  • name: NRIA_VERBOSE value: "1"

    Copy

  Once you have edited the manifest, upgrade the solution with the following command:

  bash

  Copy

  $
  kubectl apply -f your_newrelic_k8s.yaml -n <NEWRELIC_NAMESPACE>

  After collecting your verbose logs, revert the change in the manifest, and apply the manifest again.

There are three different components of the integration that are in charge of scraping ksm, controlplane, and kubelet. In each instance, there are two containers running, with one scraping the data and the other one forwarding it. The agent in the kubelet component is also in charge of scraping node data and running integrations. The diagram below shows the integration installed in a cluster with three nodes.

To fetch logs, get the name of the pods you want to retrieve logs from:

$
kubectl get pods -n <NEWRELIC_NAMESPACE>

Retrieve the logs:

$
kubectl logs <POD_NAME> --all-containers --prefix

To retrieve logs from the previous execution, add the --previous flag. To get logs from only one of the containers, remove the --all-containers --prefix flag and specify the container with the --container option.

When you use the Kubernetes integration, the infrastructure agent is distributed as a Docker image that contains both the infrastructure agent and the Kubernetes integration. The Docker image is tagged with a version, and the infrastructure agent also has its own version.

Once the agent is sending data to New Relic, you can retrieve the versions of the infrastructure agent for Kubernetes (the Docker image) you are running in your clusters by using the following NRQL query:

FROM K8sContainerSample SELECT uniqueCount(entityId) WHERE containerName LIKE 'agent' FACET clusterName, containerImage

If the agent isn't reporting any data:

Get the versions of the New Relic integration for Kubernetes that you are running in a cluster using kubectl:

$
kubectl get pods --all-namespaces -l app.kubernetes.io/name=newrelic-infrastructure -o jsonpath="{.items..spec..containers..image}"

To retrieve the version of kube-state-metrics running on your clusters, run the following NRQL query:

FROM K8sContainerSample SELECT uniqueCount(entityId) WHERE containerName LIKE '%kube-state-metrics%' facet clusterName, containerImage

For the Kubernetes integration, the infrastructure agent only adds a log entry in the event of an error. Most common errors are displayed in the standard (non-verbose) logs. If you are doing a more in-depth investigation on your own or with New Relic Support, you can enable verbose mode.

To enable verbose logging using a manifest file:

1. Enable verbose logging: In the deployment file, set the value of NRIA_VERBOSE to 1.

2. Apply the modified configuration by running:

   bash

   Copy

   $
   kubectl apply -f your_newrelic_k8s.yaml

3. Leave on verbose mode for a few minutes, or until enough activity has occurred.

4. Disable verbose mode: Set the NRIA_VERBOSE value back to 0.

5. Apply the restored configuration by running:

   bash

   Copy

   $
   kubectl apply -f your_newrelic_k8s.yaml

6. Get a list of nodes in the environment:

   bash

   Copy

   $
   kubectl get nodes --all-namespaces

7. Get a list of infrastructure and kube-state-metrics pods:

   bash

   Copy

   $
   kubectl get pods --all-namespaces -o wide | egrep 'newrelic|kube-state-metrics'

8. Get logs from the pods connecting to kube-state-metrics.

9. Retrieve kube-state-metrics service configuration.

For the Kubernetes integration, the infrastructure agent only adds a log entry in the event of an error. Most common errors are displayed in the standard (non-verbose) logs. If you are doing a more in-depth investigation on your own or with New Relic Support, you can enable verbose mode.

To enable verbose logging using Helm:

1. Run the following command:

   bash

   Copy

   $
   helm upgrade -n <namespace> --reuse-values newrelic-bundle --set newrelic-infrastructure.verboseLog=true newrelic/nri-bundle

2. Leave on verbose mode for a few minutes, or until enough activity has occurred.

3. When you have the information you need, disable verbose logging:

   bash

   Copy

   $
   helm upgrade --reuse-values newrelic-bundle --set newrelic-infrastructure.verboseLog=false newrelic/nri-bundle

4. Apply the restored configuration by running:

   bash

   Copy

   $
   kubectl apply -f your_newrelic_k8s.yaml

5. Get a list of nodes in the environment:

   bash

   Copy

   $
   kubectl get nodes --all-namespaces

6. Get a list of infrastructure and kube-state-metrics pods:

   bash

   Copy

   $
   kubectl get pods --all-namespaces -o wide | egrep 'newrelic|kube-state-metrics'

7. Get logs from the pods connecting to kube-state-metrics.

8. Retrieve kube-state-metrics service configuration.

To get the logs from pods connecting to kube-state-metrics:

1. Run the following command:

   bash

   Copy

   $
   kubectl get pods --all-namespaces -o wide | grep kube-state-metrics

   Look for output similar to this:

   bash

   Copy

   $
   kube-system   kube-state-metrics-5c6f5cb9b5-pclhh     2/2
   $
   Running   4          4d        172.17.0.3   minikube

2. Get the New Relic pods that are running on the same node as kube-state-metrics:

   bash

   Copy

   $
   kubectl describe node minikube | grep newrelic-infra

   Look for output similar to this:

   bash

   Copy

   $
   default                    newrelic-infra-5wcv6                     100m (5%)
   $
   0 (0%)      100Mi (5%)       100Mi (5%)

3. Retrieve the logs for the nodes by running:

   bash

   Copy

   $
   kubectl logs newrelic-infra-5wcv6

To get the logs from a pod running on a master node:

1. Get the nodes that are labelled as master:

   bash

   Copy

   $
   kubectl get nodes -l node-role.kubernetes.io/master=""

   Or,

   bash

   Copy

   $
   kubectl get nodes -l kubernetes.io/role="master"

   Look for output similar to this:

   bash

   Copy

   $
   NAME                         STATUS  ROLES   AGE   VERSION
   $
   ip-10-42-24-4.ec2.internal   Ready   master  42d   v1.14.8

2. Get the New Relic pods that are running on one of the nodes returned in the previous step:

   bash

   Copy

   $
   kubectl get pods --field-selector spec.nodeName=ip-10-42-24-4.ec2.internal -l name=newrelic-infra --all-namespaces

   Look for output similar to this:

   bash

   Copy

   $
   newrelic-infra-whvzt

3. Retrieve the logs for the nodes by running:

   bash

   Copy

   $
   kubectl logs newrelic-infra-whvzt