Kubernetes logs

For the Kubernetes integration debug level logs are not produced. If you are doing a more in-depth investigation on your own or with New Relic Support, you can enable verbose mode.

• To get verbose logging details using Helm and nri-bundle chart it is enough to add to the values.yaml newrelic-infrastructure.verboseLog. Notice that you can enable them in all subcharts setting global.verboseLog.

  # To enable verboseLog for newrelic-infrastructure only
  # newrelic-infrastructure:
  #   verboseLog: true
  
  # To enable verboseLog for all sub-charts
  # global:
  #   verboseLog: true

  Copy

  Once you have edited the files, upgrade the solution with the following command:

  bash

  Copy

  $
  helm upgrade <RELEASE_NAME> newrelic/nri-bundle \
  >
  --namespace <NEWRELIC_NAMESPACE> \
  >
  -f values-newrelic.yaml \
  >
  [--version fixed-chart-version]

  Then wait some time, collect the logs, revert the change in the values.yaml file, and upgrade again.

• To get verbose logging details for an integration manually using a manifest file you need to set two different environment variables:

  1. NRIA_VERBOSE="1" for all agent and forwarder containers. For example below the NRIA_LICENSE_KEY environment variable. This environment variable enables agent verbose logs.
  2. set NRI_KUBERNETES_VERBOSE=true" in all components of the integration. This environment variable enables verbose logs of the integration.

  Once you have edited the manifest upgrade the solution with the following command. Then wait some time, collect the logs, revert the change in the manifest and apply the manifest again.

  bash

  Copy

  $
  kubectl apply -f your_newrelic_k8s.yaml -n <NEWRELIC_NAMESPACE>

There are three different components of the integration in charge of scraping ksm, controlplane and kubelet. In in each instance two containers are running, one scraping the data and one forwarding it. The agent in the kubelet component is in charge also to scrape node data and run integrations.

To fetch logs, get the name of the pods you want to retrieve logs from:

$
kubectl get pods -n <NEWRELIC_NAMESPACE>

Retrieve the logs:

$
kubectl logs <POD_NAME> --all-containers --prefix

If you interested into the logs of the previous execution add the --previous flag. If you are interested in the logs of just one of the containers you can avoid --all-containers --prefix flags and specify the container with the --container option.

For the Kubernetes integration, the infrastructure agent is distributed as a Docker image that contains the infrastructure agent and the Kubernetes integration. The Docker image is tagged with a version, and the infrastructure agent also has its own version.

When the agent is successfully sending information to New Relic, you can retrieve the versions of the infrastructure agent for Kubernetes (the Docker image) you are running in your clusters by using the following NRQL query:

FROM K8sContainerSample SELECT uniqueCount(entityId) WHERE containerName LIKE 'agent' FACET clusterName, containerImage

If the agent isn't reporting any data:

Get the versions of the New Relic integration for Kubernetes that you are running in a cluster using kubectl:

$
kubectl get pods --all-namespaces -l app.kubernetes.io/name=newrelic-infrastructure -o jsonpath="{.items..spec..containers..image}"

To retrieve the version of kube-state-metrics running on your clusters, run the following NRQL query:

FROM K8sContainerSample SELECT uniqueCount(entityId) WHERE containerName LIKE '%kube-state-metrics%' facet clusterName, containerImage

For the Kubernetes integration, the infrastructure agent adds a log entry only in the event of an error. Most common errors are displayed in the standard (non-verbose) logs. If you are doing a more in-depth investigation on your own or with New Relic Support, you can enable verbose mode.

To get verbose logging details for an integration using a manifest file:

1. Enable verbose logging: In the deployment file, set the value of NRIA_VERBOSE to 1.

2. Apply the modified configuration by running:

   bash

   Copy

   $
   kubectl apply -f your_newrelic_k8s.yaml

3. Leave on verbose mode for a few minutes, or until you feel enough activity has occurred.

4. Disable verbose mode: Set the NRIA_VERBOSE value back to 0.

5. Apply the restored configuration by running:

   bash

   Copy

   $
   kubectl apply -f your_newrelic_k8s.yaml

6. Get a list of nodes in the environment:

   bash

   Copy

   $
   kubectl get nodes --all-namespaces

7. Get a list of infrastructure and kube-state-metrics pods:

   bash

   Copy

   $
   kubectl get pods --all-namespaces -o wide | egrep 'newrelic|kube-state-metrics'

8. Get logs from the pods connecting to kube-state-metrics.

9. Retrieve kube-state-metrics service configuration.

For the Kubernetes integration, the infrastructure agent adds a log entry only in the event of an error. Most common errors are displayed in the standard (non-verbose) logs. If you're doing a more in-depth investigation on your own or with New Relic Support, you can enable verbose mode.

To get verbose logging details for an integration using Helm:

1. Enable verbose logging:
   bash

   Copy

   $
   helm upgrade -n <namespace> --reuse-values newrelic-bundle --set newrelic-infrastructure.verboseLog=true newrelic/nri-bundle

2. Leave on verbose mode for a few minutes, or until enough activity has occurred.
3. When you have the information you need, disable verbose logging:
   bash

   Copy

   $
   helm upgrade --reuse-values newrelic-bundle --set newrelic-infrastructure.verboseLog=false newrelic/nri-bundle

4. Follow the steps to restore your configuration from step 5 in the section, Get verbose logs for installations using a manifest file.

To get the logs from pods connecting to kube-state-metrics:

1. Get the nodes that kube-state-metrics is running on:

   bash

   Copy

   $
   kubectl get pods --all-namespaces -o wide | grep kube-state-metrics

   Look for output similar to this:

   bash

   Copy

   $
   kube-system   kube-state-metrics-5c6f5cb9b5-pclhh     2/2
   $
   Running   4          4d        172.17.0.3   minikube

2. Get the New Relic pods that are running on the same node as kube-state-metrics:

   bash

   Copy

   $
   kubectl describe node minikube | grep newrelic-infra

   Look for output similar to this:

   bash

   Copy

   $
   default                    newrelic-infra-5wcv6                     100m (5%)
   $
   0 (0%)      100Mi (5%)       100Mi (5%)

3. Retrieve the logs for the nodes by running:

   bash

   Copy

   $
   kubectl logs newrelic-infra-5wcv6

To get the logs from a pod running on a master node:

1. Get the nodes that are labelled as master:

   bash

   Copy

   $
   kubectl get nodes -l node-role.kubernetes.io/master=""

   Or,

   bash

   Copy

   $
   kubectl get nodes -l kubernetes.io/role="master"

   Look for output similar to this:

   bash

   Copy

   $
   NAME                         STATUS  ROLES   AGE   VERSION
   $
   ip-10-42-24-4.ec2.internal   Ready   master  42d   v1.14.8

2. Get the New Relic pods that are running on one of the nodes returned in the previous step:

   bash

   Copy

   $
   kubectl get pods --field-selector spec.nodeName=ip-10-42-24-4.ec2.internal -l name=newrelic-infra --all-namespaces

   Look for output similar to this:

   bash

   Copy

   $
   newrelic-infra-whvzt

3. Retrieve the logs for the nodes by running:

   bash

   Copy

   $
   kubectl logs newrelic-infra-whvzt