Get logs and version

To generate verbose logs and get version and configuration information, follow these procedures.

Get verbose logs

For the Kubernetes integration, the Infrastructure agent adds a log entry only in the event of an error. Most common errors are displayed in the standard (non-verbose) logs. If you are doing a more in-depth investigation on your own or with New Relic Support, you can enable verbose mode.

Verbose mode will significantly increase the amount of information that is sent to log files. Temporarily enable this mode only for troubleshooting purposes, and reset the log level when finished.

To get verbose logging details:

  1. Enable verbose logging: In the deployment file, set the value of NRIA_VERBOSE to 1.

  2. Apply the modified configuration by running:

    kubectl apply -f your_newrelic_k8s.yaml
  3. Leave on verbose mode for a few minutes, or until you feel enough activity has occurred.
  4. Disable verbose mode: Set the NRIA_VERBOSE value back to 0.

  5. Apply the restored configuration by running:

    kubectl apply -f your_newrelic_k8s.yaml

  6. Get a list of nodes in the environment:

    kubectl get nodes --all-namespaces

  7. Get a list of Infrastructure and kube-state-metrics pods:

    kubectl get pods --all-namespaces -o wide | egrep 'newrelic|kube-state-metrics'
  8. Get logs from the pod connecting to kube-state-metrics.
  9. Retrieve kube-state-metrics service configuration.
Get Infrastructure version

For the Kubernetes integration, the Infrastructure agent is distributed as a Docker image that contains the Infrastructure agent and the Kubernetes integration. The Docker image is tagged with a version, and the Infrastructure agent also has its own version.

When the agent is successfully sending information to New Relic, you can retrieve the versions of the Infrastructure agent for Kubernetes (the Docker image) you are running in your clusters by using the following Insights query:

FROM K8sContainerSample SELECT uniqueCount(entityId) WHERE containerName = 'newrelic-infra' facet clusterName, containerImage
Kubernetes integration Infrastructure agent version
insights.newrelic.com > Query > (create your query): In this result you can see that cluster aws-rbac-cluster is running version 1.0.0-beta4, and the other two clusters are running 1.0.0-beta4-2.

If the agent is not reporting any data:

  1. Get the version(s) of the New Relic integration for Kubernetes that you are running in a cluster using kubectl:

    kubectl get pods --all-namespaces -l name=newrelic-infra -o jsonpath="{.items..spec..containers..image}"

  2. Look for output similar to this:

    newrelic/infrastructure-k8s:1.0.0
Get kube-state-metrics version

To retrieve the version of kube-state-metrics running on your clusters, run the following Insights NRQL query:

FROM K8sContainerSample SELECT uniqueCount(entityId) WHERE containerName LIKE '%kube-state-metrics%' facet clusterName, containerImage
Kube-state-metrics version
insights.newrelic.com > Query > (create your query): This example shows that the aws-rbac-cluster is running version 1.1.0 while the other two clusters are running version 1.2.0. New Relic's integration is only officially tested and supported with kube-state-metrics 1.1.0.
Get logs from pod connecting to kube-state-metrics

To get the logs from the pod connecting to kube-state-metrics:

  1. Get the node that kube-state-metrics is running on:

    kubectl get pods --all-namespaces -o wide | grep kube-state-metrics

    Look for output similar to this:

    kube-system   kube-state-metrics-5c6f5cb9b5-pclhh     2/2       
Running   4          4d        172.17.0.3   minikube

  2. Get the New Relic Infrastructure pod that is running on the same node as kube-state-metrics:

    kubectl describe node minikube | grep newrelic-infra

    Look for output similar to this:

    default                    newrelic-infra-5wcv6                     100m (5%)
0 (0%)      100Mi (5%)       100Mi (5%)

  3. Retrieve the logs for that node by running:

    kubectl logs newrelic-infra-5wcv6
Retrieve kube-state-metrics service configuration

To retrieve the configuration:

  1. Run:

    kubectl get pods --all-namespaces | grep "kube-state-metrics"

  2. Look for a response similar to this:

    kube-system   kube-state-metrics-5c6f5cb9b5-5wf9m     2/2       
Running       8          6d

  3. Review the namespace in the first column.

    kubectl describe service kube-state-metrics -n <namespace>

Get logs from pod running on a master node

To get the logs from a pod running on a master node:

  1. Get the nodes that are labelled as master:

    kubectl get nodes -l node-role.kubernetes.io/master=""

    Or,

    kubectl get nodes -l kubernetes.io/role="master"

    Look for output similar to this:

    NAME                         STATUS  ROLES   AGE   VERSION
ip-10-42-24-4.ec2.internal   Ready   master  42d   v1.14.8

  2. Get the New Relic Infrastructure pod that is running on one of the nodes returned in the previous step:

    kubectl get pods --field-selector spec.nodeName=ip-10-42-24-4.ec2.internal -l name=newrelic-infra --all-namespaces

    Look for output similar to this:

    newrelic-infra-whvzt

  3. Retrieve the logs for that node by running:

    kubectl logs newrelic-infra-whvzt

For troubleshooting help, see Not seeing data or Error messages.

For more help

Recommendations for learning more: