To generate verbose logs and get version and configuration information, follow these procedures.
- Get verbose logs
-
For the Kubernetes integration, the infrastructure agent adds a log entry only in the event of an error. Most common errors are displayed in the standard (non-verbose) logs. If you are doing a more in-depth investigation on your own or with New Relic Support, you can enable verbose mode.
Verbose mode significantly increases the amount of infor sent to log files. Temporarily enable this mode only for troubleshooting purposes, and reset the log level when finished.
To get verbose logging details:
- Enable
verbose
logging: In the deployment file, set the value ofNRIA_VERBOSE
to1
. -
Apply the modified configuration by running:
kubectl apply -f your_newrelic_k8s.yaml
- Leave on verbose mode for a few minutes, or until you feel enough activity has occurred.
- Disable verbose mode: Set the
NRIA_VERBOSE
value back to0
. -
Apply the restored configuration by running:
kubectl apply -f your_newrelic_k8s.yaml
-
Get a list of nodes in the environment:
kubectl get nodes --all-namespaces
-
Get a list of infrastructure and kube-state-metrics pods:
kubectl get pods --all-namespaces -o wide | egrep 'newrelic|kube-state-metrics'
- Get logs from the pod connecting to kube-state-metrics.
- Retrieve kube-state-metrics service configuration.
- Enable
- Get the infrastructure agent version
-
For the Kubernetes integration, the infrastructure agent is distributed as a Docker image that contains the infrastructure agent and the Kubernetes integration. The Docker image is tagged with a version, and the infrastructure agent also has its own version.
When the agent is successfully sending information to New Relic, you can retrieve the versions of the infrastructure agent for Kubernetes (the Docker image) you are running in your clusters by using the following Insights query:
FROM K8sContainerSample SELECT uniqueCount(entityId) WHERE containerName = 'newrelic-infra' facet clusterName, containerImage
If the agent is not reporting any data:
-
Get the version(s) of the New Relic integration for Kubernetes that you are running in a cluster using
kubectl
:kubectl get pods --all-namespaces -l name=newrelic-infra -o jsonpath="{.items..spec..containers..image}"
-
Look for output similar to this:
newrelic/infrastructure-k8s:1.0.0
-
- Get kube-state-metrics version
-
To retrieve the version of kube-state-metrics running on your clusters, run the following Insights NRQL query:
FROM K8sContainerSample SELECT uniqueCount(entityId) WHERE containerName LIKE '%kube-state-metrics%' facet clusterName, containerImage
- Get logs from pod connecting to kube-state-metrics
-
To get the logs from the pod connecting to kube-state-metrics:
-
Get the node that kube-state-metrics is running on:
kubectl get pods --all-namespaces -o wide | grep kube-state-metrics
Look for output similar to this:
kube-system kube-state-metrics-5c6f5cb9b5-pclhh 2/2 Running 4 4d 172.17.0.3 minikube
-
Get the New Relic pod that is running on the same node as kube-state-metrics:
kubectl describe node minikube | grep newrelic-infra
Look for output similar to this:
default newrelic-infra-5wcv6 100m (5%) 0 (0%) 100Mi (5%) 100Mi (5%)
-
Retrieve the logs for that node by running:
kubectl logs newrelic-infra-5wcv6
-
- Retrieve kube-state-metrics service configuration
-
To retrieve the configuration:
-
Run:
kubectl get pods --all-namespaces | grep "kube-state-metrics"
-
Look for a response similar to this:
kube-system kube-state-metrics-5c6f5cb9b5-5wf9m 2/2 Running 8 6d
-
Review the namespace in the first column.
kubectl describe service kube-state-metrics -n <namespace>
-
- Get logs from pod running on a master node
-
To get the logs from a pod running on a master node:
-
Get the nodes that are labelled as master:
kubectl get nodes -l node-role.kubernetes.io/master=""
Or,
kubectl get nodes -l kubernetes.io/role="master"
Look for output similar to this:
NAME STATUS ROLES AGE VERSION ip-10-42-24-4.ec2.internal Ready master 42d v1.14.8
-
Get the New Relic pod that is running on one of the nodes returned in the previous step:
kubectl get pods --field-selector spec.nodeName=ip-10-42-24-4.ec2.internal -l name=newrelic-infra --all-namespaces
Look for output similar to this:
newrelic-infra-whvzt
-
Retrieve the logs for that node by running:
kubectl logs newrelic-infra-whvzt
-
For troubleshooting help, see Not seeing data or Error messages.