To generate verbose logs and get version and configuration information, follow the steps below. If you are using version 2 of the integration, skip to this section.
The Kubernetes integration does not produce debug
level logs. If you are doing a more in-depth investigation on your own or with New Relic Support, you can enable verbose mode.
Caution
Verbose mode significantly increases the amount of info sent to log files. Only enable this mode temporarily for troubleshooting purposes, and reset the log level when finished.
To get verbose logging details using Helm and the
nri-bundle
chart, add the following configuration to yourvalues.yaml
file. You can also enable verbose logging for all subcharts:# To enable verboseLog for newrelic-infrastructure only# newrelic-infrastructure:# verboseLog: true# To enable verboseLog for all sub-charts# global:# verboseLog: trueOnce you have added the desired configuration, upgrade the solution with the following command:
bash$helm upgrade <RELEASE_NAME> newrelic/nri-bundle \>--namespace <NEWRELIC_NAMESPACE> \>-f values-newrelic.yaml \>[--version fixed-chart-version]After collecting your verbose logs, revert the change in your
values.yaml
file, and run theupgrade
command again.To get verbose logging details using a manifest file, you need to set two different environment variables:
NRIA_VERBOSE="1"
for allagent
andforwarder
containers; this enables agent verbose logs. You can set this belowNRIA_LICENSE_KEY
.set
NRI_KUBERNETES_VERBOSE=true"
in all components of the integration; this enables verbose logs for the integration.env:
- name: NRIA_LICENSE_KEY valueFrom: secretKeyRef: key: licenseKey name: newrelic-bundle-newrelic-infrastructure-license
- name: NRIA_VERBOSE
value: "1"
Once you have edited the manifest, upgrade the solution with the following command:
bash$kubectl apply -f your_newrelic_k8s.yaml -n <NEWRELIC_NAMESPACE>After collecting your verbose logs, revert the change in the manifest, and apply the manifest again.
There are three different components of the integration that are in charge of scraping ksm
, controlplane
, and kubelet
.
In each instance, there are two containers running, with one scraping the data and the other one forwarding it.
The agent in the kubelet
component is also in charge of scraping node data and running integrations. The diagram below shows the integration installed in a cluster with three nodes.
To fetch logs, get the name of the pods you want to retrieve logs from:
$kubectl get pods -n <NEWRELIC_NAMESPACE>
Retrieve the logs:
$kubectl logs <POD_NAME> --all-containers --prefix
To retrieve logs from the previous execution, add the --previous
flag. To get logs from only one of the containers, remove the --all-containers --prefix
flag and specify the container with the --container
option.
Important
For the best experience with our Kubernetes integration, we recommend that you use the latest version. Check out the changes introduced in version 3.
When you use the Kubernetes integration, the infrastructure agent is distributed as a Docker image that contains both the infrastructure agent and the Kubernetes integration. The Docker image is tagged with a version, and the infrastructure agent also has its own version.
Once the agent is sending data to New Relic, you can retrieve the versions of the infrastructure agent for Kubernetes (the Docker image) you are running in your clusters by using the following NRQL query:
FROM K8sContainerSample SELECT uniqueCount(entityId) WHERE containerName LIKE 'agent' FACET clusterName, containerImage
If the agent isn't reporting any data:
Get the versions of the New Relic integration for Kubernetes that you are running in a cluster using kubectl
:
$kubectl get pods --all-namespaces -l app.kubernetes.io/name=newrelic-infrastructure -o jsonpath="{.items..spec..containers..image}"
To retrieve the version of kube-state-metrics
running on your clusters, run the following NRQL query:
FROM K8sContainerSample SELECT uniqueCount(entityId) WHERE containerName LIKE '%kube-state-metrics%' facet clusterName, containerImage
Integration version 2 instructions
For the Kubernetes integration, the infrastructure agent only adds a log entry in the event of an error. Most common errors are displayed in the standard (non-verbose) logs. If you are doing a more in-depth investigation on your own or with New Relic Support, you can enable verbose mode.
Caution
Verbose mode significantly increases the amount of info sent to log files. Only enable this mode temporarily for troubleshooting purposes, and reset the log level when finished.
To enable verbose logging using a manifest file:
Enable verbose logging: In the deployment file, set the value of
NRIA_VERBOSE
to1
.Apply the modified configuration by running:
bash$kubectl apply -f your_newrelic_k8s.yamlLeave on verbose mode for a few minutes, or until enough activity has occurred.
Disable verbose mode: Set the
NRIA_VERBOSE
value back to0
.Apply the restored configuration by running:
bash$kubectl apply -f your_newrelic_k8s.yamlGet a list of nodes in the environment:
bash$kubectl get nodes --all-namespacesGet a list of infrastructure and kube-state-metrics pods:
bash$kubectl get pods --all-namespaces -o wide | egrep 'newrelic|kube-state-metrics'
For the Kubernetes integration, the infrastructure agent only adds a log entry in the event of an error. Most common errors are displayed in the standard (non-verbose) logs. If you are doing a more in-depth investigation on your own or with New Relic Support, you can enable verbose mode.
Caution
Verbose mode significantly increases the amount of info sent to log files. Only enable this mode temporarily for troubleshooting purposes, and reset the log level when finished.
To enable verbose logging using Helm:
Run the following command:
bash$helm upgrade -n <namespace> --reuse-values newrelic-bundle --set newrelic-infrastructure.verboseLog=true newrelic/nri-bundleLeave on verbose mode for a few minutes, or until enough activity has occurred.
When you have the information you need, disable verbose logging:
bash$helm upgrade --reuse-values newrelic-bundle --set newrelic-infrastructure.verboseLog=false newrelic/nri-bundleApply the restored configuration by running:
bash$kubectl apply -f your_newrelic_k8s.yamlGet a list of nodes in the environment:
bash$kubectl get nodes --all-namespacesGet a list of infrastructure and kube-state-metrics pods:
bash$kubectl get pods --all-namespaces -o wide | egrep 'newrelic|kube-state-metrics'
To get the logs from pods connecting to kube-state-metrics
:
Run the following command:
bash$kubectl get pods --all-namespaces -o wide | grep kube-state-metricsLook for output similar to this:
bash$kube-system kube-state-metrics-5c6f5cb9b5-pclhh 2/2$Running 4 4d 172.17.0.3 minikubeGet the New Relic pods that are running on the same node as
kube-state-metrics
:bash$kubectl describe node minikube | grep newrelic-infraLook for output similar to this:
bash$default newrelic-infra-5wcv6 100m (5%)$0 (0%) 100Mi (5%) 100Mi (5%)Retrieve the logs for the nodes by running:
bash$kubectl logs newrelic-infra-5wcv6
To get the logs from a pod running on a master node:
Get the nodes that are labelled as master:
bash$kubectl get nodes -l node-role.kubernetes.io/master=""Or,
bash$kubectl get nodes -l kubernetes.io/role="master"Look for output similar to this:
bash$NAME STATUS ROLES AGE VERSION$ip-10-42-24-4.ec2.internal Ready master 42d v1.14.8Get the New Relic pods that are running on one of the nodes returned in the previous step:
bash$kubectl get pods --field-selector spec.nodeName=ip-10-42-24-4.ec2.internal -l name=newrelic-infra --all-namespacesLook for output similar to this:
bash$newrelic-infra-whvztRetrieve the logs for the nodes by running:
bash$kubectl logs newrelic-infra-whvzt
For troubleshooting help, see Not seeing data or Error messages.