Problem
You are getting error messages for the New Relic Kubernetes integration from your terminal during the Kubernetes integration installation or in your New Relic infrastructure logs after the Kubernetes integration installation.
If you see the following error message during your manual Kubernetes intergration installation:
$repo newrelic not found
You may have forgotten or skipped this command for adding the newrelic repo to your helm chart:
$helm repo add newrelic https://helm-charts.newrelic.com
If the license you are using is invalid then you will see an error like this in the logs of the agent
or forwarder
containers:
$2018-04-09T14:20:17.750893186Z time="2018-04-09T14:20:17Z" level=error $msg="metric sender can't process 0 times" error="InventoryIngest: events$ were not accepted: 401 401 Unauthorized Invalid license key."
To resolve this problem make sure you specify a valid .
If the agent is not able to connect to New Relic servers you will see an error like the following in the logs of the agent
or forwarder
containers:
$2018-04-09T18:16:35.497195185Z time="2018-04-09T18:16:35Z" level=error $msg="metric sender can't process 1 times" error="Error sending events: $Post https://staging-infra-api.newrelic.com/metrics/events/bulk: $net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Depending on the exact nature of the error the message in the logs may differ.
To address this problem, see the New Relic networks documentation.
The Kubernetes integration requires kube-state-metrics
. If this is missing, you'll see an error like the following in the newrelic-infra
container logs:
$time="2022-06-21T09:12:20Z" level=error msg="retrieving scraper data: retrieving ksm data: discovering KSM endpoints: timeout discovering endpoints"
Common reasons for this error include:
kube-state-metrics
has not been deployed into the cluster.kube-state-metrics
is deployed using a custom deployment.- There are multiple versions of
kube-state-metrics
running and the Kubernetes integration is not finding the correct one.
The Kubernetes integration automatically discovers kube-state-metrics
in your cluster using by default leveraging the label "app.kubernetes.io/name=kube-state-metrics"
across all namespaces.
You can change the discovery behavior in the ksm.config
of the helm chart values.
During the guided install, an output error like the one below indicates that you are experiencing a networking connection issue between the Kubernetes client and Kubernetes API server. Please make sure your Kubernetes client can connect to your Kubernetes API server before you run the guided install again.
$Unable to connect to the server: dial tcp [7777:777:7777:7777:77::77]:443: i/o timeout
During the guided install, an output error, like nrk8s-kubelet pod is not starting
, indicates that the Kubernetes kubelet pod cannot be started within 5 minutes and the installation script exits due to this timeout.
In this case, you can run kubectl get pods -o wide -n newrelic | grep nrk8s-kubelet
to see the pod's status and restarts.
- If the pod is in
ImagePullBackOff
status, please check you network connection to allow image pulling from the domains listed here. - If the pod is in
Pending
orContainerCreating
status, please usekubectl logs newrelic-bundle-nrk8s-kubelet-***<DNT>** -n newrelic
andkubectl logs newrelic-bundle-nrk8s-kubelet-**</DNT>*** -n newrelic -c kubelet
to figure out the potential reasons from the logs.
If you have a problem with the New Relic dockerhub and Google's registry.k8s.io
during the installation, check you've added their domains to your allow list. The installation pulls the container images from this location. You may need to follow the commands to find the extra Google registry domains to add to your whitelist. registry.k8s.io
usually redirects to your local registry domain. For example, asia-northeast1-docker.pkg.dev
based on your region.