Kubernetes integration: install and configure

The easiest way to install the Kubernetes integration is to use our automated installer to generate a manifest. It bundles not just the integration DaemonSets, but also other New Relic Kubernetes configurations, like Kubernetes events, Prometheus OpenMetrics, and New Relic log monitoring.

Use automated installer

You can use the automated installer for servers, VMs, and unprivileged environments. The installer can also help you with managed services or platforms after you review a few preliminary notes. We also have separate instructions if you need a custom manifest or prefer to do a manual unprivileged installation.

Start the installer

If your New Relic account is in the EU region, access the installer from one.eu.newrelic.com.

Installs for managed services and platforms

Before starting our automated installer, check out these notes for your managed services or platforms:

The Kubernetes integration monitors worker nodes. In GKE, master nodes are managed by Google and abstracted from the Kubernetes platforms.

Before starting our automated installer to deploy the Kubernetes integration on GKE, ensure you have sufficient permissions:

Go to console.cloud.google.com/iam-admin/iam and find your username. Click edit.
Ensure you have permissions to create Roles and ClusterRoles: If you are not sure, add the Kubernetes Engine Cluster Admin role. If you cannot edit your user role, ask the owner of the GCP project to give you the necessary permissions.
Ensure you have a RoleBinding that grants you the same permissions to create Roles and ClusterRoles:
```
kubectl create clusterrolebinding YOUR_USERNAME-cluster-admin-binding --clusterrole=cluster-admin --user=YOUR_GCP_EMAIL
```
Creating a RoleBinding is necessary because of a known RBAC issue in Kubernetes and Kubernetes Engine versions 1.6 or higher. For more information, see Google Cloud's documentation on defining permissions in a role.

To deploy the Kubernetes integration with OpenShift:

Add the <>{'<release_name>'}</>-newrelic-infrastructure service account to your privileged Security Context Constraints:
```
oc adm policy add-scc-to-user privileged \
system:serviceaccount:<namespace>:<release_name>-newrelic-infrastructure
```
The default <>{'<release_name>'}</> provided by the installer is nri-bundle.
Complete the steps in our automated installer.
If you're using signed certificates, make sure they are properly configured by using the following variables in the DaemonSet portion of your manifest to set the .pem file:
```
- name: NRIA_CA_BUNDLE_DIR
  value: YOUR_CA_BUNDLE_DIR
- name: NRIA_CA_BUNDLE_FILE
  value: YOUR_CA_BUNDLE_NAME
```
YAML key path: spec.template.spec.containers.name.env
Save your changes.

Custom manifest

If the Kubernetes automated installer doesn't provide the settings you need, you can download our manifest template and install the integration manually.

To activate the Kubernetes integration, you must deploy the newrelic-infra agent onto a Kubernetes cluster as a DaemonSet:

Install kube-state-metrics and get it running on the cluster. For example:

curl -L -o kube-state-metrics-1.9.5.zip https://github.com/kubernetes/kube-state-metrics/archive/v1.9.5.zip && unzip kube-state-metrics-1.9.5.zip && kubectl apply -f kube-state-metrics-1.9.5/examples/standard

Download the manifest file:
```
curl -O https://download.newrelic.com/infrastructure_agent/integrations/kubernetes/newrelic-infrastructure-k8s-latest.yaml
```
In the DaemonSet portion of your manifest, add your New Relic license key and a cluster name to identify your Kubernetes cluster. Both values are required.
- Recommendation: Do not change the NRIA_PASSTHROUGH_ENVIRONMENT or NRIA_DISPLAY_NAME value in your manifest.
- YOUR_CLUSTER_NAME is your cluster’s id in New Relic Explorer. It doesn’t need to match the name of the cluster running in your environment.
- YAML key path: spec.template.spec.containers.name.env
```
env:
  - name: NRIA_LICENSE_KEY
    value: YOUR_LICENSE_KEY
  - name: CLUSTER_NAME
    value: YOUR_CLUSTER_NAME
```
If you need to adapt the manifest to fit your environment, review the configure section in this doc.

Confirm that kube-state-metrics is installed.

kubectl get pods --all-namespaces | grep kube-state-metrics

Create the DaemonSet:

kubectl create -f newrelic-infrastructure-k8s-latest.yaml

Confirm that the DaemonSet has been created successfully by looking for newrelic-infra in the results generated by this command:
```
kubectl get daemonsets
```

To confirm that the integration is working: wait a few minutes, then look for data in the New Relic Kubernetes cluster explorer.

If you don't see data, review the configuration procedures again, then follow the troubleshooting procedures.

Important

In the future, the number of labels collected on Kubernetes objects will be limited per object type (containers, pods, nodes, etc.). If objects have labels above the limit, you will be able to configure important labels that should always be sent to New Relic. When the limitation is in place, this documentation will be updated.

Make sure New Relic pods can be scheduled

Some of the New Relic pods are set up as DaemonSet in the manifest file so that they can run on every host. These include newrelic-infrastructure and newrelic-logging. In rare circumstances, other pods may be scheduled first and starve the New Relic pods of resources. Since each of these pods have to run on a specific host, they will stay in pending status until that host has enough resources, even if there are other hosts available. This could end up occurring for long periods of time and result in reporting gaps.

To prevent this scenario, you can configure the Kubernetes scheduler to give New Relic pods a higher priority. Using the default scheduler:

Ensure kube-scheduler flag disablePreemption is not set to true (by default it is false).
Create a PriorityClass for the New Relic DaemonSet pods:
- Set the appropriate priority value, which should generally be higher than your other pods.
- preemptionPolicy is set to PreemptLowerPriority by default. This allows New Relic pods assigned this priority class to remove lower-priority pods that are taking up resources.

Edit the manifest file to add priorityClassName to any DaemonSet specs. In the example below, the highlighted line sets the priority class for newrelic-infrastructure:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: default
  labels:
    app: newrelic-infrastructure
    chart: newrelic-infrastructure-1.0.0
    release: nri-bundle
    mode: privileged
  name: nri-bundle-newrelic-infrastructure
spec:
  priorityClassName: your-priority-class
  ...

If you have already deployed the New Relic pods, re-deploy them and confirm they have been created:

kubectl delete -f newrelic-infrastructure-k8s-latest.yaml
kubectl create -f newrelic-infrastructure-k8s-latest.yaml
kubectl get daemonsets

Unprivileged installs of the Kubernetes integration

For platforms that have stringent security requirements, we provide an unprivileged version of the Kubernetes integration. Changes from the standard Kubernetes integration are:

Runs the infrastructure agent and the Kubernetes integration as a standard user instead of root
No access to the underlying host filesystem
No access to /var/run/docker.sock
Container's root filesystem mounted as read-only
allowPrivilegeEscalation is set to false
hostnetwork is set to false

The tradeoff is that the solution will only collect metrics from Kubernetes, but it will not collect any metric from the underlying hosts directly. Kubernetes provides some data (metrics and metadata) about its nodes (hosts).

Tip

Optional: To collect the underlying host metrics, the non-containerized infrastructure agent can be deployed on the underlying host. The infrastructure agent already supports running as non-root. The combination of the Kubernetes integration in its unprivileged version and the agent running on the host will report all the metrics that our standard solution for monitoring Kubernetes receives.

Install kube-state-metrics and get it running on the cluster. For example:

curl -L -o kube-state-metrics-1.9.5.zip https://github.com/kubernetes/kube-state-metrics/archive/v1.9.5.zip && unzip kube-state-metrics-1.9.5.zip && kubectl apply -f kube-state-metrics-1.9.5/examples/standard

Download the integration manifest:

curl -O https://download.newrelic.com/infrastructure_agent/integrations/kubernetes/newrelic-infrastructure-k8s-unprivileged-latest.yaml

In the manifest, add your New Relic license key and a cluster name to identify your Kubernetes cluster. Both values are required.
Important
YOUR_CLUSTER_NAME is your cluster’s id in New Relic Explorer. It doesn’t need to match the name of the cluster running in your environment.
```
env:
  - name: NRIA_LICENSE_KEY
    value: YOUR_LICENSE_KEY
  - name: CLUSTER_NAME
    value: YOUR_CLUSTER_NAME
```
YAML key path: spec.template.spec.containers.name.env

Confirm that kube-state-metrics is installed.

kubectl get pods --all-namespaces | grep kube-state-metrics

Create the DaemonSet:

kubectl create -f newrelic-infrastructure-k8s-unprivileged-latest.yaml

Confirm that the DaemonSet has been created successfully by looking for newrelic-infra in the results generated by this command:
```
kubectl get daemonsets
```
To confirm that the integration has been configured correctly, wait a few minutes, then run this NRQL query to see if data has been reported:
```
SELECT * FROM K8sPodSample since 5 minutes ago
```

Configure the integration

The Kubernetes integration comes with a default configuration that should work in most environments. To change the configuration, modify the manifest file:

This is necessary when you are using SSL and not using the default FQDN. The Kubernetes API FQDN needs to match the FQDN of the SSL certificate.

You do not need to specify both variables. For example, if you only specify the HOST, the default PORT will be used.

- name: "KUBERNETES_SERVICE_HOST" 
  value: "KUBERNETES_API_HOST"
- name: "KUBERNETES_SERVICE_PORT" 
  value: "KUBERNETES_API_TCP_PORT"

For Kubernetes versions 1.6 to 1.7.5, uncomment these two lines in the manifest file:

- name: "CADVISOR_PORT" # Enable direct connection to cAdvisor by specifying the port.  Needed for Kubernetes versions prior to 1.7.6.
  value: "4194"

You can disable kube-state-metrics parsing for the DaemonSet by using the following configuration:

- name: "DISABLE_KUBE_STATE_METRICS"
   value: "true"

Caution

Disabling kube-state-metrics also disables data collection for the following:

ReplicaSets
DaemonSets
StatefulSets
Namespaces
Deployments
Services
Endpoints
Pods (that are pending)
Additionally, disabling this affects the Kubernetes Cluster Explorer in the following ways:
No pending pods are shown.
No filters based on services.

If several instances of kube-state-metrics are present in the cluster, uncomment and configure the following lines to specify which one to use:

- name: "KUBE_STATE_METRICS_URL"
   value: "http://KUBE_STATE_METRICS_IP_OR_FQDN:PORT"

Important

Even though a KUBE_STATE_METRICS_URL is defined, the KSM service should contain one of the following labels for the auto-discovery process:

k8s-app=kube-state-metrics
OR
app=kube-state-metrics
OR
app.kubernetes.io/name=kube-state-metrics

Important

This configuration option overrides KUBE_STATE_METRICS_POD_LABEL. If you have both defined, KUBE_STATE_METRICS_POD_LABEL has no effect.

If several instances of kube-state-metrics are present in the cluster, another option to easily target one of these instances with the Kubernetes integration is to use label-based discovery.

- name: "KUBE_STATE_METRICS_POD_LABEL"
   value: "LABEL_NAME"

Important

When a KUBE_STATE_METRICS_POD_LABEL is defined, the label should have a value equal to true. For example, if the label name is my-ksm, ensure that my-ksm=true.

Important

This configuration option is incompatible with KUBE_STATE_METRICS_URL. If you have both defined, KUBE_STATE_METRICS_URL is used.

If your instance of kube-state-metrics is behind kube-rbac-proxy, the integration can be configured in a compatible way using the combination of the label-based discovery and two other environment variables:

- name: "KUBE_STATE_METRICS_SCHEME"
   value: "https"
 - name: "KUBE_STATE_METRICS_PORT"
   value: "KSM_RBAC_PROXY_PORT"

To confirm which port should be used as the value of KUBE_STATE_METRICS_PORT, we recommend running a describe command on the kube-state-metrics pod and look for the port exposed by the container named kube-rbac-proxy-main.

Important

These two configuration options only work when using the KUBE_STATE_METRICS_POD_LABEL configuration described above.

To increase the client timeout of kube-state-metrics, add a new environment variable, TIMEOUT, to the manifest file:

env:
  - name: TIMEOUT
    value: 5000 # The default client timeout when calling kube-state-metrics, in milliseconds

Then, add this new environment variable to the NRIA_PASSTHROUGH_ENVIRONMENT

By default, the integration will cache any retrieved information from the Kubernetes API for 5 minutes.

Use the API_SERVER_CACHE_TTL environment variable to set a custom cache duration for responses from the API server. Valid time unit values are: ns, us, ms, s, m, and h. To disable caching, set to 0s.

env: 
  - name: API_SERVER_CACHE_TTL 
    value: "1m"

Use the following environment variables if any of the Kubernetes control plane components export metrics on base URLs that are different from the defaults. This is necessary for environments such as OpenShift when a control plane component metrics endpoint is using SSL or an alternate port.

Values of these environment variables must be base URLs of the form [scheme]://[host]:[port]. URLs should not include a path component. For example:

- name: "SCHEDULER_ENDPOINT_URL"
  value: "https://localhost:10259"
- name: "ETCD_ENDPOINT_URL"
  value: "https://localhost:9979"
- name: "CONTROLLER_MANAGER_ENDPOINT_URL"
  value: "https://localhost:10257"
- name: "API_SERVER_ENDPOINT_URL"
  value: "https://localhost:6443"

The /metrics path segment is added automatically. In addition, if the https scheme is used, authentication to the control plane component pod(s) is accomplished via service accounts.

Caution

If a FQDN (fully qualified domain name) is used in a multi-master cluster, inconsistent results may be returned. Therefore, it is recommended to use localhost only.

Important

Even though a custom base URL is defined for a given control plane component, the control plane component pod(s) must contain one of the labels supported by the auto-discovery process.

Important

Even though a custom ETCD_ENDPOINT_URL can be defined, ETCD will always require https and mTLS authentication to be configured.

Here are some additional configurations to consider:

Configure the infrastructure agent

The Kubernetes integration image comes with a default configurations for the agent that can be modified if needed. When installing with the manifest, you can modify the infrastructure agent configuration by editing the manifest and adding any needed configuration option of the agent as environment variables of the newrelic-infrastructure DaemonSet.

When installing with Helm, you can specify the needed infrastructure agent configuration options in the values.yaml as shown in the example in GitHub.

The config object is used to populate the configMap that is mounted automatically in the location of the infrastructure agent configuration file in the pods created by the newrelic-infrastructure DaemonSet.

Update to the latest version

Using the automated installer

To update a Kubernetes integration installed with the automated installer, just run the installer again. It will always offer a manifest pointing to the last released version of the integration.

Using helm

See Install the Kubernetes integration using Helm

Custom manifest

If you are already running the Kubernetes integration and want to update the newrelic-infra agent to the latest agent version:

Run this NRQL query to check which version you are currently running (this will return the image name by cluster):
```
SELECT latest(containerImage)  FROM K8sContainerSample 
WHERE containerImage LIKE '%newrelic/infrastructure%' FACET clusterName SINCE 1 day ago
```
If you've set a name other than newrelic/infrastructure for the integration's container image, the above query won't yield results: to make it work, edit the name in the query.

Download the integration manifest file:

curl -O https://download.newrelic.com/infrastructure_agent/integrations/kubernetes/newrelic-infrastructure-k8s-latest.yaml

Copy the changes you made to the manifest. At a minimum, include CLUSTER_NAME and NRIA_LICENSE_KEY, and paste your changes in the manifest you downloaded.
Install the latest DaemonSet with the following command (Kubernetes will automatically do a rollout upgrade for the integration's pods):
```
kubectl apply -f newrelic-infrastructure-k8s-latest.yaml
```

Uninstall the Kubernetes integration

To uninstall the Kubernetes integration:

Verify that newrelic-infrastructure-k8s-latest.yaml corresponds to the filename of the manifest as you have saved it.
Example: If you are using the unprivileged version of the integration, the default filename will be newrelic-infrastructure-k8s-unprivileged-latest.yaml.
After you verify the filename, use the following command:
```
kubectl delete -f newrelic-infrastructure-k8s-latest.yaml
```

You only need to execute this command once, regardless of the number of nodes in your cluster.

For more help

If you need more help, check out these support and learning resources:

Browse the Explorers Hub to get help from the community and join in discussions.
Find answers on our sites and learn how to use our support portal.
Run New Relic Diagnostics, our troubleshooting tool for Linux, Windows, and macOS.
Review New Relic's data security and licenses documentation.

Kubernetes integration: install and configure

Use automated installer

Installs for managed services and platforms

Amazon EKS Fargate

Google Kubernetes Engine (GKE)

OpenShift container platform

Azure Kubernetes Service (AKS)

Pivotal Container Service (PKS / VMware Tanzu)

Custom manifest

Important

Make sure New Relic pods can be scheduled

Unprivileged installs of the Kubernetes integration

Tip

Steps to complete an unprivileged install

Configure the integration

Select which processes should send their data to New Relic

Specify the Kubernetes API host and port

Kubernetes versions 1.6 to 1.7.5: Edit manifest file

Use environment variables

Disable kube-state-metrics parsing

Specify the kube-state-metrics URL

Discover kube-state-metrics pods using a label

Query kube-state-metrics behind RBAC

kube-state-metrics timeout: Increase the client timeout

Non-default namespace deployments: Edit config file

Set the TTL for the Kubernetes API responses cache

Specify base URLs for control plane component endpoints

Configure the infrastructure agent

Update to the latest version

Using the automated installer

Using helm

Custom manifest

Uninstall the Kubernetes integration

.css-3u3y73{fill:none;stroke:currentColor;stroke-width:2;stroke-linecap:round;stroke-linejoin:round;}.css-1bvd50p{width:1rem;height:1rem;fill:none;stroke:currentColor;stroke-width:2;stroke-linecap:round;stroke-linejoin:round;}For more help

Disable `kube-state-metrics` parsing

Specify the `kube-state-metrics` URL

Discover `kube-state-metrics` pods using a label

Query `kube-state-metrics` behind RBAC

`kube-state-metrics` timeout: Increase the client timeout

For more help