Kubernetes installation and configuration

Be sure to review the Kubernetes integration compatibility and requirements before you proceed.

Install Kubernetes integration

To activate the Kubernetes integration, deploy the newrelic-infra agent onto a Kubernetes cluster as a daemon set:

  1. Install kube-state-metrics and get it running on the cluster. For example, for the latest supported version, 1.5.0:

    curl -o kube-state-metrics-1.5.zip https://codeload.github.com/kubernetes/kube-state-metrics/zip/release-1.5 && unzip kube-state-metrics-1.5.zip && kubectl apply -f kube-state-metrics-release-1.5/kubernetes
  2. Download the integration configuration file:

    curl -O https://download.newrelic.com/infrastructure_agent/integrations/kubernetes/newrelic-infrastructure-k8s-latest.yaml
    
  3. In the configuration file, add your New Relic license key and a cluster name to identify your Kubernetes cluster. Both values are required.

    Recommendation: Do not change the NRIA_PASSTHROUGH_ENVIRONMENT or NRIA_DISPLAY_NAME value in your configuration file.

    env:
      - name: NRIA_LICENSE_KEY
        value: YOUR_LICENSE_KEY
      - name: CLUSTER_NAME
        value: YOUR_CLUSTER_NAME
  4. Optional: Complete these steps as applicable:

    Specify the Kubernetes API host and port

    This is necessary when you are using SSL and not using the default FQDN. The Kubernetes API FQDN needs to match the FQDN of the SSL certificate.

    You do not need to specify both variables. For example, if you only specify the HOST, the default PORT will be used.

    - name: "KUBERNETES_SERVICE_HOST" 
      value: "<Kubernetes API host>"
    - name: "KUBERNETES_SERVICE_PORT" 
      value: "<Kubernetes API TCP port>"
      
    Kubernetes versions 1.6 to 1.7.5: Edit manifest file

    For Kubernetes versions 1.6 to 1.7.5, uncomment these two lines in the manifest file:

    - name: "CADVISOR_PORT" # Enable direct connection to cAdvisor by specifying the port.  Needed for Kubernetes versions prior to 1.7.6.
      value: "4194"
      
    Use environment variables

    Use environment variables that can be passed to the Kubernetes integration if you use a proxy to configure its URL.

    Specify the kube-state-metrics URL

    If several instances of kube-state-metrics are present in the cluster, uncomment and configure the following lines to specify which one to use:

     - name: "KUBE_STATE_METRICS_URL"
       value: "http://<kube-state-metrics IP or FQDN>:<port>"
    

    Even though a KUBE_STATE_METRICS_URL is defined, the KSM service should contain one of the following labels for the auto-discovery process:

    • k8s-app=kube-state-metrics

      OR

    • app=kube-state-metrics

    kube-state-metrics timeout: Increase the client timeout

    To increase the client timeout of kube-state-metrics, add a new environment variable, TIMEOUT, to the manifest file:

    env:
      - name: TIMEOUT
        value: 5000 # The default client timeout when calling kube-state-metrics, in miliseconds
    
    Non-default namespace deployments: Edit config file

    If you want to deploy in a different namespace from default, change all values of namespace in the configuration file.

  5. Confirm that kube-state-metrics is installed.

    kubectl get pods --all-namespaces | grep kube-state-metrics
  6. Create the daemon set:

    kubectl create -f newrelic-infrastructure-k8s-latest.yaml
  7. Confirm that the daemon set has been created successfully by looking for newrelic-infra in the results generated by this command:

    kubectl get daemonsets
  8. To confirm that the integration has been configured correctly, wait a few minutes, then run this New Relic Insights query to see if data has been reported:

    SELECT * FROM K8sPodSample since 1 day ago

If you do not see Kubernetes data, review these Kubernetes integration and configuration procedures again, then follow the troubleshooting procedures. If necessary, follow the uninstall procedures.

Install on Kubernetes managed services and platforms

In addition to installation directly on a server or when using VMs, the Kubernetes integration can be installed on the following platforms:

To deploy in Amazon EKS, use the version of kubectl provided by AWS. Then, follow New Relic's standard procedures to install and configure the Kubernetes integration.

The Kubernetes integration monitors worker nodes. In Amazon EKS, master nodes are managed by Amazon and abstracted from the Kubernetes platforms.

The Kubernetes integration monitors worker nodes. In GKE, master nodes are managed by Google and abstracted from the Kubernetes platforms.

Before installing the Kubernetes integration on GKE, ensure you have sufficient permissions:

  1. Go to https://console.cloud.google.com/iam-admin/iam and find your username. Click edit.
  2. Ensure you have permissions to create Roles and ClusterRoles: If you are not sure, add the Kubernetes Engine Cluster Admin role. If you cannot edit your user role, ask the owner of the GCP project to give you the necessary permissions.

  3. Ensure you have a RoleBinding that grants you the same permissions to create Roles and ClusterRoles:

    kubectl create clusterrolebinding YOUR_USERNAME-cluster-admin-binding --clusterrole=cluster-admin --user=YOUR_GCP_EMAIL

    Creating a RoleBinding is necessary because of a known RBAC issue in Kubernetes and Kubernetes Engine versions 1.6 or higher. This issue is related to the manner in which Kubernetes Engine checks permissions when you create a Role or ClusterRole. As a workaround, the code creates a RoleBinding that gives your Google identity a cluster-admin role. For more information, see Google Cloud's documentation on defining permissions in a role.

  4. Follow New Relic's standard procedures to install and configure the Kubernetes integration.

To install the Kubernetes integration with OpenShift:

  1. Add the newrelic service account to your privileged Security Context Constraints:

    oc adm policy add-scc-to-user privileged \
    system:serviceaccount:<namespace>:newrelic
            
  2. Follow New Relic's standard procedures to install and configure the Kubernetes integration.
  3. Edit the Kubernetes integration configuration file. In the securityContext: section, add a privileged: true setting to be run and deployed in the default namespace:

    spec:
      serviceAccountName: newrelic
      containers:
         - name: newrelic-infra
           image: newrelic/infrastructure-k8s:1.7.0
           securityContext:
             privileged: true
           resources:
             limits:
               memory: 100Mi
  4. If you're using signed certificates, make sure they are properly configured by using the following variables to set the .pem file:

    - name: NRIA_CA_BUNDLE_DIR 
      value: YOUR_CA_BUNDLE_DIR 
    - name: NRIA_CA_BUNDLE_FILE  
      value: YOUR_CA_BUNDLE_NAME     
  5. Save your changes.

To deploy in Azure Kubernetes Service (AKS), follow New Relic's standard procedures to install and configure the Kubernetes integration.

The Kubernetes integration monitors worker nodes. In Azure Kubernetes Service, master nodes are managed by Azure and abstracted from the Kubernetes platforms.

Update to the latest version

If you are already running the Kubernetes integration and want to update the newrelic-infra agent to the latest agent version:

  1. Download the integration configuration file:

    curl -O https://download.newrelic.com/infrastructure_agent/integrations/kubernetes/newrelic-infrastructure-k8s-latest.yaml
    
  2. Copy the changes you made to the configuration file. At a minimum, include CLUSTER_NAME and NRIA_LICENSE_KEY, and paste your changes in the configuration file you downloaded.

  3. Delete the daemon set currently running:

    kubectl delete -f PREVIOUS_CONFIGURATION_FILE
    
  4. Install the latest daemon set with the following command:

    kubectl create -f newrelic-infrastructure-k8s-latest.yaml
    

Install unprivileged version of the Kubernetes integration

For platforms that have tight security requirements; we provide an unprivileged version of the Kubernetes integration. Changes from the standard Kubernetes integration are:

  • Runs the Infrastructure agent and the Kubernetes integration as a standard user instead of root
  • No access to the underlying host filesystem
  • No access to /var/run/docker.sock
  • Container's root filesystem mounted as read-only
  • allowPrivilegeEscalation is set to false
  • hostnetwork is set to false

The tradeoff is that the solution will only collect metrics from Kubernetes, but it will not collect any metric from the underlying hosts directly. Kubernetes does provide some data (metrics and metadata) about its nodes (hosts).

Optionally and to collect the underlying host metrics, the non-containerized Infrastructure agent can be deployed on the underlying host. The Infrastructure agent already supports running as non-root. The combination of the Kubernetes integration in its unprivileged version and the agent running on the host will report all the metrics that our standard solution for monitoring Kubernetes receives.

Install the unprivileged Kubernetes integration

  1. Install kube-state-metrics and get it running on the cluster. For example, for the latest supported version, 1.5.0:

    curl -o kube-state-metrics-1.5.zip https://codeload.github.com/kubernetes/kube-state-metrics/zip/release-1.5 && unzip kube-state-metrics-1.5.zip && kubectl apply -f kube-state-metrics-release-1.5/kubernetes
  2. Get the integration manifest file from the email.
  3. Download the integration configuration file:

    curl -O https://download.newrelic.com/infrastructure_agent/integrations/kubernetes/newrelic-infrastructure-k8s-unprivileged-latest.yaml
    
  4. In the configuration file, add your New Relic license key and a cluster name to identify your Kubernetes cluster. Both values are required.

    env:
      - name: NRIA_LICENSE_KEY
        value: YOUR_LICENSE_KEY
      - name: CLUSTER_NAME
        value: YOUR_CLUSTER_NAME
  5. Confirm that kube-state-metrics is installed.

    kubectl get pods --all-namespaces | grep kube-state-metrics
  6. Create the daemon set:

    kubectl create -f newrelic-infrastructure-k8s-unprivileged-latest.yaml
  7. Confirm that the daemon set has been created successfully by looking for newrelic-infra in the results generated by this command:

    kubectl get daemonsets
  8. To confirm that the integration has been configured correctly, wait a few minutes, then run this New Relic Insights query to see if data has been reported:

    SELECT * FROM K8sPodSample since 5 minutes ago
    

Uninstall Kubernetes integration

To uninstall the Kubernetes integration, use the following command:

kubectl delete -f newrelic-infrastructure-k8s-latest.yaml

Verify that newrelic-infrastructure-k8s-latest.yaml corresponds to the filename of the manifest as you have saved it. If you are using the unprivileged version of the integration, the default filename will be newrelic-infrastructure-k8s-unprivileged-latest.yaml.

You only need to execute this command once, regardless of the number of nodes in your cluster.

For more help