Configure control plane monitoring

New Relic Infrastructure provides control plane support for your Kubernetes integration, allowing you to monitor and collect metrics from your cluster's control plane components. That data can then be found in New Relic Infrastructure and in New Relic Insights.

What is the Kubernetes control plane?

The Kubernetes control plane is the system that maintains a record of all Kubernetes objects, manages the state of those objects, and responds to any changes in the cluster.

The control plane components, such as the Kubernetes master and kubelet processes, govern how Kubernetes communicates with your cluster. At any given time, the control plane’s control loops will respond to changes in the cluster and work to make the actual state of all the objects in the system match the desired state that you provided.

Master components provide the cluster’s control plane. These master components make global decisions about the cluster (for example, scheduling), and detect and respond to cluster events (for example, starting up a new pod when a deployment’s replicas field is unsatisfied).

As part of our control plane support, we monitor and collect metrics from the following control plane components:

  • ETCD: a distributed and consistent key-value store. This is where the current and desired state of your cluster is stored. This includes information about all pods, deployments, services, secrets, etc. This is the only place where Kubernetes stores its information. For a list of supported metrics see, ETCD data.
  • API server: the central RESTful HTTP API that handles all requests coming from users (kubectl), nodes (kubelet & kube-proxy), control plane components (scheduler & controller manager), automation, and more. It handles authentication, authorization, validation of all objects, and is responsible for storing said objects in ETCD. It’s the only component that talks with ETCD. For a list of supported metrics see, API server data.
  • Scheduler: the scheduler is responsible for assigning newly created pods to a worker node that is capable of running said pod. It takes several factors into consideration when selecting a worker node, such as: requested CPU/memory vs available on the node, tolerations to taints, any set affinity or anti-affinity. For a list of supported metrics see, Scheduler data.
  • Controller manager: is where all the controllers run. Controllers, like the scheduler, use the watch capabilities of the API server to be notified of state changes. When notified, they work to get the actual cluster state to the desired state. For example, if we create a new object that creates X number of pods, the associated controller is the one in charge of bringing the current cluster state of X pods to Y number of pods. For a list of supported metrics see, Controller manager data.

Compatibility and requirements

Control plane monitoring requires Kubernetes integration version 1.11.0 or higher.

Control plane monitoring support is not enabled for managed clusters. This is because providers (like EKS, GKE, AKS, etc.) abstract away the concept of master nodes and control plane components, so that access to them is limited or non-existent.

The unprivileged version of the Kubernetes integration does not support control plane monitoring.

Discovery of master nodes and control plane components

The Kubernetes integration relies on the kubeadm labeling conventions to discover the master nodes and the control plane components. This means that master nodes should be labeled with node-role.kubernetes.io/master="" or kubernetes.io/role="master".

The control plane components should have either the k8s-app or the tier and component labels. Refer to the following table for accepted label combinations and values:

Component Label Endpoint
API server

k8s-app=kube-apiserver

tier=control-plane component=kube-apiserver

localhost:8080/metrics (default)
ETCD

k8s-app=etcd-manager-main

tier=control-plane component=etcd

localhost:4001/metrics
Scheduler

k8s-app=kube-scheduler

tier=control-plane component=kube-scheduler

localhost:10251/metrics
Controller manager

k8s-app=kube-controller-manager

tier=control-plane component=kube-controller-manager

localhost:10252/metrics

When the integration detects that it is running inside a master node, it will try to find which components are running on the node by looking for pods that match the labels listed in the table above. For every running component, the integration will make a request to its metrics endpoint.

Configuration

Control plane monitoring is automatic for agents running inside master nodes. The only component that requires an extra step to run is ETCD, as it uses mutual TLS authentication (mTLS) for client requests. The API Server can also be configured to be queried using the Secure Port.

ETCD

In order to set mTLS for querying ETCD, there are two configuration options that need to be set:

Option Value
ETCD_TLS_SECRET_NAME

Name of a Kubernetes secret that contains the mTLS configuration.

The secret should contain the following keys:

  • cert: the certificate that will identify the client making the request. It should be signed by an ETCD trusted CA.

  • key: the private key used to generate the client certificate.

  • cacert: the root CA used to identify the ETCD server certificate.

If the ETCD_TLS_SECRET_NAME option is not set, ETCD metrics won't be fetched.

For a step by step instructions on how to create a certificate and sign it with the ETCD client CA, see Set up mTLS from the ETCD client CA.

ETCD_TLS_SECRET_NAMESPACE The namespace where the secret specified in the ETCD_TLS_SECRET_NAME was created. If not set, the default namespace will be used.

API server

By default, the API server metrics are queried using the localhost:8080 unsecured endpoint. If, this port is disabled, you can also query these metrics over the secure port. To enable this, the following configuration option should be set:

Option Value
API_SERVER_SECURE_PORT

The (secure) port to query the metrics. The API server uses port 6443 by default.

Ensure that:

  • the value is surrounded by quotes: "6443"
  • the ClusterRole has been updated to the newest version found in the manifest

Added in version 1.13.0

Set up mTLS from the ETCD client CA

The instructions below are heavily based on the Kubernetes documentation. For more information, see Managing TLS certificates in a cluster.

To set up mTLS from the ETCD client CA:

  1. Download and install the tool cfssl, selecting the correct binaries for your OS from the list.
  2. Once installed, execute the following command:

    cat <<EOF | cfssl genkey - | cfssljson -bare server
    {
      "hosts": [
        "localhost"
      ],
      "CN": "newrelic-infra.pod.cluster.local",
      "key": {
        "algo": "ecdsa",
        "size": 256
      }
    }
    EOF

    This command generates two files; server.csr containing the PEM encoded pkcs#10 certification request and server-key.pem containing the PEM encoded key to the certificate that is still to be created.

  3. Use the generated certificate authority (CA) of ETCD to sign your CSR. Depending on your cluster configuration, you may already have this information. For default install configuration, download the CA certificate and the private key directly from ETCD with the following commands:

    kubectl cp $(kubectl get pods -l k8s-app=etcd-manager-main -n kube-system -o jsonpath="{.items[0].metadata.name}"):/etc/kubernetes/pki/etcd-manager/etcd-clients-ca.crt ./cacert -n kube-system
    kubectl cp $(kubectl get pods -l k8s-app=etcd-manager-main -n kube-system -o jsonpath="{.items[0].metadata.name}"):/etc/kubernetes/pki/etcd-manager/etcd-clients-ca.key ./cacert.key -n kube-system

    This requires that the etcd-manager-main pod has the label k8s-app=etcd-manager-main which is a requirement for control plane monitoring . As we are using the kube-system namespace, if your etc-manager-main pod is located in a different namespace, change the -n kube-system flags accordingly.

  4. With those files downloaded, use the following command to sign your CSRF:

    cfssl sign -ca cacert -ca-key cacert.key server.csr | cfssljson -bare cert
    
  5. Create the secret that will be used to retrieve the TLS config for making requests to ETC, renaming the certificate and the private key to make everything easier:

    cp cert.pem cert && cp server-key.pem key
    
    kubectl -n default create secret generic newrelic-infra-etcd-tls-secret --from-file=./cert --from-file=./key --from-file=./cacert

    Note: To ease the process for future installations, you can use the following commands to simultaneously create the CSR, retrieve the CA, generate the certificate by signing the CSR, and create the secret with all the required fields:

    cat <<EOF | cfssl genkey - | cfssljson -bare server && \
    kubectl cp $(kubectl get pods -l k8s-app=etcd-manager-main -n kube-system -o jsonpath="{.items[0].metadata.name}"):/etc/kubernetes/pki/etcd-manager/etcd-clients-ca.crt ./cacert -n kube-system && \
    kubectl cp $(kubectl get pods -l k8s-app=etcd-manager-main -n kube-system -o jsonpath="{.items[0].metadata.name}"):/etc/kubernetes/pki/etcd-manager/etcd-clients-ca.key ./cacert.key -n kube-system && \
    cp server-key.pem key && \
    cfssl sign -ca cacert -ca-key cacert.key server.csr | cfssljson -bare cert && \
    cp cert.pem cert && \
    kubectl -n default create secret generic newrelic-infra-etcd-tls-secret --from-file=./cert --from-file=./key --from-file=./cacert
    {
      "hosts": [
        "localhost"
      ],
      "CN": "newrelic-infra.pod.cluster.local",
      "key": {
        "algo": "ecdsa",
        "size": 256
      }
    }
    EOF
  6. The last step is to update the integration configuration in the manifest and apply it.

    In the configuration section, there are two options related to ETCD mTLS:

    • ETCD_TLS_SECRET_NAME with the name of the secret that we just created
    • ETCD_TLS_SECRET_NAMESPACE with the namespace that we used to create the secret

    To complete the installation, add these variables to the container spec of the integration Daemonset and apply the changes:

    - name: "ETCD_TLS_SECRET_NAME”
      value: "newrelic-infra-etcd-tls-secret"
    - name: "ETCD_TLS_SECRET_NAMESPACE"
      value: "default"

For more help

Recommendations for learning more: