Changes introduced in the Kubernetes integration version 3

From version 3 onwards, the Kubernetes solution of New Relic features an architecture which aims to be more modular and configurable, giving you more power to choose how the solution is deployed and making it compatible with more environments.

Data reported by the Kubernetes Integration version 3 hasn't changed since version 2. For version 3, we focused on configurability, stability, and user experience.

Migration Guide

To make migration from earlier versions as easy as possible, we have developed a compatibility layer that translates most of the options that could be specified in the old newrelic-infrastructure chart to their new counterparts. This compatibility layer is temporary and will be removed in the future. Therefore, we encourage you to read this guide carefully and migrate the configuration with human supervision.

Kube State Metrics (KSM) configuration

• disableKubeStateMetrics has been replaced by ksm.enabled. The default is still the same (KSM scraping enabled).
• kubeStateMetricsScheme, kubeStateMetricsPort, kubeStateMetricsUrl, kubeStateMetricsPodLabel, and kubeStateMetricsNamespace have been replaced by the more comprehensive and flexible ksm.config.
• Note, KSM v2+ disables labels and annotations metrics by default. You can enable the target labels and annotations metrics to be monitored by using the metricLabelsAllowlist and metricAnnotationsAllowList options described here in your Kubernetes clusters.

The ksm.config object has the following structure:

ksm:
  config:
   # When autodiscovering KSM, force the following scheme. By default, `http` is used.
   scheme: "http"
   # Label selector to find kube-state-metrics endpoints. Defaults to `app.kubernetes.io/name=kube-state-metrics`.
   selector: "app.kubernetes.io/name=kube-state-metrics"
   # Restrict KSM discovery to this particular namespace. Defaults to all namespaces.
   namespace: ""
   # When autodiscovering, only consider endpoints that use this port. By default, all ports from the discovered `endpoint` are probed.
   #port: 8080
   # Override autodiscovery mechanism completely and specify the KSM url directly instead
   #staticUrl: "http://test.io:8080/metrics"

Control plane configuration

Control plane configuration has changed substantially. If you previously had control plane monitoring enabled, we encourage you to take a look at the Configure control plane monitoring dedicated page.

The following options have been replaced by more comprehensive configuration, covered in the section linked above:

• apiServerSecurePort
• etcdTlsSecretName
• etcdTlsSecretNamespace
• controllerManagerEndpointUrl, etcdEndpointUrl, apiServerEndpointUrl, and schedulerEndpointUrl

Agent configuration

Agent config file, previously specified in config has been moved to common.agentConfig. Format of the file has not changed, and the full range of options that can be configured can be found here.

The following agent options were previously "aliased" in the root of the values.yml file, and are no longer available:

• logFile has been replaced by common.agentConfig.log_file.
• eventQueueDepth has been replaced by common.agentConfig.event_queue_depth.
• customAttributes has changed in format to a yaml object. The previous format, a manually json-encoded string e.g. {"team": "devops"} is deprecated.
• Previously, customAttributes had a default clusterName entry which might have unwanted consequences if removed. This is no longer the case, users may now safely override customAttributes on its entirety.
• discoveryCacheTTL has been completely removed, as the discovery is now performed using kubernetes informers which have a built-in cache.

Integrations configuration

Integrations were previously configured under integrations_config, using an array format:

integrations_config:
  - name: nri-redis.yaml
    data:
      discovery:  # ...
      integrations:  # ...

The mechanism remains the same, but we have changed the format to be more user-friendly:

integrations:
  nri-redis-sampleapp:
    discovery:  # ...
    integrations:  # ...

Moreover, now the --port and --tls flags are mandatory on the discovery command. In the past, the following would work:

integrations:
  nri-redis-sampleapp:
    discovery:
      command:
        exec: /var/db/newrelic-infra/nri-discovery-kubernetes

From v3 onwards, you must specify --port and --tls:

integrations:
  nri-redis-sampleapp:
    discovery:
      command:
        exec: /var/db/newrelic-infra/nri-discovery-kubernetes --tls --port 10250

This change is required because in v2 and below, the nrk8s-kubelet component (or its equivalent) ran with hostNetwork: true, so nri-discovery-kubernetes could connect to the kubelet using localhost and plain http. For security reasons, this is no longer the case, hence the need to specify both flags from now on.

For more details on how to configure on-host integrations in Kubernetes please check the Monitor services in Kubernetes page.

Miscellaneous chart values

While not related to the integration configuration, the following miscellaneous options for the helm chart have also changed:

• runAsUser has been replaced by securityContext, which is templated directly into the pods and more configurable.
• resources has been removed, as now we deploy three different workloads. Resources for each one can be configured individually under:
• ksm.resources
• kubelet.resources
• controlPlane.resources
• Similarly, tolerations has been split into three and the previous one is no longer valid:
• ksm.tolerations
• kubelet.tolerations
• controlPlane.tolerations

• All three default to tolerate any value for NoSchedule and NoExecute

• image and all its subkeys have been replaced by individual sections for each of the three images that are now deployed:
• images.forwarder.* to configure the infrastructure-agent forwarder.
• images.agent.* to configure the image bundling the infrastructure-agent and on-host integrations.
• images.integration.* to configure the image in charge of scraping k8s data.

Upgrade from v2

In order to upgrade from the Kubernetes integration version 2 (included in nri-bundle chart versions 3.x), we strongly encourage you to create a values-newrelic.yaml file with your desired license key and configuration. If you had previously installed our chart from the CLI directly, for example using a command like the following:

$
helm install newrelic/nri-bundle \
>
--set global.licenseKey=<New Relic License key> \
>
--set global.cluster=<Cluster name> \
>
--set infrastructure.enabled=true \
>
--set prometheus.enabled=true \
>
--set webhook.enabled=true \
>
--set ksm.enabled=true \
>
--set kubeEvents.enabled=true \
>
--set logging.enabled=true

You can take the provided --set arguments and put them in a yaml file like the following:

# values-newrelic.yaml
global:
  licenseKey: <New Relic License key>
  cluster: <Cluster name>
infrastructure:
  enabled: true
prometheus:
  enabled: true
webhook:
  enabled: true
ksm:
  enabled: true
kubeEvents:
  enabled: true
logging:
  enabled: true

After doing this, and adapting any other setting you might have changed according to the section above, you can upgrade by running the following command:

$
helm upgrade newrelic newrelic/nri-bundle \
>
--namespace newrelic --create-namespace \
>
-f values-newrelic.yaml