Monitor services running on Kubernetes

With New Relic's Kubernetes integration you can monitor both Kubernetes and the services running on it, such as Cassandra, Redis, MySQL, and other supported services.

Get started

Our Kubernetes integration comes bundled with some of our on-host integrations (like Cassandra, MySQL, and Apache). This lets you get data for those supported services by adding a section to the Kubernetes integration's configuration, which lives as a ConfigMap inside a manifest.

For an example of how to monitor Redis running on a Kubernetes PHP Guestbook, see this tutorial.

What you need

To monitor services running on Kubernetes, you only need a Kubernetes cluster running the Kubernetes integration, version 1.13.0 or higher (install | check version | update).

We support the following services running on Kubernetes:

Enable monitoring of services

To enable our Kubernetes integration to monitor one or more services:

  1. Expand this dropdown and get the YAML snippets for the service(s) you want to monitor:

    Get the config YAML for the integration

    For the services you want to monitor, follow the links to GitHub to get the YAML snippets you'll need for the next step:

  2. Add the snippet to the Kubernetes integration's ConfigMap, after the data: section:
    Example configuration

    This example shows the YAML config for the Apache integration (highlighted) added to the Kubernetes integration's config. Respect the indentation levels.

    ---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nri-integration-cfg
  namespace: default
data:
  apache-config.yaml: |
    ---
    # Run auto discovery to find pods with label "app=apache"
    # https://docs.newrelic.com/docs/integrations/host-integrations/installation/container-auto-discovery
    discovery:
      command:
        # Use the optional arguments:
        # --namespaces: Comma separated namespaces to discover pods on
        # --tls: Use secure (TLS) connection
        # --port: Port used to connect to the kubelet. Default is 10255
        exec: /var/db/newrelic-infra/nri-discovery-kubernetes --port PORT --tls
        match:
          label.app: apache
    integrations:
      - name: nri-apache
        env:
          # Use the discovered IP as the host address
          STATUS_URL: http://${discovery.ip}/server-status?auto
          METRICS: 1

    You can add snippets for multiple services to the same config file. See an example.

  3. Depending on your environment, you may need or want to set additional config options. Expand the dropdown below for links to configuration options.

    Select a service to see available config options:

  4. Verify monitoring is enabled: Go to one.newrelic.com > Infrastructure, select Third party services, and then select the service's dashboard. You should see data being reported.

Additional notes about enabling services:

  • Enabling multiple services may use more resources than what is set in the resource limits of the Kubernetes integration config file. If this becomes an issue, raise the limit in the resources section.
  • The Kubernetes integration does not automatically update. For best results, regularly update.

If you installed our Kubernetes integration using Helm, to monitor services you need to update the existing installation with the new configuration, which contains the services to monitor:

helm upgrade --reuse-values -f values.yaml [RELEASE] [CHART]

If you use nri-bundle charts, you need to update the children's chart values. Find some examples here.

More resources for learning about configuration:

Manually configure service monitoring

The enable procedure should be all you need to get monitoring working, but if you run into problems, understanding some technical details about configuration can be helpful. This section goes into more detail about how configuration works.

For each service you wish to monitor, you must add a configuration file for that integration to our Kubernetes integration's configuration. This document will cover these subjects:

How the service-specific YAML config works

Our Kubernetes integration's configuration follows the ConfigMap format. Using a ConfigMap allows us to decouple the configuration for the integrations from the Kubernetes image. The other benefit is that a ConfigMap can be updated automatically without reloading the running container.

Because the infrastructure agent uses YAML to configure its associated integrations, ConfigMaps are a good choice for storing YAML. (For more information on config file format, see the Integration config file format.)

The Kubernetes integration image comes with an auto-discovery feature that simplifies the configuration of multiple instances of services using a single configuration file. For example, if you have several NGINX instances running, creating an NGINX integration configuration file for every instance would be hard to implement and hard to update. With our auto-discovery option, you can discover and monitor all your NGINX instances with a single configuration file.

Each integration has its own specific configuration YAML. Our NGINX integration default config file looks like this:

  nginx-config.yml: |
    ---
    discovery:
      command:
        exec: /var/db/newrelic-infra/nri-discovery-kubernetes
        match:
          label.app: nginx
    integrations:
      - name: nri-nginx
        env:
          STATUS_URL: http://${discovery.ip}/status
          STATUS_MODULE: discover
          METRICS: 1

The above config enables the following:

  • Runs nri-discovery-kubernetes to query the data for the node we are currently on.
  • Parses the data that comes back and looks for any Kubernetes pod that has a Kubernetes container with an app= label with value nginx.
  • For any matches, it attempts to run the NGINX integration. The status URL is built from:
    • The pod's IP address
    • The status page is pulled from the label on K8s pod called status_url

This automatic discovery works the same as the container auto-discovery used by the infrastructure agent. For more advanced options, see Container auto-discovery.

Add a service YAML to the Kubernetes integration config

It's best practice to configure enabled integrations alongside the Kubernetes integration configuration. This is easier than maintaining configuration files for every single service/integration instance.

Below is an example of a Kubernetes integration's ConfigMap. The highlighted section shows where an integration configuration YAML (in this case, NGINX) is placed.

For more information on discovery:, see Container auto-discovery for on-host integrations.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nri-integration-cfg
  namespace: default
data:
  nginx-config.yml: |
    ---
    # Run auto discovery to find pods with label "app=nginx"
    # https://docs.newrelic.com/docs/integrations/host-integrations/installation/container-auto-discovery
    discovery:
      command:
        # Use the following optional arguments:
        # --namespaces: Comma separated list of namespaces to discover pods on
        # --tls: Use secure (TLS) connection
        # --port: Port used to connect to the kubelet. Default is 10255
        exec: /var/db/newrelic-infra/nri-discovery-kubernetes --port PORT --tls
        match:
          label.app: nginx
    integrations:
      - name: nri-nginx
        env:
          # If you're using ngx_http_api_module be certain to use the full path up to and including the version number
          # Use the discovered IP as the host address
          STATUS_URL: http://${discovery.ip}/status
          # Comma separated list of ngx_http_api_module, NON PARAMETERIZED, Endpoints
          # endpoints: /nginx,/processes,/connections,/ssl,/slabs,/http,/http/requests,/http/server_zones,/http/caches,/http/upstreams,/http/keyvals,/stream,/stream/server_zones,/stream/upstreams,/stream/keyvals,/stream/zone_sync
          # Name of Nginx status module OHI is to query against. discover | ngx_http_stub_status_module | ngx_http_status_module | ngx_http_api_module
          STATUS_MODULE: discover
          METRICS: 1

This configuration map can then be referenced in the DaemonSet, the same as the one that was generated via the command line.

Make sure the namespace used is the same one used by the Kubernetes integration manifest. If you haven't changed it in the downloaded manifest file, the value is default.

Add multiple services to the same config

You can monitor several services using the same Kubernetes integration config file. To do this, add another integration configuration YAML to the same Kubernetes integration config file. Below is the Kubernetes config created in the last section, with a new section for the Cassandra integration's config (highlighted).

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nri-integration-cfg
  namespace: default
data:
  nginx-config.yml: |
    ---
    # Run auto discovery to find pods with label "app=nginx"
    # https://docs.newrelic.com/docs/integrations/host-integrations/installation/container-auto-discovery
    discovery:
      command:
        # Run discovery for Kubernetes. Use the following optional arguments:
        # --namespaces: Comma separated list of namespaces to discover pods on
        # --tls: Use secure (TLS) connection
        # --port: Port used to connect to the kubelet. Default is 10255
        exec: /var/db/newrelic-infra/nri-discovery-kubernetes --port PORT --tls
        match:
          label.app: nginx
    integrations:
      - name: nri-nginx
        env:
          # If you're using ngx_http_api_module be certain to use the full path up to and including the version number
          # Use the discovered IP as the host address
          STATUS_URL: http://${discovery.ip}/status
          # Comma separated list of ngx_http_api_module, NON PARAMETERIZED, Endpoints
          # endpoints: /nginx,/processes,/connections,/ssl,/slabs,/http,/http/requests,/http/server_zones,/http/caches,/http/upstreams,/http/keyvals,/stream,/stream/server_zones,/stream/upstreams,/stream/keyvals,/stream/zone_sync
          # Name of Nginx status module OHI is to query against. discover | ngx_http_stub_status_module | ngx_http_status_module | ngx_http_api_module
          STATUS_MODULE: discover
          METRICS: 1
  cassandra-configuration.yml: |
    ---
    # Run auto discovery to find pods with label "app=cassandra"
    # https://docs.newrelic.com/docs/integrations/host-integrations/installation/container-auto-discovery
    discovery:
      command:
        # Run discovery for Kubernetes. Use the following optional arguments:
        # --namespaces: Comma separated list of namespaces to discover pods on
        # --tls: Use secure (TLS) connection
        # --port: Port used to connect to the kubelet. Default is 10255
        exec: /var/db/newrelic-infra/nri-discovery-kubernetes --port PORT --tls
        match:
          label.app: cassandra
    integrations:
      - name: nri-cassandra
        env:
          # Use the discovered IP as the host address
          HOSTNAME: ${discovery.ip}
          PORT: 7199
          USERNAME: cassandra
          PASSWORD: cassandra
          METRICS: 1/mark

The Kubernetes integration config is now set up to monitor these two services. Additionally, depending on your environment, there may be some additional service-specific configuration you must do.

When you've completed configuration, our infrastructure agent looks for any pod with a label cassandra and runs the integration against it.

For more help

If you need more help, check out these support and learning resources: