Monitor services running on Kubernetes

BETA

With New Relic's Kubernetes integration you can monitor both Kubernetes and the services running on it, such as Cassandra, Redis, MySQL, and other supported services.

For a tutorial, see Tutorial: Enable monitoring of Redis running on Kubernetes.

Get started

This feature is in beta. For questions and feedback, contact the product manager at jjoly @ newrelic.com.

Our Kubernetes integration comes bundled with some of our on-host integrations (like Cassandra, MySQL, and Apache). This lets you get data for those supported services by adding a configuration section to the Kubernetes integration's configuration file.

For an example of how to monitor Redis running on a Kubernetes PHP Guestbook, see this tutorial.

Requirements

Enable service monitoring

To enable our Kubernetes integration to monitor one or more services:

  1. Expand this dropdown and get the YAML snippets for the service(s) you want to monitor:

    Get integration config YAML

    For the services you want to monitor, follow the links to GitHub to get the YAML snippets you'll need for the next step:

  2. Add the snippet to the Kubernetes integration's configuration file, after the data: section:
    Example configuration

    This example shows the YAML config for the Apache integration (highlighted) added to the Kubernetes integration's configuration file. Respect the indentation levels.

    ---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nri-integration-cfg
  namespace: default
data:
  apache-config.yaml: |
    ---
    # Run auto discovery to find pods with label "app=apache"
    # https://docs.newrelic.com/docs/integrations/host-integrations/installation/container-auto-discovery
    discovery:
      command:
        # Run NRI Discovery for Kubernetes
        # https://github.com/newrelic/nri-discovery-kubernetes
        exec: /var/db/newrelic-infra/nri-discovery-kubernetes
        match:
          label.app: apache
    integrations:
      - name: nri-apache
        env:
          # Use the discovered IP as the host address
          STATUS_URL: http://${discovery.ip}/server-status?auto
          METRICS: 1

    You can add snippets for multiple services to the same config file. See an example.

  3. Depending on your environment, you may need or want to set additional config options. Expand the dropdown below for links to configuration options.

    Select a service to see available config options:

  4. Verify monitoring is enabled: Go to infrastructure.newrelic.com, select Third party services, and select the service's dashboard. You should see data being reported.

Additional notes about enabling services:

  • Enabling multiple services may use more resources than what is set in the resource limits of the Kubernetes integration config file. If this becomes an issue, raise the limit in the config file resources section.
  • The Kubernetes integration does not automatically update. For best results, regularly update.

More resources for learning about configuration:

Details on configuration

This section goes into more detail about the configuration process that's part of the enable monitoring instructions.

For each service you wish to monitor, you must add a configuration file for that integration to our Kubernetes integration's configuration. This section will give more detail about these subjects:

How the service-specific YAML config works

Our Kubernetes integration's configuration follows the ConfigMap format. Using a ConfigMap allows us to decouple the configuration for the integrations from the Kubernetes image. The other benefit is that a ConfigMap can be updated automatically without reloading the running container.

Because the Infrastructure agent uses YAML to configure its associated integrations, ConfigMaps are a good choice for storing YAML. (For more information on config file format, see the Integration config file format.)

The Kubernetes integration image comes with an auto-discovery feature that simplifies the configuration of multiple instances of services using a single configuration file. For example, if you have several NGINX instances running, creating an NGINX integration configuration file for every instance would be hard to implement and hard to update. With our auto-discovery option, you can discover and monitor all your NGINX instances with a single configuration file.

Each integration has its own specific configuration YAML. Our NGINX integration default config file looks like this:

  nginx-config.yml: |
    ---
    # Run auto discovery to find pods with label "app=nginx"
    # https://docs.newrelic.com/docs/integrations/host-integrations/installation/container-auto-discovery
    discovery:
      command:
        # Run NRI Discovery for Kubernetes
        # https://github.com/newrelic/nri-discovery-kubernetes
        exec: /var/db/newrelic-infra/nri-discovery-kubernetes
        match:
          label.app: nginx
    integrations:
      - name: nri-nginx
        env:
          # If you're using ngx_http_api_module be certain to use the full path up to and including the version number
          # Use the discovered IP as the host address
          STATUS_URL: http://${discovery.ip}/status
          # Comma separated list of ngx_http_api_module, NON PARAMETERIZED, Endpoints
          # endpoints: /nginx,/processes,/connections,/ssl,/slabs,/http,/http/requests,/http/server_zones,/http/caches,/http/upstreams,/http/keyvals,/stream,/stream/server_zones,/stream/upstreams,/stream/keyvals,/stream/zone_sync
          # Name of Nginx status module OHI is to query against. discover | ngx_http_stub_status_module | ngx_http_status_module | ngx_http_api_module
          STATUS_MODULE: discover
          METRICS: 1

The above config enables the following:

  • Runs nri-discovery-kubernetes to query the data for the node we are currently on.
  • Parses the data that comes back and looks for any Kubernetes Pod that has a Kubernetes container with an app= label with value nginx.
  • For any matches, it attempts to run the NGINX integration. The status URL is built from:
    • The Pod's IP address
    • The status page is pulled from the label on K8s Pod called status_url

This automatic discovery works the same as the container auto-discovery used by the Infrastructure agent. For more advanced options, see Container auto-discovery.

New Relic integrations can contain two different config-related files: a configuration file and a definition file. Here, we explain only how to set up the configuration file. This is because the Kubernetes integration already has definition files included; if you want to customize these, you can do that by passing custom values via CLI arguments within the exec field in the configuration side.

Add a service YAML to the Kubernetes integration config

It's best practice to configure enabled integrations alongside the Kubernetes integration configuration. This is easier than maintaining configuration files for every single service/integration instance.

Below is an example of a Kubernetes integration's ConfigMap-format configuration file. The highlighted section shows where an integration configuration YAML (in this case, NGINX) is placed.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nri-integration-cfg
  namespace: default
data:
  nginx-config.yml: |
    ---
    # Run auto discovery to find pods with label "app=nginx"
    # https://docs.newrelic.com/docs/integrations/host-integrations/installation/container-auto-discovery
    discovery:
      command:
        # Run NRI Discovery for Kubernetes
        # https://github.com/newrelic/nri-discovery-kubernetes
        exec: /var/db/newrelic-infra/nri-discovery-kubernetes
        match:
          label.app: nginx
    integrations:
      - name: nri-nginx
        env:
          # If you're using ngx_http_api_module be certain to use the full path up to and including the version number
          # Use the discovered IP as the host address
          STATUS_URL: http://${discovery.ip}/status
          # Comma separated list of ngx_http_api_module, NON PARAMETERIZED, Endpoints
          # endpoints: /nginx,/processes,/connections,/ssl,/slabs,/http,/http/requests,/http/server_zones,/http/caches,/http/upstreams,/http/keyvals,/stream,/stream/server_zones,/stream/upstreams,/stream/keyvals,/stream/zone_sync
          # Name of Nginx status module OHI is to query against. discover | ngx_http_stub_status_module | ngx_http_status_module | ngx_http_api_module
          STATUS_MODULE: discover
          METRICS: 1

This configuration map can then be referenced in the Daemon set, the same as the one that was generated via the command line.

Make sure the namespace used is the same one used by the Kubernetes integration manifest. If you haven't changed it in the downloaded manifest file, the value is default.

Add multiple services to the same config

You can monitor several services using the same Kubernetes integration config file. To do this, add another integration configuration YAML to the same Kubernetes integration config file. Below is the Kubernetes config created in the last section, with a new section for the Cassandra integration's config (highlighted).

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nri-integration-cfg
  namespace: default
data:
  nginx-config.yml: |
    ---
    # Run auto discovery to find pods with label "app=nginx"
    # https://docs.newrelic.com/docs/integrations/host-integrations/installation/container-auto-discovery
    discovery:
      command:
        # Run NRI Discovery for Kubernetes
        # https://github.com/newrelic/nri-discovery-kubernetes
        exec: /var/db/newrelic-infra/nri-discovery-kubernetes
        match:
          label.app: nginx
    integrations:
      - name: nri-nginx
        env:
          # If you're using ngx_http_api_module be certain to use the full path up to and including the version number
          # Use the discovered IP as the host address
          STATUS_URL: http://${discovery.ip}/status
          # Comma separated list of ngx_http_api_module, NON PARAMETERIZED, Endpoints
          # endpoints: /nginx,/processes,/connections,/ssl,/slabs,/http,/http/requests,/http/server_zones,/http/caches,/http/upstreams,/http/keyvals,/stream,/stream/server_zones,/stream/upstreams,/stream/keyvals,/stream/zone_sync
          # Name of Nginx status module OHI is to query against. discover | ngx_http_stub_status_module | ngx_http_status_module | ngx_http_api_module
          STATUS_MODULE: discover
          METRICS: 1
  cassandra-configuration.yml: |
    ---
    # Run auto discovery to find pods with label "app=cassandra"
    # https://docs.newrelic.com/docs/integrations/host-integrations/installation/container-auto-discovery
    discovery:
      command:
        # Run NRI Discovery for Kubernetes
        # https://github.com/newrelic/nri-discovery-kubernetes
        exec: /var/db/newrelic-infra/nri-discovery-kubernetes
        match:
          label.app: cassandra
    integrations:
      - name: nri-cassandra
        env:
          # Use the discovered IP as the host address
          HOSTNAME: ${discovery.ip}
          PORT: 7199
          USERNAME: cassandra
          PASSWORD: cassandra
          METRICS: 1/mark

The Kubernetes integration config file is now set up to monitor these two services.

Additionally, depending on your environment, there may be some additional service-specific configuration you must do.

When you've completed configuration, our Infrastructure agent looks for any Pod with a label cassandra and runs the integration against it.

For another example, see the tutorial, which enables our Redis integration.

For more help

Recommendations for learning more: