Find and use your Kubernetes data

You can build your own charts and query all your Kubernetes integration data using the query builder and the NerdGraph API. Our integration collects Kubernetes data by instrumenting the container orchestration layer. For a simpler and more visual experience, use the cluster explorer.

one.newrelic.com > Dashboards: Using the query builder you can query your Kubernetes data and create clear visualizations.

Query Kubernetes data

The simplest way to query your Kubernetes data is using the query builder, which accepts NRQL queries. Alternatively, you can use the NerdGraph API to retrieve Kubernetes data.

Events and attributes

Kubernetes data is attached to the following events. You can learn more about this data in our data dictionary.

Event name	Type of Kubernetes data	Available since
`K8sNodeSample`	Node data	v1.0.0
`K8sNamespaceSample`	Namespace data	v1.0.0
`K8sDeploymentSample`	Deployment data	v1.0.0
`K8sReplicasetSample`	ReplicaSet data	v1.0.0
`K8sDaemonsetSample`	DaemonSet data	v1.13.0
`K8sStatefulsetSample`	StatefulSet data	v1.13.0
`K8sPodSample`	Pod data	v1.0.0
`K8sClusterSample`	Cluster data	v1.0.0
`K8sContainerSample`	Container data	v1.0.0
`K8sVolumeSample`	Volume data	v1.0.0
`K8sApiServerSample`	API server data	v1.11.0
`K8sControllerManagerSample`	Controller manager data	v1.11.0
`K8sSchedulerSample`	Scheduler data	v1.11.0
`K8sEtcdSample`	etcd data	v1.11.0
`K8sEndpointSample`	Endpoint data	v1.13.0
`K8sServiceSample`	Service data	v1.13.0
`K8sHpaSample`	Horizontal Pod Autoscaler data	v2.3.0

To learn more about these events and their attributes, see our data dictionary.

Kubernetes metadata in APM-monitored applications

By linking your applications with Kubernetes, the following attributes are added to application trace and distributed trace:

nodeName
containerName
podName
clusterName
deploymentName
namespaceName

Manage alerts

You can be notified about alert violations for your Kubernetes data:

To create an alert condition for the Kubernetes integration:

Go to one.newrelic.com > Infrastructure > Settings > Alerts & AI > Kubernetes, then select Create alert condition.
To filter the alert to Kubernetes entities that only have the chosen attributes, select Filter.
Select the threshold settings. For more on the Trigger an alert when... options, see Alert types.
Select an existing alert policy, or create a new one.
Select Create.
When an alert condition's threshold is triggered, New Relic sends a notification to the policy's notification channels.

The Kubernetes integration comes with its own alert policy and alert conditions. To see what the predefined alert conditions are, see Kubernetes integration: Predefined alert policy.

In addition, you can create an alert condition for any metric collected by any New Relic integration you use, including the Kubernetes integration:

Select the alert type Integrations.
From the Select a data source dropdown, select a Kubernetes (K8s) data source.

When an alert condition's threshold is triggered, New Relic sends a message to the notification channel(s) chosen in the alert policy. Depending on the type of notification, you may have the following options:

View the incident.
Acknowledge the incident.
Go to a chart of the incident data by selecting the identifier name.
The entity identifier that triggered the alert appears near the top of the notification message. The format of the identifier depends on the alert type:
Available pods are less than desired pods alerts:
```
K8s:CLUSTER_NAME:PARENT_NAMESPACE:replicaset:REPLICASET_NAME
```
CPU or memory usage alerts:
```
K8s:CLUSTER_NAME:PARENT_NAMESPACE:POD_NAME:container:CONTAINER_NAME
```
Here are some examples.
For Available pods are less than desired pods alerts, the ID of the ReplicaSet triggering the issue might look like this:
k8s:beam-production:default:replicaset:nginx-deployment-1623441481
This identifier contains the following information:
Cluster name: beam-production
Parent namespace: default
ReplicaSet name: nginx-deployment-1623441481
For container CPU or memory usage alerts, the entity might look like this:
k8s:beam-production:kube-system:kube-state-metrics-797bb87c75-zncwn:container:kube-state-metrics
This identifier contains the following information:
Cluster name: beam-production
Parent namespace: kube-system
Pod namespace: kube-state-metrics-797bb87c75-zncwn
Container name: kube-state-metrics

Reduce data ingest

Our charts support setting an option to reduce the amount of data ingested at the cost of granularity. To enable it, set global.lowDataMode to true in the nri-bundle chart.

lowDataMode affects four specific components of the nri-bundle chart outlined below.

New Relic Infrastructure

If lowDataMode is enabled, the default scrape interval changes from 15s to 30s. You can also specify a custom value for it using config.interval, which will take preference over lowDataMode.

Prometheus Agent Integration

If lowDataMode is enabled, the metrics that are prefixed with the following are excluded by default as they're already collected and used by the New Relic Kubernetes Integration.

- kube_
- container_
- machine_
- cadvisor_

New Relic Logging

If lowDataMode is enabled, Labels and Annotations are set to Off in the Filter section of the fluent-bit.conf file. This means that this detail will be dropped from the container log files which reduces the overall data ingest into New Relic.

The following fields are retained:

Allowlist_key  container_name
Allowlist_key  namespace_name
Allowlist_key  pod_name
Allowlist_key  stream
Allowlist_key  log

Low Data Mode Log Example

Complete Log Record

[
  {
    "cluster_name": "api-test",
    "kubernetes": {
      "annotations": {
        "kubernetes.io/psp": "eks.privileged"
      },
      "container_hash": "fryckbos/test@sha256:5b098eaf3c7d5b3585eb10cebee63665b6208bea31ef31a3f0856c5ffdda644b",
      "container_image": "fryckbos/test:latest",
      "container_name": "newrelic-logging",
      "docker_id": "134e1daf63761baa15e035b08b7aea04518a0f0e50af4215131a50c6a379a072",
      "host": "ip-192-168-17-123.ec2.internal",
      "labels": {
        "app": "newrelic-logging",
        "app.kubernetes.io/name": "newrelic-logging",
        "controller-revision-hash": "84db95db86",
        "pod-template-generation": "1",
        "release": "nri-bundle"
      },
      "namespace_name": "nrlogs",
      "pod_id": "54556e3e-719c-46b5-af69-020b75d69bf1",
      "pod_name": "nri-bundle-newrelic-logging-jxnbj"
    },
    "message": "[2021/09/14 12:30:49] [ info] [engine] started (pid=1)\n",
    "plugin": {
      "source": "kubernetes",
      "type": "fluent-bit",
      "version": "1.8.1"
    },
    "stream": "stderr",
    "time": "2021-09-14T12:30:49.138824971Z",
    "timestamp": 1631622649138
  }
]

Log Record after enabling lowDataMode.

[
  {
    "cluster_name": "api-test",
    "container_name": "newrelic-logging",
    "namespace_name": "nrlogs",
    "pod_name": "nri-bundle-newrelic-logging-jxnbj",
    "message": "[2021/09/14 12:30:49] [ info] [engine] started (pid=1)\n",
    "stream": "stderr",
    "timestamp": 1631622649138
  }
]

New Relic Pixie Integration

If lowDataMode is enabled, the newrelic-pixie integration performs heavier sampling on Pixie spans and reduces the collection interval from 10 seconds to 15 seconds.

lowDataMode settings:

HTTP_SPAN_LIMIT: 750
DB_SPAN_LIMIT: 250
COLLECT_INTERVAL_SEC: 15

The default settings for these parameters and others can be found in the newrelic-pixie-integration Github repo.

To take full advantage of New Relic's kubernetes integration, you'll need a free New Relic account. If you haven't already, create yours below to start monitoring your data today.

Query Kubernetes data

Events and attributes

Kubernetes metadata in APM-monitored applications

Manage alerts

Create an alert condition

Use the predefined alert types and thresholds

Select alert notifications

Pod alert notification example

Container resource notification example

Create alert conditions using NRQL

Reduce data ingest

New Relic Infrastructure

Prometheus Agent Integration

New Relic Logging

New Relic Pixie Integration