Understand and use data

This document explains how to find and use data reported from the New Relic Kubernetes integration.

Find and use data

To view the Kubernetes integration's dashboard:

  1. Go to infrastructure.newrelic.com > Third-party services.
  2. Select the Kubernetes dashboard link to open the Kubernetes dashboard.
  3. To create your own dashboards, go to insights.newrelic.com and create NRQL queries.

Kubernetes data is attached to the following event types.

Event name Type of Kubernetes data
K8sNodeSample Node data
K8sNamespaceSample Namespace data
K8sDeploymentSample Deployment data
K8sReplicasetSample Replica set data
K8sPodSample Pod data
K8sClusterSample Cluster data
K8sContainerSample Container data
K8sVolumeSample Volume data

Manage alerts

You can be notified about alert violations for your Kubernetes data:

Create an alert condition

To create an alert condition for the Kubernetes integration:

  1. Go to infrastructure.newrelic.com > Settings > Alerts > Kubernetes, then select Create alert condition.
  2. To filter the alert to Kubernetes entities that only have the chosen attributes, select Filter.
  3. Select the threshold settings. For more on the Trigger an alert when... options, see Alert types.
  4. Select an existing alert policy, or create a new one.
  5. Select Create.

When an alert condition's threshold is triggered, New Relic sends a notification to the policy's notification channels.

Kubernetes integration alert condition
infrastructure.newrelic.com > Settings > Alerts > Kubernetes > Create alert condition: Infrastructure includes alert conditions specific to Kubernetes.
Use alert types and thresholds

To use any of the available Kubernetes-specific alert criteria, select the Kubernetes alert type:

Kubernetes alert types Comments
Available pods are less than desired pods

This alert type monitors replica sets. The alert triggers if the number of available replicas (pods) for a deployment is less than the number of replicas you chose when creating the deployment. This can happen if there are not enough resources in your cluster to schedule all pods for a deployment. New Relic applies the alert conditions individually to each deployment matching the specified filter.

Container CPU usage

This alert type compares the CPU consumption of a container with the limit that you defined when it was created. The alert triggers if usage exceeds the threshold. Container CPU usage is defined as:

(CPU cores used / CPU cores limit) * 100
Container memory usage

This alert type compares the memory consumption of a container with the limit that was defined when it was created. The alert triggers if usage exceeds the threshold. Container memory usage is defined as:

(memory used / memory limit) * 100

In addition, you can create an alert condition for any metric collected by any New Relic integration you use, including the Kubernetes integration:

  1. Select the alert type Integrations.
  2. From the Select a data source dropdown, select a Kubernetes (K8s) data source.
Select alert notifications

When an alert condition's threshold is triggered, New Relic sends a message to the notification channel(s) chosen in the alert policy. Depending on the type of notification, you may have the following options:

The entity identifier that triggered the alert appears near the top of the notification message. The format of the identifier depends on the alert type:

  • Available pods are less than desired pods alerts:

    K8s:CLUSTER_NAME:PARENT_NAMESPACE:replicaset:REPLICASET_NAME
  • CPU or memory usage alerts:

    K8s:CLUSTER_NAME:PARENT_NAMESPACE:POD_NAME:container:CONTAINER_NAME

Here are some examples.

Pod alert notification example

For Available pods are less than desired pods alerts, the ID of the replica set triggering the issue might look like this:

k8s:beam-production:default:replicaset:nginx-deployment-1623441481

This identifier contains the following information:

  • Cluster name: beam-production
  • Parent namespace: default
  • ReplicaSet name: nginx-deployment-1623441481
Container resource notification example

For container CPU or memory usage alerts, the entity might look like this:

k8s:beam-production:kube-system:kube-state-metrics-797bb87c75-zncwn:container:kube-state-metrics

This identifier contains the following information:

  • Cluster name: beam-production
  • Parent namespace: kube-system
  • Pod namespace: kube-state-metrics-797bb87c75-zncwn
  • Container name: kube-state-metrics
Create alert conditions using NRQL

Follow standard procedures to create alert conditions for NRQL queries.

Kubernetes attributes and metrics

The Kubernetes integration collects the following metrics and other attributes. For more on using integration data, see Find and use data.

Node data

Query the K8sNodeSample event in New Relic Insights for node data:

Node attribute Description
clusterName Name that you assigned to the cluster when you installed the Kubernetes integration.
cpuUsedCoreMilliseconds Node CPU usage measured in core milliseconds.
cpuUsedCores Node CPU usage measured in cores.
fsAvailableBytes Bytes available in the node filesystem.
fsCapacityBytes Total capacity of the node filesystem in bytes.
fsInodes Total number of inodes in the node filesystem.
fsInodesFree Free inodes in the node filesystem.
fsInodesUsed Used inodes in the node filesystem.
fsUsedBytes Used bytes in the node filesystem.
memoryAvailableBytes Bytes of memory available in the node.
memoryMajorPageFaultsPerSecond Number of major page faults per second in the node.
memoryRssBytes Bytes of rss memory.
memoryUsedBytes Bytes of memory used.
memoryWorkingSetBytes Bytes of memory in the working set.
net.errorCountPerSecond Number of errors per second while receiving/transmitting over the network.
nodeName Host name that the pod is running on.
runtimeAvailableBytes Bytes available to the container runtime filesystem.
runtimeCapacityBytes Total capacity assigned to the container runtime filesystem in bytes.
runtimeInodes Total number of inodes in the container runtime filesystem.
runtimeInodesFree Free inodes in the container runtime filesystem.
runtimeInodesUsed Used inodes in the container runtime filesystem.
runtimeUsedBytes Used bytes in the container runtime filesystem.

Namespace data

Query the K8sNamespaceSample event in New Relic Insights for namespace data:

Namespace attribute Description
clusterName Name that you assigned to the cluster when you installed the Kubernetes integration.
createdAt Timestamp of the namespace when it was created.
namespace Name of the namespace to be used as an identifier.
label.LABEL_NAME Labels associated with your namespace, so you can filter and query for specific namespaces.
status

Current status of the namespace.

The value can be Active or Terminated.

Deployment data

Query the K8sDeploymentSample event in New Relic Insights for deployment data:

Deployment attribute Description
clusterName Name that you assigned to the cluster when you installed the Kubernetes integration.
createdAt Timestamp of when the deployment was created.
deploymentName Name of the deployment to be used as an identifier.
namespace Name of the namespace that the deployment belongs to.
label.LABEL_NAME Labels associated with your deployment, so you can filter and query for specific deployments.
podsAvailable Number of replicas that are currently available.
podsDesired Number of replicas that you defined in the deployment.
podsTotal Total number of replicas that are currently running.
podsUnavailable Number of replicas that are currently unavailable.
podsUpdated Number of replicas that have been updated to achieve the desired state of the deployment.
updatedAt Timestamp of when the deployment was updated.

Replica set data

Query the K8sReplicasetSample event in New Relic Insights for replica set data:

Replica attribute Description
clusterName Name that you assigned to the cluster when you installed the Kubernetes integration.
createdAt Timestamp of when the replica set was created.
deploymentName Name of the deployment to be used as an identifier.
namespace Name of the namespace that the replica set belongs to.
observedGeneration Integer representing generation observed by the replica set.
podsDesired Number of replicas that you defined in the deployment.
podsFullyLabeled Number of pods that have labels that match the replica set pod template labels.
podsReady Number of replicas that are ready for this replica set.
podsTotal Total number of replicas that are currently running.
replicasetName Name of the replica set to be used as an identifier.

Pod data

Query the K8sPodSample event in New Relic Insights for pod data:

Pod attribute Description
clusterName Name that you assigned to the cluster when you installed the Kubernetes integration.
createdAt Timestamp of when the pod was created.
createdBy Name of the Kubernetes object that created the pod.

For example, newrelic-infra.

createdKind Kind of Kubernetes object that created the pod.

For example, DaemonSet.

deploymentName Name of the deployment to be used as an identifier.
isReady Boolean representing whether or not the pod is ready to serve requests.
isScheduled Boolean representing whether or not the pod has been scheduled to run on a node.
label.LABEL_NAME Labels associated with your pod, so you can filter and query for specific pods.
message Details related to the last pod status change.
namespace Name of the namespace that the replica set belongs to.
net.errorCountPerSecond Number of errors per second while receiving/transmitting over the network.
net.rxBytesPerSecond Number of bytes per second received over the network.
net.txBytesPerSecond Number of bytes per second transmitted over the network.
nodeIP Host IP address that the pod is running on.
nodeName Host name that the pod is running on.
podName Name of the pod to be used as an identifier.
reason Reason why the pod is in the current status.
startTime Timestamp of when the pod started running.
status Current status of the pod.

The value can be Pending, Running, Succeeded, Failed, Unknown.

Cluster data

Query the K8sClusterSample event in New Relic Insights to see cluster data:

Cluster attribute Description
clusterName Name that you assigned to the cluster when you installed the Kubernetes integration.

Container data

Query the K8sContainerSample event in New Relic Insights for container data:

Container attribute Description
clusterName Name that you assigned to the cluster when you installed the Kubernetes integration.
containerID Unique ID associated with the container. If you are running Docker, this is the Docker container id.
containerImage Name of the image that the container is running.
containerImageID Unique ID associated with the image that the container is running.
containerName Name associated with the container.
cpuLimitCores Integer representing limit CPU cores defined for the container in the pod specification.
cpuRequestedCores Requested CPU cores defined for the container in the pod specification.
cpuUsedCores CPU cores actually used by the container.
deploymentName Name of the deployment to be used as an identifier.
isReady Boolean. Whether or not the container's readiness check succeeded.
label.LABEL_NAME Labels associated with your container, so you can filter and query for specific containers.
memoryLimitBytes Integer representing limit bytes of memory defined for the container in the pod specification.
memoryRequestedBytes Integer. Requested bytes of memory defined for the container in the pod specification.
memoryUsedBytes Integer. Bytes of memory actually used by the container.
namespace Name of the namespace that the container belongs to.
nodeIP Host IP address the container is running on.
nodeName Host name that the container is running on.
podName Name of the pod that the container is in, to be used as an identifier.
reason Provides a reason why the container is in the current status.
restartCount Number of times the container has been restarted.
status Current status of the container.

The value can be Running, Terminated, or Unknown.

Volume data

Query the K8sVolumeSample event in New Relic Insights for volume data:

Volume attribute Description
volumeName Name that you assigned to the volume at creation.
clusterName Cluster where the volume is configured.
namespace Namespace where the volume is configured.
podName The pod that the volume is attached to. The Kubernetes monitoring integration lists Volumes that are attached to a pod.
persistent If this is a persistent volume, this value is set to "true".
pvcNamespace Namespace where the Persistent Volume Claim is configured.
pvcName Name that you assigned to the Persistent Volume Claim at creation.
fsCapacityBytes Capacity of the volume in Bytes.
fsUsedBytes Usage of the volume in Bytes.
fsAvailableBytes Capacity available of the volume in Bytes.
fsUsedPercent Usage of the volume in Percentage.
fsInodes Total Inodes of the volume.
fsInodesUsed Inodes used in the volume.
fsInodesFree Inodes available in the volume.

Volume data is available for volume plugins that implement the MetricsProvider interface:

  • AWSElasticBlockStore
  • AzureDisk
  • AzureFile
  • Cinder
  • Flexvolume
  • Flocker
  • GCEPersistentDisk
  • GlusterFS
  • iSCSI
  • StorageOS
  • VsphereVolume

Kubernetes metadata in APM-monitored applications

By linking your applications with Kubernetes, the following attributes are added to application trace and distributed trace:

  • nodeName
  • containerName
  • podName
  • clusterName
  • deploymentName
  • namespaceName

For more help

Other Kubernetes integration resources:

  • For discussions about the Kubernetes integration, visit New Relic's Explorers Hub.