• /
  • Log in

Find and use your Kubernetes data

You can build your own charts and query all your Kubernetes integration data using the query builder and the NerdGraph API. Our integration collects Kubernetes data by instrumenting the container orchestration layer. For a simpler and more visual experience, use the cluster explorer.

New Relic One - Kubernetes dashboard

one.newrelic.com > Dashboards: Using the query builder you can query your Kubernetes data and create clear visualizations.

Query Kubernetes data

The simplest way to query your Kubernetes data is using the query builder, which accepts NRQL queries. Alternatively, you can use the NerdGraph API to retrieve Kubernetes data.

Event types

Kubernetes data is attached to the following event types:

Event name

Type of Kubernetes data

Available since

K8sNodeSample

Node data

v1.0.0

K8sNamespaceSample

Namespace data

v1.0.0

K8sDeploymentSample

Deployment data

v1.0.0

K8sReplicasetSample

ReplicaSet data

v1.0.0

K8sDaemonsetSample

DaemonSet data

v1.13.0

K8sStatefulsetSample

StatefulSet data

v1.13.0

K8sPodSample

Pod data

v1.0.0

K8sClusterSample

Cluster data

v1.0.0

K8sContainerSample

Container data

v1.0.0

K8sVolumeSample

Volume data

v1.0.0

K8sApiServerSample

API server data

v1.11.0

K8sControllerManagerSample

Controller manager data

v1.11.0

K8sSchedulerSample

Scheduler data

v1.11.0

K8sEtcdSample

ETCD data

v1.11.0

K8sEndpointSample

Endpoint data

v1.13.0

K8sServiceSample

Service data

v1.13.0

K8sHpaSample

Horizontal Pod Autoscaler data

v2.3.0

Manage alerts

You can be notified about alert violations for your Kubernetes data:

Kubernetes attributes and metrics

The Kubernetes integration collects the following metrics and other attributes.

Node data

Query the K8sNodeSample event for node data:

Node attribute

Description

allocatableCpuCores

Node allocatable CPU cores

allocatableMemoryBytes

Node allocatable memory bytes

allocatablePods

Node allocatable pods

allocatableEphemeralStorageBytes

Node allocatable ephemeral-storage bytes

capacityCpuCores

Node CPU capacity

capacityMemoryBytes

Node memory capacity (in bytes)

capacityPods

Pod capacity of the node

capacityEphemeralStorageBytes

Node ephemeral-storage capacity

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration

condition.{conditionName}={conditionValue}

Status of the current observed node condition. The reported conditions can vary depending on your Kubernetes flavor and installed operators.
Examples of common conditions are: Ready, DiskPressure, MemoryPressure, PIDPressure and NetworkUnavailable.
Condition values can be 1 (true), 0 (false), or -1 (unknown).

cpuUsedCoreMilliseconds

Node CPU usage measured in core milliseconds

cpuUsedCores

Node CPU usage measured in cores

cpuRequestedCores

Total amount of CPU cores requested

allocatableCpuCoresUtilization

Percentage of CPU cores actually used with respect to the CPU cores allocatable

fsAvailableBytes

Bytes available in the node filesystem

fsCapacityBytes

Total capacity of the node filesystem in bytes

fsInodes

Total number of inodes in the node filesystem

fsInodesFree

Free inodes in the node filesystem

fsInodesUsed

Used inodes in the node filesystem

fsUsedBytes

Used bytes in the node filesystem

fsCapacityUtilization

Percentage of used bytes in the node filesystem with respect to the capacity

memoryAvailableBytes

Bytes of memory available in the node

memoryMajorPageFaultsPerSecond

Number of major page faults per second in the node

memoryPageFaults

Number of page faults in the node

memoryRssBytes

Bytes of rss memory

memoryUsedBytes

Bytes of memory used

memoryWorkingSetBytes

Bytes of memory in the working set

memoryRequestedBytes

Total amount of requested memory

allocatableMemoryUtilization

Percentage of bytes of memory in the working set with respect to the node allocatable memory

net.errorCountPerSecond

Number of errors per second while receiving/transmitting over the network

nodeName

Host name that the pod is running on

runtimeAvailableBytes

Bytes available to the container runtime filesystem

runtimeCapacityBytes

Total capacity assigned to the container runtime filesystem in bytes

runtimeInodes

Total number of inodes in the container runtime filesystem

runtimeInodesFree

Free inodes in the container runtime filesystem

runtimeInodesUsed

Used inodes in the container runtime filesystem

runtimeUsedBytes

Used bytes in the container runtime filesystem

unschedulable

Status of node schedulability of new pods. Its value can be 0 (false) or 1 (true)

label.LABEL_NAME

Labels associated with your node, so you can filter and query for specific nodes

Namespace data

Query the K8sNamespaceSample event for namespace data:

Namespace attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration

createdAt

Timestamp of the namespace when it was created

namespace

Name of the namespace to be used as an identifier

label.LABEL_NAME

Labels associated with your namespace, so you can filter and query for specific namespaces

status

Current status of the namespace.

The value can be Active or Terminated

Deployment data

Query the K8sDeploymentSample event for deployment data:

Deployment attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration

createdAt

Timestamp of when the deployment was created

deploymentName

Name of the deployment to be used as an identifier

namespace

Name of the namespace that the deployment belongs to

label.LABEL_NAME

Labels associated with your deployment, so you can filter and query for specific deployments

podsAvailable

Number of replicas that are currently available

podsDesired

Number of replicas that you defined in the deployment

podsTotal

Total number of replicas that are currently running

podsUnavailable

Number of replicas that are currently unavailable

podsUpdated

Number of replicas that have been updated to achieve the desired state of the deployment

podsMissing

Total number of replicas that are missing (number of desired replicas, podsDesired, minus the total number of replicas, podsTotal)

ReplicaSet data

Query the K8sReplicasetSample event for ReplicaSet data:

Replica attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration

createdAt

Timestamp of when the ReplicaSet was created

deploymentName

Name of the deployment to be used as an identifier

namespace

Name of the namespace that the ReplicaSet belongs to

observedGeneration

Integer representing generation observed by the ReplicaSet

podsDesired

Number of replicas that you defined in the deployment

podsFullyLabeled

Number of pods that have labels that match the ReplicaSet pod template labels

podsReady

Number of replicas that are ready for this ReplicaSet

podsTotal

Total number of replicas that are currently running

podsMissing

Total number of replicas that are currently missing (number of desired replicas, podsDesired, minus the number of ready replicas, podsReady)

replicasetName

Name of the ReplicaSet to be used as an identifier

DaemonSet data

Query the K8sDaemonsetSample event for DaemonSet data:

DaemonSet attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration

createdAt

Timestamp of when the DaemonSet was created

namespaceName

Name of the namespace that the DaemonSet belongs to

label.LABEL_NAME

Labels associated with your DaemonSet, so you can filter and query for specific DaemonSet

daemonsetName

Name associated with the DaemonSet

podsDesired

The number of nodes that should be running the daemon pod

podsScheduled

The number of nodes running at least one daemon pod and are supposed to

podsAvailable

The number of nodes that should be running the daemon pod and have one or more of the daemon pod running and available

podsReady

The number of nodes that should be running the daemon pod and have one or more of the daemon pod running and ready

podsUnavailable

The number of nodes that should be running the daemon pod and have none of the daemon pod running and available

podsMisscheduled

The number of nodes running a daemon pod but are not supposed to

podsUpdatedScheduled

The total number of nodes that are running updated daemon pod

podsMissing

Total number of replicas that are currently missing (number of desired replicas, podsDesired, minus the number of ready replicas, podsReady)

metadataGeneration

Sequence number representing a specific generation of the desired state

StatefulSet data

Query the K8sStatefulsetSample event for StatefulSet data:

StatefulSet attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration

createdAt

Timestamp of when the StatefulSet was created

namespaceName

Name of the namespace that the StatefulSet belongs to

label.LABEL_NAME

Labels associated with your StatefulSet, so you can filter and query for specific StatefulSet

statefulsetName

Name associated with the StatefulSet

podsDesired

Number of desired pods for a StatefulSet

podsReady

The number of ready replicas per StatefulSet

podsCurrent

The number of current replicas per StatefulSet

podsTotal

The number of replicas per StatefulSet

podsUpdated

The number of updated replicas per StatefulSet

podsMissing

Total number of replicas that are currently missing (number of desired replicas, podsDesired, minus the number of ready replicas, podsReady)

observedGeneration

The generation observed by the StatefulSet controller

metadataGeneration

Sequence number representing a specific generation of the desired state for the StatefulSet

currentRevision

Indicates the version of the StatefulSet used to generate pods in the sequence. Value range: between 0 and podsCurrent

updateRevision

Indicates the version of the StatefulSet used to generate pods in the sequence. Value range: between podsDesired-podsUpdated and podsDesired

Pod data

Query the K8sPodSample event for pod data:

Pod attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration

createdAt

Timestamp of when the pod was created in epoch seconds

createdBy

Name of the Kubernetes object that created the pod.

For example, newrelic-infra

createdKind

Kind of Kubernetes object that created the pod.

For example, DaemonSet.

deploymentName

Name of the deployment to be used as an identifier

isReady

Boolean representing whether or not the pod is ready to serve requests

isScheduled

Boolean representing whether or not the pod has been scheduled to run on a node

label.LABEL_NAME

Labels associated with your pod, so you can filter and query for specific pods

message

Details related to the last pod status change

namespace

Name of the namespace that the pod belongs to

net.errorCountPerSecond

Number of errors per second while receiving/transmitting over the network

net.errorsPerSecond

Number of errors per second

net.rxBytesPerSecond

Number of bytes per second received over the network

net.txBytesPerSecond

Number of bytes per second transmitted over the network

nodeIP

Host IP address that the pod is running on

nodeName

Host name that the pod is running on

podIP

IP address of the pod. If it doesn't have an IP, it'll be empty

podName

Name of the pod to be used as an identifier

reason

Reason why the pod is in the current status

startTime

Timestamp of when the pod started running in epoch seconds

status

Current status of the pod.

Value can be Pending, Running, Succeeded, Failed, Unknown

Cluster data

Query the K8sClusterSample event to see cluster data:

Cluster attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration

clusterK8sVersion

Kubernetes version that the cluster is running

Container data

Query the K8sContainerSample event for container data:

Container attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration

containerID

Unique ID associated with the container. If you are running Docker, this is the Docker container id

containerImage

Name of the image that the container is running

containerImageID

Unique ID associated with the image that the container is running

containerName

Name associated with the container

cpuLimitCores

Integer representing limit CPU cores defined for the container in the pod specification

cpuRequestedCores

Requested CPU cores defined for the container in the pod specification

cpuUsedCores

CPU cores actually used by the container

cpuCoresUtilization

Percentage of CPU cores actually used by the container with respect to the CPU limit specified. This percentage is based on this calculation: (cpuUsedCores / cpuLimitCores) * 100

requestedCpuCoresUtilization

Percentage of CPU cores actually used by the container with respect to the CPU request specified

deploymentName

Name of the deployment to be used as an identifier

isReady

Boolean. Whether or not the container's readiness check succeeded

label.LABEL_NAME

Labels associated with your container, so you can filter and query for specific containers

memoryLimitBytes

Integer representing limit bytes of memory defined for the container in the pod specification

memoryRequestedBytes

Integer. Requested bytes of memory defined for the container in the pod specification

memoryUsedBytes

Integer. Bytes of memory actually used by the container

memoryUtilization

Percentage of memory actually used by the container with respect to the memory limit specified

requestedMemoryUtilization

Percentage of memory actually used by the container with respect to the memory request specified

memoryWorkingSetBytes

Integer. Bytes of memory in the working set

memoryWorkingSetUtilization

Percentage of working set memory actually used by the container with respect to the memory limit specified

requestedMemoryWorkingSetUtilization

Percentage of working set memory actually used by the container with respect to the memory request specified

namespace

Name of the namespace that the container belongs to

nodeIP

Host IP address the container is running on

nodeName

Host name that the container is running on

podName

Name of the pod that the container is in, to be used as an identifier

reason

Provides a reason why the container is in the current status

restartCount

Number of times the container has been restarted

status

Current status of the container.

Value can be Running, Terminated, or Unknown

containerCpuCfsPeriodsDelta

Delta change of elapsed enforcement period intervals

containerCpuCfsThrottledPeriodsDelta

Delta change of throttled period intervals

containerCpuCfsThrottledSecondsDelta

Delta change of duration the container has been throttled, in seconds

containerCpuCfsPeriodsTotal

Total number of elapsed enforcement period intervals

containerCpuCfsThrottledPeriodsTotal

Total number of throttled period intervals

containerCpuCfsThrottledSecondsTotal

Total time duration the container has been throttled, in seconds

containerMemoryMappedFileBytes

Total size of memory mapped files used by this container, in bytes

Volume data

Query the K8sVolumeSample event for volume data:

Volume attribute

Description

volumeName

Name that you assigned to the volume at creation

clusterName

Cluster where the volume is configured

namespace

Namespace where the volume is configured

podName

The pod that the volume is attached to. The Kubernetes monitoring integration lists Volumes that are attached to a pod

persistent

If this is a persistent volume, this value is set to true

pvcNamespace

Namespace where the Persistent Volume Claim is configured

pvcName

Name that you assigned to the Persistent Volume Claim at creation

fsCapacityBytes

Capacity of the volume, in bytes

fsUsedBytes

Usage of the volume, in bytes

fsAvailableBytes

Capacity available of the volume, in bytes

fsUsedPercent

Usage of the volume in percentage

fsInodes

Total inodes of the volume

fsInodesUsed

inodes used in the volume

fsInodesFree

inodes available in the volume

Volume data is available for volume plugins that implement the MetricsProvider interface:

  • AWSElasticBlockStore
  • AzureDisk
  • AzureFile
  • Cinder
  • Flexvolume
  • Flocker
  • GCEPersistentDisk
  • GlusterFS
  • iSCSI
  • StorageOS
  • VsphereVolume

API server data

Query the K8sApiServerSample event to see API Server data. For more information, see Configure control plane monitoring:

API server attribute

Description

processResidentMemoryBytes

Resident memory size, in bytes

processCpuSecondsDelta

Difference of the user and system CPU time spent, in seconds

goThreads

Number of OS threads created

goGoroutines

Number of goroutines that currently exist

apiserverRequestDelta_verb_VERB_code_CODE

Difference of the number of apiserver requests, broken out for each verb and HTTP response code

apiserverRequestRate_verb_VERB_code_CODE

Rate of apiserver requests, broken out for each verb and HTTP response code

restClientRequestsDelta_code_CODE_method_METHOD

Difference of the number of HTTP requests, partitioned by method and code

restClientRequestsRate_code_CODE_method_METHOD

Rate of the number of HTTP requests, partitioned by method and code

etcdObjectCounts_resource_RESOURCE-KIND

Number of stored objects at the time of last check, split by kind

Controller manager data

Query the K8sControllerManagerSample event to see Controller manager data. For more information, see Configure control plane monitoring:

Controller manager attribute

Description

processResidentMemoryBytes

Resident memory size, in bytes

processCpuSecondsDelta

Difference of the user and system CPU time spent in seconds

goThreads

Number of OS threads created

goGoroutines

Number of goroutines that currently exist

workqueueAddsDelta_name_WORK-QUEUE-NAME

Difference of the total number of adds handled by workqueue

workqueueDepth_name_WORK-QUEUE-NAME

Current depth of workqueue

workqueueRetriesDelta_name_WORK-QUEUE-NAME

Difference of the total number of retries handled by workqueue

leaderElectionMasterStatus

Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master

Scheduler data

Query the K8sSchedulerSample event in New Relic Insights to see Scheduler data. For more information, see Configure control plane monitoring:

Scheduler attribute

Description

processResidentMemoryBytes

Resident memory size, in bytes

processCpuSecondsDelta

Difference of the user and system CPU time spent in seconds

goThreads

Number of OS threads created

goGoroutines

Number of goroutines that currently exist

leaderElectionMasterStatus

Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master

httpRequestDurationMicroseconds_handler_HANDLER_quantile_QUANTILE

The HTTP request latencies in microseconds, per quantile

httpRequestDurationMicroseconds_handler_HANDLER_sum

The sum of the HTTP request latencies, in microseconds

httpRequestDurationMicroseconds_handler_HANDLER_count

The number of observed HTTP requests events

restClientRequestsDelta_code_CODE_host_HOST_method_METHOD

Difference of the number of HTTP requests, partitioned by status code, method, and host

restClientRequestsRate_code_CODE_host_HOST_method_METHOD

Rate of the number of HTTP requests, partitioned by status code, method, and host

schedulerScheduleAttemptsDelta_result_RESULT

Difference of the number of attempts to schedule pods, by the result. unschedulable means a pod could not be scheduled, while error means an internal scheduler problem

schedulerScheduleAttemptsRate_result_RESULT

Rate of the number of attempts to schedule pods, by the result. unschedulable means a pod could not be scheduled, while error means an internal scheduler problem

schedulerSchedulingDurationSeconds_operation_OPERATION_quantile_QUANTILE

Scheduling latency in seconds split by sub-parts of the scheduling operation

schedulerSchedulingDurationSeconds_operation_OPERATION_sum

The sum of scheduling latency in seconds split by sub-parts of the scheduling operation

schedulerSchedulingDurationSeconds_operation_OPERATION_count

The number of observed events of schedulings split by sub-parts of the scheduling operation.

schedulerPreemptionAttemptsDelta

Difference of the total preemption attempts in the cluster till now

schedulerPodPreemptionVictims

Number of selected preemption victims

ETCD data

Query the K8sEtcdSample event to see ETCD data. For more information, see Configure control plane monitoring:

ETCD attribute

Description

processResidentMemoryBytes

Resident memory size, in bytes

processCpuSecondsDelta

Difference of the user and system CPU time spent in seconds

goThreads

Number of OS threads created

goGoroutines

Number of goroutines that currently exist

etcdServerHasLeader

Whether or not a leader exists. 1 is existence, 0 is not

etcdServerLeaderChangesSeenDelta

Difference of the number of leader changes seen

etcdMvccDbTotalSizeInBytes

Total size of the underlying database physically allocated, in bytes

etcdServerProposalsCommittedDelta

Difference of the total number of consensus proposals committed

etcdServerProposalsCommittedRate

Rate of the total number of consensus proposals committed

etcdServerProposalsAppliedDelta

Difference of the total number of consensus proposals applied

etcdServerProposalsAppliedRate

Rate of the total number of consensus proposals applied

etcdServerProposalsPending

The current number of pending proposals to commit

etcdServerProposalsFailedDelta

Difference of the total number of failed proposals seen

etcdServerProposalsFailedRate

Rate of the total number of failed proposals seen

processOpenFds

Number of open file descriptors

processMaxFds

Maximum number of open file descriptors

processFdsUtilization

Percentage open file descriptors with respect to the maximum number that can be opened

etcdNetworkClientGrpcReceivedBytesRate

Rate of the total number of bytes received from gRPC clients

etcdNetworkClientGrpcSentBytesRate

Rate of the total number of bytes sent to gRPC clients

Endpoint data

Query the K8sEndpointSample event in New Relic Insights for endpoint data:

Endpoint attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration

createdAt

Timestamp of when the endpoint was created

namespaceName

Name of the namespace that the endpoint belongs to

endpointName

Name associated with the endpoint

label.LABEL_NAME

Labels associated with your endpoint, so you can filter and query for specific endpoints

addressAvailable

Number of addresses available in endpoint

addressNotReady

Number of addresses not ready in endpoint

Service data

Query the K8sServiceSample event for service data:

Service attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration

createdAt

Timestamp of when the service was created

namespaceName

Name of the namespace that the service belongs to

label.LABEL_NAME

Labels associated with your service, so you can filter and query for specific service

serviceName

Name associated with the service

loadBalancerIP

The IP of the external load balancer, if Spectype is LoadBalancer.

externalName

The external name value, if Spectype is ExternalName

clusterIP

The internal cluster IP, if Spectype is ClusterIP

specType

Type of the service

selector.LABEL_NAME

The label selector that this service targets

Horizontal Pod Autoscaler data

Query the K8sHpaSample event in New Relic Insights for Horizontal Pod Autoscaler data:

HPA attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration

label.LABEL_NAME

Labels associated with your HPA, so you can filter and query for specific autoscaler

currentReplicas

Current number of replicas of pods managed by this autoscaler

desiredReplicas

Desired number of replicas of pods managed by this autoscaler

minReplicas

Lower limit for the number of pods that can be set by the autoscaler, 1 by default

maxReplicas

Upper limit for the number of pods that can be set by the autoscaler; cannot be smaller than minReplicas

targetMetric

The metric specifications used by this autoscaler when calculating the desired replica count

isAble

Boolean representing whether or not the autoscaler is able to fetch and update scales, as well as whether or not any backoff-related conditions would prevent scaling

isActive

Boolean representing whether or not the autoscaler is enabled (if it's able to calculate the desired scales)

isLimited

Boolean representing whether or not the autoscaler is capped, either up or down, by the maximum or minimum replicas configured

labels

Number of Kubernetes labels converted to Prometheus labels

metadataGeneration

The generation observed by the HorizontalPodAutoscaler controller

Kubernetes metadata in APM-monitored applications

By linking your applications with Kubernetes, the following attributes are added to application trace and distributed trace:

  • nodeName
  • containerName
  • podName
  • clusterName
  • deploymentName
  • namespaceName

For more help

Create issueEdit page
Copyright © 2021 New Relic Inc.