Create and modify Kubernetes alerts

With New Relic, you can stay informed about Kubernetes issues by defining the specific data you want to monitor, setting customizable thresholds, and configuring the notifications you want to receive. You can add a set of recommended alert policies that you can tailor to your needs, or you can create completely custom alerts, policies, and workflows to suit your environment.

Create a Kubernetes alert condition

1. Go to one.newrelic.com > All capabilities > Infrastructure and click Alerts on the left navigation pane.

2. Click + New Alert Condition.

   

3. Choose Kubernetes as an alert type and define your thresholds.

   

4. Define your thresholds.

5. Configure your alert policy. You can create a new policy or add this condition to an existing policy.

6. Set the condition status of the condition.

7. (Optional) Add a runbook URL.

8. Click Create.

Create a Kubernetes alert policy

1. Go to one.newrelic.com > All capabilities > Infrastructure and click Alerts on the left navigation pane.

2. Click + New alert policy.

   

3. Enter a meaningful name for the policy (maximum 64 characters).

4. Choose the way you want to group the incidents from this policy.

5. (Optional) Check the box Suppress noise with machine learning correlation to enable correlation for the alert policy.

   

6. Click Set up notifications.

Kubernetes alert notifications

Once you've configured your alert conditions, policies, and workflows, you'll begin to receive notifications when the threshold configured in an alert condition is triggered. You have these options:

• View the incident

• Acknowledge the incident

• Go to a chart of the incident data by selecting the name of the identifier. The entity identifier that triggered the alert appears near the top of the notification message. The format of the identifier depends on the alert type:

  • Available pods are less than desired pods alerts:

    K8s:CLUSTER_NAME:PARENT_NAMESPACE:replicaset:REPLICASET_NAME

    Copy

  • CPU or memory usage alerts:

    K8s:CLUSTER_NAME:PARENT_NAMESPACE:POD_NAME:container:CONTAINER_NAME

    Copy

  Here are some examples.

  Pod alert notification example

  For Available pods are less than desired pods alerts, the ID of the ReplicaSet triggering the issue might look like this:

  k8s:beam-production:default:replicaset:nginx-deployment-1623441481

  Copy

  This identifier contains the following information:

  • Cluster name: beam-production
  • Parent namespace: default
  • ReplicaSet name: nginx-deployment-1623441481

  Container resource notification example

  For container CPU or memory usage alerts, the entity might look like this:

  k8s:beam-production:kube-system:kube-state-metrics-797bb87c75-zncwn:container:kube-state-metrics

  Copy

  This identifier contains this information:

  • Cluster name: beam-production
  • Parent namespace: kube-system
  • Pod namespace: kube-state-metrics-797bb87c75-zncwn
  • Container name: kube-state-metrics