Muting rules: Suppress notifications

Alerts sends out timely notifications when your system is having problems. Sometimes there are notifications you know you don't need to see: You can use muting rules to stop being bombarded by messages you don't want.

Once you've spotted the common elements in your unwanted notifications, you can define muting rules that specifically target those elements, while letting other notifications through. Even when a notification is muted, Alerts still gathers data on those violations. Muting rules don't interfere with the alerts process and are applied at the point right before a notification is sent.

You can read more about why we added muting rules in this blog post.

Manage muting rules

A muting rule condition is the set of individual expressions made up of attributes, operators, and values that define which violations to target for muting.

You can create, enable, disable, and manage muting rules. Go to one.newrelic.com, in the top nav click Alerts & AI, then click Muting rules. Muting rules can be enabled or disabled at any time.

Rules can have one of the following statuses:

  • Active: Muting is enabled and active.
  • Scheduled: Muting is enabled but not active yet (there's a future schedule).
  • Ended: Muting is enabled, but no longer active (there's no future schedule).
  • Inactive: Muting is disabled.
Manage muting rules
one.newrelic.com > Alerts & AI > Muting rules: You can create complex muting rules to target a small or large set of unwanted notifications.

Create a muting rule

Before creating muting rules, you'll need to create policies and conditions that generate violation notifications.

To create a muting rule, click + Add a rule in the Muting rules screen. Enter a name and a description for the muting rule, and select the account to which the rule will apply.

Next, build the violation filter. You can use violation event attributes and sub-condition operators. Values can be compared against one of your violation attributes, such as an Alerts policy ID or a condition name.

Muting rule edit screen
one.newrelic.com > Alerts & AI > Muting rules: You can create complex muting rules to target a small or large set of unwanted notifications.

Schedule a muting rule

If needed, you can schedule your muting rules.

To do this, select a start time and/or end time. Optionally, you can set the muting rule to last the whole day.

You can also choose to select a time zone for the muting rule schedule. The default is the time zone selected in your user preferences.

Schedule_mute_rule.png

Manage muting rules with NerdGraph

In NerdGraph, you can use the following queries and mutations with your muting rules. You can see the schema in more detail in the API Explorer.

  • actor.account.alerts.mutingRule: Fetch a muting rule by id.
  • actor.account.alerts.mutingRules: Fetch a list of muting rules for an account.
  • alertsMutingRuleCreate: Create a muting rule for an account.
  • alertsMutingRuleUpdate: Update a muting rule by id and account id.
  • alertsMutingRuleDelete: Delete a muting rule by id and account id.

We've given you some sample queries and mutations examples in this document.

A muting rule has the following fields and components:

Muting rule Fields and components
id The unique identifier for the muting rule.
name (Required)

A text field for the user-friendly name of the muting rule. This is used when listing or referencing a rule. We don't require that the name be unique, but it is recommended.

description

This is an optional text field describing the muting rule. It can be a useful way to provide more context for your muting rule. This data is only used for management display purposes.

accountId

The muting rule's account ID. A muting rule will only affect violations that occur in a single account. To mute violations across multiple accounts, you'll need to create a muting rule for each account separately.

createdAt

The timestamp when the muting rule was created (UTC).

createdBy

The user ID of the person who created the muting rule.

updatedAt

The timestamp when the muting rule was last modified (UTC).

updatedBy

The user ID of the person who last modified the muting rule.

enabled

Enable or disable the muting rule (Boolean). Muting rules must be manually enabled and disabled.

condition

The set of individual expressions that define which violations to target. A muting rule condition has:

  • operator: The boolean operator AND or OR that defines how to combine the set of conditions.
  • conditions: The set of individual expressions (sub-conditions) that target attributes within a violation. These are evaluated together based on the operator. You can have a maximum of 20 sub-conditions for a single muting rule.

A sub-condition has:

  • attribute: A single attribute within a violation. Go here for a list of Violation event attributes.
  • operator: The comparison function used to compare the selected violation attribute against values in the condition. Go here for a list of sub-condition operators.
  • values: An array of string values to compare against selected violation attributes. When the muting rules evaluate a condition, if necessary, values will be coerced from strings. You can use a maximum of 500 values when using an operator that supports comparison against multiple values, such as IN.
schedule

The time window when the MutingRule should actively mute violations.

  • startTime: The datetime stamp that represents when the MutingRule should start. This is in local ISO 8601 format without an offset. Example: '2020-07-08T14:30:00'
  • endTime: The datetime stamp that represents when the MutingRule should end. This is in local ISO 8601 format without an offset. Example: '2020-07-15T14:30:00'
  • timeZone: The time zone that applies to the muting rule schedule. Example: 'America/Los_Angeles'. See https://en.wikipedia.org/wiki/List_of_tz_database_time_zones.

How muting rules work

Muting rules are applied at the end of the default alert lifecycle in order to suppress, or mute, notifications. They don't disable existing policies or conditions. For example, you can mute notifications during known system disruptions, such as maintenance windows and deployments. System disruption violations will still be identified, even though the notifications for those violations are muted.

A muting rule uses a set of conditions that match against attributes in a violation event. The muting rules tell us how to:

  1. Identify individual violations after they're created, but before an incident is opened.
  2. Override their default condition to indicate that they should be "muted."

Currently, muting a violation means that the normal Alerts incident lifecycle will be maintained, except that an incident that only contains muted violations will not send out any notifications.

Muting rules override specific violations. They don't disable existing policies or conditions. This allows you to mute violations from specific entities that may be covered by a policy or condition that covers a large number of entities. This also keeps you from having to over-mute your monitoring when you are performing maintenance on a subset of your system.

Muting behavior

The following table describes how the Alerts incident lifecycle is affected by muted violations:

IF AND THEN
Event: Incident opens
An incident opens because of a violation that is not muted An “Open incident” notification will be sent (default).
An incident opens due to a violation that is muted An “Open incident” notification will not be sent (muted).
Event: Violation added to an open incident
A new muted violation is added to an already open incident No notification action is triggered (default).
A new unmuted violation is added to an open incident An “Open incident” notification has not been sent An “Open incident” notification will be sent.
A new unmuted violation is added to an open incident An “Open incident” notification has already been sent No notification action is triggered (default).
Event: Incident closed
An Incident is closed An “Open incident” notification has not been sent A “Close incident” notification will not be sent.
An Incident is closed An “Open incident” notification has been sent A “Close incident” notification will be sent.
Event: Incident acknowledged
An Incident is acknowledged An “Open incident” notification has not been sent An “Incident acknowledged” notification will not be sent.
An Incident is acknowledged An “Open incident” notification has been sent An “Incident acknowledged” notification will be sent.

View muted violations and incidents

When viewing an open or closed incident, violations and incidents are marked as Muted:

Go to one.newrelic.com, in the top nav click Alerts & AI, then click Incidents.

Alert incident lifecycle: Muting rule violations
An incident with a critical violation that's been muted.

Violations and incidents are marked as Muted in the State column when viewing lists of incidents, violations, or events:

Go to one.newrelic.com, in the top nav click Alerts & AI, click Incidents, then click Open incidents.

Alert incident lifecycle: Muting rule incidents
An example of an open incident that triggered the muting rule.

Go to one.newrelic.com, in the top nav click Alerts & AI, then click Events.

Alert incident lifecycle: Muting rule critical violations
An example of a list of violations. The State column shows which violations have been muted.

Go to one.newrelic.com, in the top nav click Alerts & AI, click Events, then click All events.

Alert incident lifecycle: Muting rule events
An example of a list of alert events. The State column shows which events have been muted.

Use attributes for muting

For an explanation of what attributes (metadata) is available for choosing what violations are muted, see Violation event attributes.

Mute faceted results using tag.

To mute results of faceted queries, use the tag.FACETED_ATTRIBUTE attribute, where FACETED_ATTRIBUTE represents the attributed you've run a NRQL FACET query on. For example: if your NRQL alert condition includes FACET host in its query, you can target that FACET attribute using tag.host.

NRQL condition queries can accept multiple facet attributes. If you want to be able to filter from attributes in your events or metric time series that have been aggregated, you must add those attributes to your NRQL query FACET clause; for example: FACET host, region, cluster.

For an example of using tag., see Create muting rule.

Sub-condition operators

The following are the logical operators available to use to compare attributes when you're constructing muting rules. For examples of these being used, see Examples.

  • EQUALS: Where the supplied value equals the violation attribute value.
  • NOT_EQUALS: Where the supplied value does not equal the violation attribute value.
  • IN: Where the violation attribute value is present in a list of supplied values (up to 500).
  • NOT_IN: Where the violation attribute value is not present in a list of supplied values (up to 500).
  • CONTAINS: Where the supplied value string is present in the violation attribute value.
  • NOT_CONTAINS: Where the supplied value string is not present in the violation attribute value.
  • ANY: Caution: A condition with this operator will mute all violations on the account.
  • IS_BLANK: Where the violation attribute value is blank. Null, empty string, etc.
  • IS_NOT_BLANK: Where the violation attribute value is not blank. Null, empty string, etc.
  • STARTS_WITH: Where the violation attribute value begins with the supplied value string.
  • NOT_STARTS_WITH: Where the violation attribute value does not begin with the supplied value string.
  • ENDS_WITH: Where the violation attribute value ends with the supplied value string.
  • NOT_ENDS_WITH: Where the violation attribute value does not end with the supplied value string.

Muting examples

For more information about making requests to NerdGraph, see the NerdGraph documentation, including GraphQL tutorials.

Fetch muting rules for account

To fetch the list of muting rules (with name and condition) for an account:

{
  actor {
    account(id: YOUR_ACCOUNT_ID) {
      alerts {
        mutingRules {
          name
          condition {
            operator
            conditions {
              attribute
              operator
              values
            }
          }
        }
      }
    }
  }
}
Create muting rule

To create a muting rule and return the muting rule's id for muting violations from NRQL conditions where host facet is host-1:

mutation {
  alertsMutingRuleCreate(accountId: YOUR_ACCOUNT_ID, rule: {
    name: "host-1 mute rule",
    description: "Mute host-1 violations",
    enabled: true,
    condition: {
      operator: AND,
      conditions: [{
        attribute: "tag.host",
        operator: EQUALS,
        values: ["host-1"]
      }]
    }
  }) {
    id
  }
}
Create muting rule for specific hosts or products

To create a muting rule and return the muting rule's id for muting violations from Infrastructure conditions where the target host is host-1 or the violation was from Synthetic monitoring or Mobile products:

mutation {
  alertsMutingRuleCreate(accountId: YOUR_ACCOUNT_ID, rule: {
    name: "host-1 and Synthetics/Mobile mute rule",
    description: "Mute host-1 and Synthetics/Mobile violations",
    enabled: true,
    condition: {
      operator: OR,
      conditions: [
        {
          attribute: "targetName",
          operator: EQUALS,
          values: ["host-1"]
        },
        {
          attribute: "product",
          operator: IN,
          values: ["SYNTHETICS", "MOBILE"]
        }
      ]
    }
  }) {
    id
  }
}
Rename and disable muting rule

To change the name and disable a muting rule and return its new name:

mutation {
  alertsMutingRuleUpdate(accountId: YOUR_ACCOUNT_ID, id: MUTING_RULE_ID, rule: {
    name: "updated name",
    enabled: false
  }) {
    name
  }
}
Delete muting rule

To delete a muting rule:

mutation {
 alertsMutingRuleDelete(accountId: YOUR_ACCOUNT_ID, id: MUTING_RULE_ID) {
    id
  }
}

For more help

If you need more help, check out these support and learning resources: