Muting rules: Suppress notifications

Muting rules override the default New Relic Alerts lifecycle of defined alert policies and conditions in order to mute, or suppress, notifications during known system disruptions, such as maintenance windows and deployments.

How it works

A muting rule contains a set of conditions that match against attributes in the violation object. The muting rules tell the lifecycle override system in New Relic Alerts how to:

  1. Identify individual violations after they are created, but before an incident is opened.
  2. Override their default condition to indicate that they should be "muted."

Currently, muting a violation means that the normal Alerts incident lifecycle will be maintained, except that an incident that only contains muted violations will not send out any notifications.

Muting rules override specific violations. They do not disable the existing policies or conditions. This allows you to mute violations from specific entities that may be covered by a policy or condition that covers a large number of entities. This also keeps you from having to over-mute your monitoring when you are performing maintenance on a subset of your system.

Permissions and availability

The muting rules functionality is available to all New Relic accounts that have access to the more advanced Alerts product features, such as baseline conditions. This covers most paid New Relic accounts. Individual users and roles within the account can be also configured for limited access.

To set permissions for individual users and roles to view, modify, or delete muting rules:

  1. Go to rpm.newrelic.com > (account dropdown) > Account settings > Users and roles > Roles.
  2. Use the Lifecycle overrides capability in the Alerts permissions setting.
Alert incident lifecycle: Muting rule permissions
rpm.newrelic.com > (account dropdown) > Account settings > Users and roles > Roles: In the Alerts > Lifecycle overrides section, set permissions to add, modify, or delete muting rules.

Manage muting rules

For the initial release, muting rules are created, managed, enabled, and disabled using NerdGraph (our GraphQL API). A full user interface for managing muting rules will be released later.

The following queries and mutations are available in NerdGraph. View the NerdGraph GraphiQL tool to see the schema more in depth.

  • actor.account.alerts.mutingRule: Fetch a muting rule by id.
  • actor.account.alerts.mutingRules: Fetch a list of muting rules for an account.
  • alertsMutingRuleCreate: Create a muting rule for an account.
  • alertsMutingRuleUpdate: Update a muting rule by id and account id.
  • alertsMutingRuleDelete: Delete a muting rule by id and account id.

Also see the examples in this document for some sample queries and mutations.

A muting rule consists of the following fields and components:

Muting rule Fields and components
id The unique identifier for the muting rule.
name (Required)

Text field for the user-friendly name of the muting rule. This is used when listing or referencing a rule. We do not require that the name be unique, but it is recommended.

description

Optional text field to describe the muting rule. Currently this data is only used for management display purposes.

accountId

The account id that the muting rule belongs to. A muting rule will only affect violations that occur in the same account. To mute violations across multiple accounts, you must create multiple muting rules.

createdAt

The timestamp when the muting rule was created in UTC.

createdBy

The id of the user who created the muting rule.

updatedAt

The timestamp when the muting rule was last modified in UTC.

updatedBy

The id of the user who last modified the muting rule.

enabled

A boolean to enable or disable the muting rule. Currently muting rules must be manually enabled and disabled. An upcoming release will provide duration timing and scheduling.

condition

The set of individual expressions that define which violations to target. A muting rule condition has the following shape:

  • operator: The boolean operator AND or OR that defines how to combine the set of expressions, see “Conditions” below.
  • conditions: The set of individual expressions (sub-conditions) that target attributes on a violation. These are evaluated together based on the operator. There can be a maximum of 20 sub-conditions for a single muting rule.

A sub-condition has the following shape:

  • attribute: An attribute on a violation. See the “Violation Event Attributes” section below for a list of targetable attributes.
  • operator: The comparison function used to compare the selected violation attribute against the value(s) in the condition. See the “Sub-Condition Operators” section below for a list of available operators.
  • values: An array of values as strings to compare against the selected violation attribute. Values will be coerced from strings if necessary when evaluating the condition. There is a maximum of 20 values when using an operator that supports comparison against multiple values, such as IN.

Muting behavior

The following table describes how the Alerts incident lifecycle is affected by muted violations:

IF AND THEN
Event: Incident opens
An incident opens because of a violation that is not muted An “open incident” notification will be sent (default).
An incident opens due to a violation that is muted An “Open Incident” notification will not be sent (muted).
Event: Violation added to an open Incident
A new muted violation is added to an already open incident No notification action is triggered (default).
A new unmuted violation is added to an open incident An “Open Incident” notification has not been sent An “Open Incident” notification will be sent.
A new unmuted violation is added to an open incident An “Open Incident” notification has already been sent No notification action is triggered (default).
Event: Incident Closed
An Incident is closed An “Open Incident” notification has not been sent A “Close Incident” notification will not be sent.
An Incident is closed An “Open Incident” notification has been sent A “Close Incident” notification will be sent.
Event: Incident Acknowledged
An Incident is acknowledged An “Open Incident” notification has not been sent An “Incident Acknowledged” notification will not be sent.
An Incident is acknowledged An “Open Incident” notification has been sent An “Incident Acknowledged” notification will be sent.

View muted violations and incidents

When viewing an open or closed incident in the New Relic Alerts UI, violations and incidents are marked as Muted:

Alerts > Incidents > Open Incidents > Incident ####

Alert incident lifecycle: Muting rule violations
Here is an example of an incident with a critical violation that has been muted.

Violations and incidents are marked as Muted in the State column when viewing lists of incidents, violations, or events:

Alerts > Incidents > Open Incidents

Alert incident lifecycle: Muting rule incidents
Here is an example of an open incident that triggered the muting rule.

Alerts > Events > Violations

Alert incident lifecycle: Muting rule critical violations
Here is an example of a list of alert violations. The State column shows which violations have been muted.

Alerts > Events > All events

Alert incident lifecycle: Muting rule events
Here is an example of a list of alert events. The State column shows which events have been muted.

Violation event attributes

Violation event attributes include:

  • accountId: The account id of the violation.
  • conditionId: The id of the alert condition that triggered the violation.
  • conditionName: The name of the alert condition that triggered the violation.
  • conditionType: The type of alert condition that triggered the violation. Values include:

    • static
    • baseline
    • outlier
    • metric
    • external service
    • jvm metric
    • key transaction metric
    • response time percentile
    • host not reporting
    • process running
    • location failure
    • monitor failure
  • policyId: The id of the alert policy that triggered the violation.
  • policyName: The name of the alert policy that triggered the violation.
  • conditionRunbookUrl: The runbook URL on the alert condition that triggered the violation.
  • product: The product of the violation’s target. Values include:

    • APM
    • BROWSER
    • INFRASTRUCTURE
    • MOBILE
    • NRQL
    • PLUGINS
    • SYNTHETICS
  • targetName: The name of the violation’s target.
  • nrqlEventType: The event type targeted by an alert NRQL condition.
  • nrqlQuery: The NRQL query string targeted by an alert NRQL condition.
  • tag.*: Arbitrary key-value metadata, or tags, associated with the violation.

    Example: If your NRQL condition has a query with FACET host, you can target that FACET attribute with tag.host. Facet tags will only appear on a violation if they are included in the alerts condition’s query.

    NRQL condition queries can accept multiple facet attributes. If you want to be able to filter from attributes in your events or metric time series that have been aggregated, you must add those attributes to your NRQL query FACET clause; for example: FACET host, region, cluster.

Sub-condition operators

  • EQUALS: Where the supplied value equals the violation attribute value.
  • NOT_EQUALS: Where the supplied value does not equal the violation attribute value.
  • IN: Where the violation attribute value is present in a list of supplied values (up to 20).
  • NOT_IN: Where the violation attribute value is not present in a list of supplied values (up to 20).
  • CONTAINS: Where the supplied value string is present in the violation attribute value.
  • NOT_CONTAINS: Where the supplied value string is not present in the violation attribute value.
  • ANY: Caution: A condition with this operator will mute all violations on the account.
  • IS_BLANK: Where the violation attribute value is blank. Null, empty string, etc.
  • IS_NOT_BLANK: Where the violation attribute value is not blank. Null, empty string, etc.
  • STARTS_WITH: Where the violation attribute value begins with the supplied value string.
  • NOT_STARTS_WITH: Where the violation attribute value does not begin with the supplied value string.
  • ENDS_WITH: Where the violation attribute value ends with the supplied value string.
  • NOT_ENDS_WITH: Where the violation attribute value does not end with the supplied value string.

Examples

For more information about making requests to NerdGraph, see the NerdGraph documentation, including GraphQL tutorials.

Fetch muting rules for account

To fetch the list of muting rules (with name and condition) for an account:

{
  actor {
    account(id: YOUR_ACCOUNT_ID) {
      alerts {
        mutingRules {
          name
          condition {
            operator
            conditions {
              attribute
              operator
              values
            }
          }
        }
      }
    }
  }
}
Create muting rule

To create a muting rule and return the muting rule's id for muting violations from NRQL conditions where host facet is host-1:

mutation {
  alertsMutingRuleCreate(accountId: YOUR_ACCOUNT_ID, rule: {
    name: "host-1 mute rule",
    description: "Mute host-1 violations",
    enabled: true,
    condition: {
      operator: AND,
      conditions: [{
        attribute: "tag.host",
        operator: EQUALS,
        values: ["host-1"]
      }]
    }
  }) {
    id
  }
}
Create muting rule for specific hosts or products

To create a muting rule and return the muting rule's id for muting violations from Infrastructure conditions where the target host is host-1 or the violation was from the Synthetics or Mobile products:

mutation {
  alertsMutingRuleCreate(accountId: YOUR_ACCOUNT_ID, rule: {
    name: "host-1 and Synthetics/Mobile mute rule",
    description: "Mute host-1 and Synthetics/Mobile violations",
    enabled: true,
    condition: {
      operator: OR,
      conditions: [
        {
          attribute: "targetName",
          operator: EQUALS,
          values: ["host-1"]
        },
        {
          attribute: "product",
          operator: IN,
          values: ["SYNTHETICS", "MOBILE"]
        }
      ]
    }
  }) {
    id
  }
}
Rename and disable muting rule

To change the name and disable a muting rule, and return its new name:

mutation {
  alertsMutingRuleUpdate(accountId: YOUR_ACCOUNT_ID, id: MUTING_RULE_ID, rule: {
    name: "updated name",
    enabled: false
  }) {
    name
  }
}
Delete muting rule

To delete a muting rule:

mutation {
 alertsMutingRuleDelete(accountId: YOUR_ACCOUNT_ID, id: MUTING_RULE_ID) {
    id
  }
}

For more help

Recommendations for learning more: