You can customize New Relic alerting loss of signal detection and gap filling using our NerdGraph API. For example, you can configure how long to wait before considering the signal lost, or what value should be used for filling gaps in the time series.
Loss of signal occurs when New Relic stops receiving data for a while; technically, we detect loss of signal after a significant amount of time has elapsed since data was last received in a time series. Loss of signal can be used to trigger or resolve an incident, which you can use to set up alerts.
Gap filling can help you solve issues caused by lost data points. When gaps are detected between valid data points, we automatically fill those gaps with replacement values, such as the last known values or a static value. Gap filling can prevent alerts from triggering or resolving when they shouldn't.
Tip
The system fills gaps in actively reported signals. This signal history is dropped after 2 hours of inactivity. For gap filling, data points received after this period of inactivity are treated as new signals.
To learn more about signal loss, gap filling, and how to request access to these features, see this Support Forum post.
In this guide we cover the following:
Customize your loss of signal detection
Loss of signal detection opens or closes incidents if no data is received after a certain amount of time. For example, if you set the duration of the expiration period to 60 seconds and an integration doesn't seem to send data for more than a minute, a loss of signal threshold would be triggered.
You can configure the duration of the signal loss and whether to open an incident or close it by using these three fields in NerdGraph:
expiration.expirationDuration
: How long to wait, in seconds, after the last data point is received by our platform before considering the signal as lost. This is based on the time when data arrives at our platform and not on data timestamps. The default is to leave this null, and therefore this wouldn't enable Loss of Signal Detection.expiration.openViolationOnExpiration
: Iftrue
, a new incident is opened when a signal is lost. Default isfalse
. To use this field, a duration must be specified.expiration.closeViolationsOnExpiration
: Iftrue
, open incidents related to the signal are closed on expiration. Default isfalse
. To use this field, a duration must be specified.
You also have the option to skip opening an incident when the signal is expected to be lost. To do this, use this field in NerdGraph:
expiration.ignoreOnExpectedTermination
: Iftrue
, an incident will not be opened when the signal is expected to be lost. To indicate a signal is expected to be lost, the tagtermination: expected
must be present on the entity. Default isfalse
.
View loss of signal settings for an existing condition
Existing NRQL conditions may have their loss of signal settings already configured. To view the existing condition settings, select the fields under nrqlCondition
> expiration
:
{ actor { account(id: YOUR_ACCOUNT_ID) { alerts { nrqlCondition(id: NRQL_CONDITION_ID) { ... on AlertsNrqlStaticCondition { id name nrql { query } expiration { closeViolationsOnExpiration expirationDuration openViolationOnExpiration ignoreOnExpectedTermination } } } } } }}
You should see a result like this:
{ "data": { "actor": { "account": { "alerts": { "nrqlCondition": { "expiration": { "closeViolationsOnExpiration": false, "expirationDuration": 300, "openViolationOnExpiration": true, "ignoreOnExpectedTermination": false }, "id": "YOUR_ACCOUNT_ID", "name": "Any less than - Extrapolation", "nrql": { "query": "SELECT average(value) FROM AlertsSmokeTestSignals WHERE wave_type IN ('min-max', 'single-gap') FACET wave_type" } } } } } }, ...
Create a new condition with loss of signal settings
Let's say that you want to create a new create a NRQL static condition that triggers a loss of signal threshold after no data is received for two minutes. You would set expirationDuration
to 120 seconds and set openViolationOnExpiration
to true
, like in the example below.
mutation { alertsNrqlConditionStaticCreate( accountId: YOUR_ACCOUNT_ID policyId: YOUR_POLICY_ID condition: { name: "Low Host Count - Catastrophic" enabled: true nrql: { query: "SELECT uniqueCount(host) FROM Transaction WHERE appName='my-app-name'" } signal { aggregationWindow: 60 aggregationMethod: EVENT_FLOW aggregationDelay: 120 } terms: [{ threshold: 2 thresholdOccurrences: AT_LEAST_ONCE thresholdDuration: 600 operator: BELOW priority: CRITICAL }] valueFunction: SINGLE_VALUE violationTimeLimitSeconds: 86400 expiration: { expirationDuration: 120 openViolationOnExpiration: true } } ) { id name }}
Update the loss of signal settings of a condition
What if you want to update loss of signal parameters for an alert condition? The following mutation allows you to update a NRQL static condition with new expiration
values.
mutation { alertsNrqlConditionStaticUpdate( accountId: YOUR_ACCOUNT_ID id: YOUR_STATIC_CONDITION_ID condition: { expiration: { closeViolationsOnExpiration: true expirationDuration: 300 openViolationOnExpiration: false ignoreOnExpectedTermination: true } } ) { id expiration { closeViolationsOnExpiration expirationDuration openViolationOnExpiration ignoreOneExpectedTermination } }}
Customize gap filling
Gap filling replaces gap values in a time series with either the last value found or a static, arbitrary value of your choice. We fill gaps only after another data point has been received after the gaps in signal (after data reception has been restored).
You can configure both the type of filling and the value, if the type is set to static:
signal.fillOption
: Type of replacement value for lost data points. Values can be:NONE
: Gap filling is disabled.LAST_VALUE
: The last value seen in the time series.STATIC
: An arbitrary value, defined infillValue
.
signal.fillValue
: Value to use for replacing lost data points whenfillOption
is set toSTATIC
.
Important
Gap filling is also affected by expiration.expirationDuration
. When a gap is longer than the expiration duration, the signal is considered expired and the gap will no longer be filled.
For example, here's how to create a static NRQL condition with gap filling configured:
mutation { alertsNrqlConditionStaticCreate( accountId: YOUR_ACCOUNT_ID policyId: YOUR_POLICY_ID condition: { enabled: true name: "Example Gap Filling Condition" nrql: { query: "SELECT count(*) FROM Transaction" } terms: { operator: ABOVE priority: CRITICAL threshold: 1000 thresholdDuration: 300 thresholdOccurrences: ALL } valueFunction: SINGLE_VALUE violationTimeLimitSeconds: 28800 signal: { aggregationWindow: 60 aggregationMethod: EVENT_FLOW aggregationDelay: 120 fillOption: STATIC fillValue: 1 } } ) { id }}