Drop data using NerdGraph

For some types of data in New Relic, you can drop (not report) that data. You might want to do this:

  • To filter out unimportant, low-value data
  • To filter out potentially sensitive data

Requirements

You must have a user role with permissions for dropping data.

Currently the following types of data can be targeted by data dropping rules:

  • APM-reported events
  • Browser-reported events
  • Mobile-reported events
  • Synthetics-reported events
  • Custom events (like those generated by the APM agent APIs or the Event API)
  • Log data
  • Trace spans
  • Metrics from the metric API
    • Currently only DROP_ATTRIBUTES, not DROP_DATA​​​​, can be used on metrics
    • Metrics generated from events-to-metrics are not eligible

Support for additional types are planned for the future.

Create drop data rule

Use caution when deciding to drop data. The data you drop is not recoverable. Before using this feature, please review caution information below.

To drop data, create a NerdGraph-format drop rule that includes:

  • A NRQL string that specifies what data types to drop
  • An action type specifying how to apply the NRQL string

You can form and make the call in the NerdGraph explorer.

There are two ways to drop data:

  • Drop entire data types or a data subset (with optional filter). This uses the DROP_DATA action type and uses NRQL of the form:

    SELECT * FROM DATA_TYPE_1, DATA_TYPE_2 (WHERE OPTIONAL_FILTER)

    For this type of drop rule, you cannot use anything other than * in the SELECT clause.

  • Drop attributes from data types (with optional filter). This uses the DROP_ATTRIBUTES action type and uses NRQL of the form:

    SELECT dropAttr1, dropAttr2 FROM DATA_TYPE (WHERE OPTIONAL_FILTER)

    For this type of drop rule, you must pass in a non-empty list of raw attributes names.

NRQL restrictions

Not all NRQL clauses make sense for generating drop rules. You can provide a WHERE clause to select data with specific attributes. Other features such as TIMESERIES, COMPARE WITH, FACET, and other clauses cannot be used.

The two action types have these restrictions:

  • DROP_DATA can use only SELECT *.
  • DROP_ATTRIBUTES requires use of SELECT with "raw" attributes (attributes with no aggregator function applied). This also means you cannot use SELECT *. Additionally, there are some attributes that are integral to their data type and cannot be dropped (such as timestamp on event data). If you include them, registration will fail.

Example drop rules

Here are some example drop rules:

Drop two event types

Let's say you notice you have some event types being sent to New Relic that are not important to you. Also, stopping the source from sending those event types quickly is unrealistic, requiring changes to agents and/or API instrumentation. Using a drop rule is an easier way to accomplish the same goal.

Here is an example NerdGraph call that drops two event types: Event1 and Event2.

mutation {
    nrqlDropRulesCreate(accountId: YOUR_ACCOUNT_ID, rules: [
        {
            action: DROP_DATA
            nrql: "SELECT * FROM Event1, Event2"
            description: "Drops all data for Event1 and Event2."
        }
    ])
    {
        successes { id }
        failures {
            submitted { nrql }
            error { reason description }
        }
    }
}
Drop events meeting certain criteria

Let’s say you have a high volume custom event type that arrives from multiple sources. If you don't find all of that data important, you can use a drop rule. Here is an example of a drop rule that filters out events based on specific criteria.

mutation {
    nrqlDropRulesCreate(accountId: YOUR_ACCOUNT_ID, rules: [
        {
            action: DROP_DATA
            nrql: "SELECT * FROM MyCustomEvent WHERE appName='LoadGeneratingApp' AND environment='development'"
            description: "Drops all data for MyCustomEvent that comes from the LoadGeneratingApp in the dev environment, because there is too much and we don’t look at it."
        }
    ])
    {
        successes { id }
        failures {
            submitted { nrql }
            error { reason description }
        }
    }
}
Drop sensitive attributes while maintaining the rest of the data

Let's say you noticed an event has attributes that contain Personally Identifiable Information (PII). You are working to update your services to stop sending the data, but until then, you need to cease storing further PII in New Relic. Although you could drop all of the data as it comes in the door with a DROP_DATA rule, the rest of the data still provides value. Therefore, you can register a drop rule to remove only the offending PII from your data:

mutation {
    nrqlDropRulesCreate(accountId: YOUR_ACCOUNT_ID, rules: [
        {
            action: DROP_ATTRIBUTES
            nrql: "SELECT userEmail, userName FROM MyCustomEvent
            description: "Removes the user name and email fields from MyCustomEvent"
        }
    ])
    {
        successes { id }
        failures {
            submitted { nrql }
            error { reason description }
        }
    }
}

Verify your drop rule works

After you create a drop rule, verify that it is working as expected. The rule should take effect quickly after a successful registration, so try running a TIMESERIES version of the query you registered to see that the data drops off.

Drop rule type NRQL
DROP_DATA

Drop rule NRQL:

SELECT * FROM MyEvent WHERE foo = bar

Validation NRQL:

SELECT count(*) FROM MyEvent WHERE foo = bar TIMESERIES

This should drop to 0. To verify that it did not affect any thing else, invert the WHERE clause.

DROP_ATTRIBUTES

Drop rule NRQL:

SELECT dropAttr1, dropAttr2 FROM MyEvent WHERE foo = bar

Validation NRQL:

SELECT count(dropAttr1), count(dropAttr2) FROM MyEvent WHERE foo = bar TIMESERIES

Both lines should drop to 0. To verify that it did not affect events that contained these attributes and still should, invert the WHERE clause.

View rules

Here is an example NerdGraph call that returns the drop rules set on an account:

{
    actor {
        account(id: YOUR_ACCOUNT_ID) {
            nrqlDropRules {
                list {
                    rules {
                        id
                        nrql
                        accountId
                        action
                        createdBy
                        createdAt
                        description
                    }
                    error { reason description } 
                }
            }
        }
    }
}

Delete drop rules

Here is an example NerdGraph call deleting two specific drop rules:

mutation {
    nrqlDropRulesDelete(accountId: YOUR_ACCOUNT_ID, ruleIds: ["48", "98"]) {
	successes {
	    id
	    nrql
	    accountId
	    action
	    description
	}
	failures {
	    error { reason description }
	    submitted { ruleId accountId }
        }
    }
}

Audit drop rule history

To see who created and deleted drop rules, query your account audit logs. The list endpoint also includes the user ID of the person who created the rule.

Cautions when dropping data

When creating drop rules, you are responsible for ensuring that the rules accurately identify and discard the data that meets the conditions that you have established. You are also responsible for monitoring the rule, as well as the data you disclose to New Relic.

New Relic cannot guarantee that this functionality will completely resolve data disclosure concerns you may have. New Relic does not review or monitor how effective the rules you develop are.

Creating rules about sensitive data can leak information about what kinds of data you maintain, including the format of your data or systems (for example, through referencing email addresses or specific credit card numbers). Rules you create, including all information in those rules, can be viewed and edited by any user with the relevant role-based access control permissions.

Learn more

Recommendations for learning more:

For more help

If you need more help, check out these support and learning resources: