After log event data has been sent to New Relic, it can either be stored in our NRDB database or dropped (discarded). To drop log data, you can use the logs management UI, as explained in this document. You can also use NerdGraph to drop data. NerdGraph is our GraphQL Explorer.
Savings, security, speed
Drop filter rules help you accomplish several important goals:
- Lower costs by storing only the logs relevant to your account.
- Protect privacy and security by removing personal identifiable information (PII).
- Reduce noise by removing irrelevant events and attributes.
Caution
Use caution when deciding to drop data. The data you drop is not recoverable. Before using this feature, review the responsibilities and considerations for dropping data.
How drop filter rules work
A drop filter rule matches data based on a query. When triggered, the drop filter rule removes the matching data from the ingestion pipeline before it is written to the New Relic database (NRDB).
This creates an explicit demarcation between the logs being forwarded from your domain and the data that New Relic collects. Since the data removed by the drop filter rule doesn't reach our backend, it cannot be queried: the data is gone and cannot be restored.
During the ingestion process, customer log data can be parsed, transformed, or dropped before being stored in the New Relic database (NRDB).
Cautions when dropping data
When creating drop rules, you are responsible for ensuring that the rules accurately identify and discard the data that meets the conditions that you have established. You are also responsible for monitoring the rule, as well as the data you disclose to New Relic.
New Relic cannot guarantee that this functionality will completely resolve data disclosure concerns you may have. New Relic doesn't review or monitor how effective the rules you develop are.
Creating rules about sensitive data can leak information about what kinds of data you maintain, including the format of your data or systems (for example, through referencing email addresses or specific credit card numbers). Any user with the relevant role-based access control permissions can view and edit all information in the rules you create.
Caution
Drop rules are partition specific. When you create a drop rule you must also specify the partition(s) where the logs are located. If you later change or remove the partition associated with your logs, it's likely that logs will no longer match your drop rule(s). If you are using Partitions and Drop rules, take care to validate your drop rule(s) are still valid after updating Partition rules.
Create drop filter rules
For permissions-related requirements, see Drop data requirements.
Once a drop filter rule is active, it's applied to all log events ingested from that point onwards. Rules are not applied retroactively. Logs collected before creating a rule are not filtered by that rule.
Filter or query the set of logs that contain the data you want to drop. Then, from Manage data on the left nav of the logs UI, click Create drop filter.
To create a new drop filter rule, you can use new or existing log queries. There are two ways you can create a drop filter:
Manage drop filter rules via NerdGraph API
To manage your drop filter rules programmatically, you can use NerdGraph, our GraphQL Explorer, to create, query, and delete your drop filter rules.
Types of drop filter rules
The drop filters UI prompts you to select whether to drop logs based on the query or on specific attributes.
Drop log events
The default type of drop filter rule is to drop logs. This option drops the entire log events that match the filter or query. When creating a rule, try to provide a specific query that only matches log data that should be dropped.
Our drop filters process won't let you create drop filter rules without values in the matching query. This prevents badly formed rules from dropping all log data.
Drop attributes
You can specify attributes to be dropped in a log event that matches your query. At least one or more attributes must be selected. Any attribute which is selected will be dropped; all remaining attributes will be kept and stored in NRDB.
Tip
We recommend this method for removing fields that could contain personal identifiable information (PII) or other sensitive attributes without losing valuable monitoring data.
View or delete drop filter rules
After you delete a drop filter rule from here, the ingested log events are no longer filtered.
To view or delete a drop filter rule:
- Go to one.newrelic.com > All capabilities > Logs.
- From Manage data on the left nav of the logs UI, click Drop filters.