After log event data has been shipped to New Relic, it can either be stored in our NRDB database or dropped (discarded). We can drop both log events and event attributes via drop filter rules.
You can manage drop filter rules using our Logs UI, as explained in this document. You can also use NerdGraph.
Drop filter rules help you accomplish several important goals:
- Lower costs by storing only logs relevant to your account.
- Protect privacy and security by removing personal identifiable information (PII).
- Reduce noise by removing irrelevant events and attributes.
Use caution when deciding to drop data. The data you drop is not recoverable. Before using this feature, review the responsibilities and considerations for dropping data.
A drop filter rule matches data based on a query. When triggered, the drop filter rule removes the matching data from the ingestion pipeline before it is written to NRDB.
This creates an explicit demarcation between the logs being forwarded from your domain and the data that New Relic collects. Since the data removed by the drop filter rule doesn't reach our backend, it cannot be queried: the data is gone and cannot be restored.
During the ingestion process, log data can be parsed, transformed, or dropped before being stored.
When creating drop rules, you are responsible for ensuring that the rules accurately identify and discard the data that meets the conditions that you have established. You are also responsible for monitoring the rule, as well as the data you disclose to New Relic.
New Relic cannot guarantee that this functionality will completely resolve data disclosure concerns you may have. New Relic doesn't review or monitor how effective the rules you develop are.
Creating rules about sensitive data can leak information about what kinds of data you maintain, including the format of your data or systems (for example, through referencing email addresses or specific credit card numbers). Any user with the relevant role-based access control permissions can view and edit all information in the rules you create.
To create and edit drop filters, you must have admin permissions in New Relic, or you must be a member of a role with create and edit permissions for Logging Parsing Rules.
Once a drop filter rule is active, it's applied to all log events ingested from that point onwards. Rules are not applied retroactively. Logs collected before creating a rule are not filtered by that rule.
one.newrelic.com > Logs: Filter or query the set of logs that contain the data you want to drop. Then, from Manage Data on the left nav of the Logs UI, click Create drop filter.
To create a new drop filter rule, you can use new or existing log queries.
- Go to one.newrelic.com > Logs.
- Filter or query to the specific set of logs that contain the data to be dropped.
- Once the query is active, from Manage Data on the left nav of the Logs UI, click Create drop filter.
- Recommendation: Change the drop rule's default name to a meaningful name.
- Choose to either drop the entire log event that matches the query or just a specific subset of attributes in the matching events.
- Review the log partitions where this drop rule applies.
- Save the drop filter rule.
The drop filters UI prompts you to select whether to drop logs based on the query or on specific attributes.
The default type of drop filter rule is to drop logs. This option drops the entire log events that match the filter or query. When creating a rule, try to provide a specific query that only matches log data that should be dropped.
Our drop filters process won't let you create drop filter rules without values in the matching query. This prevents badly formed rules from dropping all log data.
You can specify attributes to be dropped in a log event that matches your query. At least one or more attributes must be selected. Any attribute which is selected will be dropped; all remaining attributes will be kept and stored in NRDB.
We recommend this method for removing fields that could contain personal identifiable information (PII) or other sensitive attributes without losing valuable monitoring data.
To view or delete a drop filter rule:
- Go to one.newrelic.com > Logs.
- From Manage Data on the left nav of the Logs UI, click Drop filters.
- Click the delete icon next to the drop filter rule you want to remove.
Once deleted, rules no longer filter ingested log events.