--- title: Drop data using Pipeline Control cloud rules source: https://docs.newrelic.com/docs/new-relic-control/pipeline-control/cloud-rules/api-reference --- One way to [manage your data ingestion](https://docs.newrelic.com/docs/data-apis/manage-data/manage-data-coming-new-relic) is by using Pipeline Control cloud rules. To create Pipeline cloud rules, you must be on New Relic Compute usage-based pricing. There are two categories of rules you can create: - **Drop data rule** - Drop entire data types or a data subset _(with optional filter)_, with NRQL in the form of: ```sql DELETE FROM DATA_TYPE_1, DATA_TYPE_2 (WHERE OPTIONAL_FILTER) ``` - **Drop attribute rule** - Drop attributes from data types _(with optional filter)_, with NRQL in the form of: ```sql DELETE dropAttr1, dropAttr2 FROM DATA_TYPE (WHERE OPTIONAL_FILTER) ``` - For this type of rule, you must pass in a non-empty list of **raw** attributes names in the `DELETE` clause. > #### 💡 TIP > > Pipeline Control cloud rules only apply to data that arrives from the moment you create the rule, they don't delete data that's [already been ingested](https://docs.newrelic.com/docs/telemetry-data-platform/ingest-manage-data/manage-data/manage-data-retention#data-deletion). To learn more about what data counts as billable or not, see [Data ingest](https://docs.newrelic.com/docs/accounts/accounts-billing/new-relic-one-pricing-billing/data-ingest-billing/#usage-calculation). ## Cloud rules data scope [#data-scope] Use cloud rules to target the following data types: - APM-reported events - Browser-reported events - Mobile-reported events - Synthetics-reported events - Custom events (like those generated by the [APM agent APIs](https://docs.newrelic.com/docs/insights/insights-data-sources/custom-data/insert-custom-events-new-relic-apm-agents) or the [Event API](https://docs.newrelic.com/docs/insights/insights-data-sources/custom-data/introduction-event-api)) - Log data (you can also [use the UI to drop data](https://docs.newrelic.com/docs/logs/ui-data/drop-data-drop-filter-rules)) - Distributed tracing spans - [Default infrastructure monitoring events](https://docs.newrelic.com/docs/infrastructure/manage-your-data/data-instrumentation/default-infrastructure-monitoring-data) and [infrastructure integrations](https://docs.newrelic.com/docs/integrations/infrastructure-integrations/get-started/introduction-infrastructure-integrations) events. > #### ⚠️ IMPORTANT > > From February 2026 onwards, drop rules targeting infrastructure events in `SystemSample`, `ProcessSample`, `NetworkSample`, and `StorageSample` will correctly drop aggregated data, too. - [Dimensional metrics](https://docs.newrelic.com/docs/data-apis/understand-data/new-relic-data-types/#dimensional-metrics). Some caveats: - For metrics generated by the [events-to-metrics service](https://docs.newrelic.com/docs/data-ingest-apis/get-data-new-relic/metric-api/events-metrics-service-create-metrics): Cloud rules won't work but these metrics can be stopped or attributes pruned by disabling or re-configuring the events-to-metric rule. - Metric timeslice data can't be dropped with cloud rules. For more information about APM metric timeslice data see [this doc](https://docs.newrelic.com/docs/data-apis/understand-data/new-relic-data-types/#timeslice-data). ## NRQL restrictions [#restrictions] - The limit on NRQL query length is **4096** characters. If it exceeds the length, the `INVALID_NRQL_TOO_LONG` error occurs. If you need to drop data based on a longer query that cannot be split, contact your [New Relic support](https://support.newrelic.com). - `JOIN` and [subqueries](https://docs.newrelic.com/docs/query-your-data/nrql-new-relic-query-language/get-started/subqueries-in-nrql) are not supported. - You can provide a [`WHERE`](https://docs.newrelic.com/docs/query-data/nrql-new-relic-query-language/getting-started/nrql-syntax-clauses-functions#sel-where) clause to select data with specific attributes. - You shouldn't use entity tags in a [`WHERE`](https://docs.newrelic.com/docs/query-data/nrql-new-relic-query-language/getting-started/nrql-syntax-clauses-functions#sel-where) clause to drop data, because entity tags are fetched in a separate process and are not attributes of the MELT data evaluated by the pipeline cloud rule. - Features such as `LIMIT`, `TIMESERIES`, `COMPARE WITH`, `FACET`, and other clauses cannot be used. - `SINCE` and `UNTIL` are not supported. If you have time-specific rules (say, drop everything until a time in the future), use `WHERE timestamp < (epoch milliseconds in the future)`. - You can't use `SINCE` to drop historical data. Cloud rules only apply to data reported after the rule was created. If you need to delete data that has already been reported, refer to [Delete existing data](https://docs.newrelic.com/docs/data-apis/manage-data/manage-data-retention/#deleting-data) or contact [New Relic support](https://support.newrelic.com). ## Audit rule history [#history] To see who created and deleted cloud rules, query your [account audit logs](https://docs.newrelic.com/docs/insights/use-insights-ui/manage-account-data/query-account-audit-logs-nrauditevent). The [list endpoint](#view-rules) also includes the user ID of the person who created the rule. ## Cautions when dropping data [#caution] Cloud rules apply to each data point independently. For example, let's look at the following three data drop rules: > #### ⚠️ IMPORTANT > > When creating rules, you are responsible for ensuring that the rules accurately identify and discard the data that meets the conditions that you have established. You are also responsible for monitoring the rule, as well as the data you disclose to New Relic. ```sql 1. DELETE FROM MyEvent WHERE myAttr not in ('staging') 2. DELETE FROM MyEvent WHERE myAttr not in ('production') 3. DELETE FROM MyEvent WHERE myAttr in ('development') ``` These three rules are applied independently to each data point; in summary, all `MyEvent` events containing `myAttr` with any value will be dropped: - `myAttr: 'staging'` -> matches rule 2 - `myAttr: 'production'` -> matches rule 1 - `myAttr: 'development'` -> matches rules 1, 2, and 3 - `myAttr: 'uuid-random-string'` -> matches rules 1 and 2 New Relic cannot guarantee that this functionality will completely resolve data disclosure concerns you may have. New Relic does not review or monitor how effective the rules you develop are. Always test and retest your queries and, after the drop rule is created, make sure it works as intended. Creating rules about sensitive data can leak information about what kinds of data you maintain, including the format of your data or systems (for example, through referencing email addresses or specific credit card numbers). Rules you create, including all information in those rules, can be viewed and edited by any user with the relevant role-based access control permissions. Only new data will be dropped. Existing data [cannot be edited or deleted](https://docs.newrelic.com/docs/telemetry-data-platform/ingest-manage-data/manage-data/manage-data-retention#data-deletion). # Managing cloud rules [#how-to] To create and edit rules, you can either use the Pipeline Control UI or the [NerdGraph](https://docs.newrelic.com/docs/apis/nerdgraph/get-started/introduction-new-relic-nerdgraph) API explorer _(**[one.newrelic.com](https://one.newrelic.com) > Apps > NerdGraph API explorer**)_. > #### ⚠️ CAUTION > > Use caution when deciding to drop data. The data you drop can't be recovered. For more details on potential issues, see [Caution notes](#caution). ## Use case examples [#example-mutations] **Drop two event types** Let's say you notice you have some event types being sent to New Relic that are not important to you. Also, stopping the source from sending those event types quickly is unrealistic, requiring changes to agents and/or API instrumentation. Using a cloud rule is an easier way to accomplish the same goal. Here is an example NerdGraph call that drops two event types: `Event1` and `Event2`. ````graphql mutation { entityManagementCreatePipelineCloudRule( pipelineCloudRuleEntity: { description: "Event1 and Event2 are unimportant, see ticket DM-1234", name: "Drop all data for Event1 and Event2", nrql: "DELETE FROM Event1, Event2", scope: { id: "your_nr_account_id", type: ACCOUNT } } ) { entity { id name nrql } } } ``` ```` **Drop events meeting certain criteria** Let’s say you have a high volume custom event type that arrives from multiple sources. If you don't find all of that data important, you can use a cloud rule. Here is an example of a cloud rule that filters out events based on specific criteria. ````graphql mutation { entityManagementCreatePipelineCloudRule( pipelineCloudRuleEntity: { description: "Drops all data for MyCustomEvent that comes from the LoadGeneratingApp in the dev environment, because there is too much and we don’t look at it", name: "Drop MyCustomEvent from LoadGeneratingApp in dev", nrql: "DELETE FROM MyCustomEvent WHERE appName='LoadGeneratingApp' AND environment='development'", scope: { id: "your_nr_account_id", type: ACCOUNT } } ) { entity { id name nrql } } } ``` ```` **Drop sensitive attributes while maintaining the rest of the data** Let's say you noticed an event has attributes that contain Personally Identifiable Information (PII). You are working to update your services to stop sending the data, but until then, you need to cease storing further PII in New Relic. Although you could drop all of the data as it comes in the door with `Drop data`, the rest of the data still provides value. Therefore, you can register a cloud rule to remove only the offending PII from your data: ````graphql mutation { entityManagementCreatePipelineCloudRule( pipelineCloudRuleEntity: { description: "Removes the user name and email fields from MyCustomEvent", name: "Drop username and email from MyCustomEvent", nrql: "DELETE userName, userEmail FROM MyCustomEvent", scope: { id: "your_nr_account_id", type: ACCOUNT } } ) { entity { id name nrql } } } ``` ```` ## Verify your rule works [#verify] After you create a cloud rule, you might wish to verify that it is working as expected. The rule should take effect quickly after a successful registration, so try running a `TIMESERIES` version of the query you registered to see that the data drops off. _Note: Timeseries data is rendered with event time (not processing time) as the x-axis. Since New Relic accepts data with a timestamp up to twenty-four hours in the future, you might see some data that was sent to New Relic before the rule was created but with an event timestamp past rule creation._ | Cloud rule type | NRQL | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Drop data` | **Cloud rule NRQL:** ````sql DELETE FROM MyEvent WHERE foo = bar ``` **Validation NRQL:** ```sql SELECT count(*) FROM MyEvent WHERE foo = bar TIMESERIES ``` This should drop to 0. To verify that it did not affect any thing else, invert the `WHERE` clause. ```` | | `Drop attributes` | **Cloud rule NRQL:** ````sql DELETE dropAttr1, dropAttr2 FROM MyEvent WHERE foo = bar ``` **Validation NRQL:** ```sql SELECT count(dropAttr1), count(dropAttr2) FROM MyEvent WHERE foo = bar TIMESERIES ``` Both lines should drop to 0. To verify that it did not affect events that contained these attributes and still should, invert the `WHERE` clause. ```` | # NerdGraph examples [#examples] ## Create cloud rules [#create-rules] When creating a cloud rule, the `enabled` field is optional and defaults to `true`. If you wish to create a rule in a disabled state, you must explicitly set `enabled: false`. ### Drop data: ```graphql mutation { entityManagementCreatePipelineCloudRule( pipelineCloudRuleEntity: { description: "Since we only care about MyEvent in staging and production, let's drop all MyEvent data in the test environment", name: "Drop MyEvent in test environment", nrql: "DELETE FROM MyEvent WHERE environment = 'test'", scope: { id: "your_nr_account_id", type: ACCOUNT } } ) { entity { id name nrql enabled } } } ``` ### Drop attributes: ```graphql mutation { entityManagementCreatePipelineCloudRule( pipelineCloudRuleEntity: { description: "We don't care about jvmId and targetAttr in the test environment, let's drop those attributes", name: "Drop jvmId and targetAttr from MyEvent in test environment", nrql: "DELETE jvmId, targetAttr FROM MyEvent WHERE environment = 'test'", scope: { id: "your_nr_account_id", type: ACCOUNT } } ) { entity { id name nrql enabled } } } ``` ## Update a cloud rule [#update-rule] You can modify existing cloud rules to adjust their filtering criteria, data types, deployment scope or operational status. > #### ⚠️ IMPORTANT > > Cloud rules only apply to data that arrives from the moment you create or update the rule. They do not affect data that has already been ingested. Always test and monitor your updated rules to ensure they're working as intended. To update an existing Cloud rule, you'll use the `updatePipelineCloudRule` mutation. This allows you to change the rule's name, description, and, crucially, its NRQL. ### Identify the rule You'll need the unique entity ID of the cloud rule you wish to update. You can find this ID by querying your rules using NerdGraph. For example: ```graphql actor { entityManagement { entitySearch( query: "type = 'PIPELINE_CLOUD_RULE'" ) { results { entities { id name enabled } } } } } ``` ### Construct your mutation: Use the `entityManagementUpdatePipelineCloudRule` mutation, providing the rule's `id` and a `pipelineCloudRuleEntity` object containing the fields you want to change. - **`id`:** The entity ID of the cloud rule you identified in step 1. - **`pipelineCloudRuleEntity`:** An object containing the updated details. You can provide any or all of the below fields: - **`nrql`:** The updated NRQL query that defines what data or attributes to drop. This should be a `DELETE NRQL` query (for example, `DELETE FROM MyEvent or DELETE attribute FROM MyEvent`). - **`name`:** The new name for your rule (optional). - **`description`:** The new description for your rule (optional). - **`enabled`:** A boolean to instantly activate (true) or deactivate (false) the rule (optional). Note: Omitting this field retains the rule's current status. Suppose you have an existing rule that drops `dropAttr1` from `MyEventToDrop` eventType. If you want to change it to drop `dropAttr2` instead, call the following mutation: ```graphql mutation { entityManagementUpdatePipelineCloudRule( id: "MTAyNTY1MHxOR0VQfFBJUEVMSU5FX0NMT1VEX1JVTEV8MDE5ODgwOTUtMmZjNC03MTQ3LTkxMjQtZDk3YjhiY2Y4NGNj", # Replace with your rule's actual entity ID pipelineCloudRuleEntity: { description: "Since we do not need dropAttr2 ingested we would drop this attribute.", name: "Update for Dropping dropAttr2 from MyEventToDrop", nrql: "DELETE dropAttr2 FROM MyEventToDrop" } ) { entity { id type name description nrql enabled } } } ``` ### Disabling a Cloud Rule [#disable-rule] To disable a rule from dropping data, set the enabled field to `false` in your update mutation. ```graphql mutation { entityManagementUpdatePipelineCloudRule( id: "MTAyNTY1MHxOR0VQfFBJUEVMSU5FX0NMT1VEX1JVTEV8MDE5ODgwOTUtMmZjNC03MTQ3LTkxMjQtZDk3YjhiY2Y4NGNj" pipelineCloudRuleEntity: { enabled: false # Stops the rule from dropping data } ) { entity { id enabled } } } ``` ### Re-Enabling a Cloud Rule [#re-enable-rule] When you update a rule from `enabled: false` to `enabled: true`, the system performs an internal entitlement check to ensure the re-activation is authorized. This is critical for maintaining New Relic's operational policies. > #### ⚠️ IMPORTANT > > The entitlement check is performed against the following internal policies: > Policy Name: legacy_drop_rules_eol (Product Line: generic) > Policy Name: pipeline_cloud_drop_rules_ccu (Product Line: nr_queries) - If the rule was disabled by a customer: The rule will re-enable successfully. - If the rule was disabled by New Relic (for example, EOL policy): The request will fail unless the user possesses the necessary internal entitlements to override the policy. ```graphql mutation { entityManagementUpdatePipelineCloudRule( id: "MTAyNTY1MHxOR0VQfFBJUEVMSU5FX0NMT1VEX1JVTEV8MDE5ODgwOTUtMmZjNC03MTQ3LTkxMjQtZDk3YjhiY2Y4NGNj" pipelineCloudRuleEntity: { enabled: true # Re-activates the rule } ) { entity { id enabled } } } ``` ### Verify the update - **Check the API response:** The entity object in the response will reflect the updated name, description, NRQL and enabled status. - **Monitor your data:** Send new data that would be affected by the rule and confirm that the dropping behavior has changed as expected. For example, if you changed from dropping attr1 to attr2, ensure attr1 is now ingested and attr2 is no longer appearing. ## Delete a cloud rule [#delete-rule] ```graphql mutation { entityManagementDelete(id: "MTAyNTY1MHxOR0VQfFBJUEVMSU5FX0NMT1VEX1JVTEV8MDE5NWI0NDYtNjk5My03NGE5LWEyYjktMzBjMzQ1ODM0NTUz") { id } } ``` ## View cloud rules [#view-rules] Get a single cloud rule: ```graphql { actor { entityManagement { entity(id: "MTAyNTY1MHxOR0VQfFBJUEVMSU5FX0NMT1VEX1JVTEV8MDE5NWI0M2UtYmFhNy03NDk3LWI0N2ItNjUyMmEzZDFmZTFi") { id ... on EntityManagementPipelineCloudRuleEntity { id name description nrql enabled metadata { createdBy { id } createdAt } } } } } } ``` List all cloud rules: ```graphql { actor { entityManagement { entitySearch(query: "type = 'PIPELINE_CLOUD_RULE'") { entities { id type ... on EntityManagementPipelineCloudRuleEntity { id name nrql enabled } metadata { createdBy { id } } } } } } } ``` ## Non-droppable events and attributes [#non-droppable-data] You cannot drop the following events and attributes using cloud rules: **Non-droppable events** | Event type | Description | | -------------------- | ------------------------------------------------------------------------- | | `NrAuditEvent` | Important auditing information about the API usage that can’t be dropped. | | `NrIntegrationError` | Important information about issues between you and New Relic. | **Non-droppable attributes** | Attribute | Description | | | ---------------------------- | -------------------------------- | - | | Any `nr.` prefixed attribute | Used internally by New Relic | | | `timestamp` | Integral to all data types | | | `appId` | Integral to all data types | | | `metricName` | Integral to the metric data type | | | `instrumentation.provider` | Used for billing and usage data | | ## Drop attributes on dimensional metric rollups [#drop-attributes-on-dimensional-metric-rollups] [Dimensional metrics](https://docs.newrelic.com/docs/data-apis/understand-data/new-relic-data-types/#metrics-conceptual) aggregate metrics into rollups for long term storage and as a way to optimize longer term queries. [Metric cardinality limits](https://docs.newrelic.com/docs/data-apis/ingest-apis/metric-api/metric-api-limits-restricted-attributes) are applied to this data. You can use this feature to decide which attributes you don't need for long term storage and query, but would like to maintain for real time queries. For example, adding `containerId` as an attribute can be useful for live troubleshooting or recent analysis, but may not be needed when querying over longer periods of time for larger trends. Due to how unique something like `containerId` can be, it can quickly drive you towards your [metric cardinality limits](https://docs.newrelic.com/docs/data-apis/ingest-apis/metric-api/metric-api-limits-restricted-attributes) which when hit stops the synthesis of rollups for the remainder of that UTC day. This feature also allows you to keep the [high cardinality](https://docs.newrelic.com/docs/data-apis/ingest-apis/metric-api/NRQL-high-cardinality-metrics/) attributes on the raw data and drop it from rollups which gives you more control over how quickly you approach your cardinaliity limits. ### Usage **Drop attributes from dimensional metrics rollups** (with optional filter). This uses NRQL of the form: ```sql DELETE dropAttr1, dropAttr2 FROM MetricAggregate (WHERE OPTIONAL_FILTER) ``` Here is an example NerdGraph request: ```graphql mutation { entityManagementCreatePipelineCloudRule( pipelineCloudRuleEntity: { description: "We don't care about targetAttr in the test environment in dimensional metric rolloups, let's drop those attributes", name: "Drop targetAttr from Metric aggregate rollups in test environment", nrql: "DELETE targetAttr FROM MetricAggregate WHERE environment = 'test'", scope: { id: "your_nr_account_id", type: ACCOUNT } } ) { entity { id name nrql enabled } } } ``` To verify it's working, wait 3 to 5 minutes for the rule to be picked up and for aggregate data to be generated. Then assuming the example NRQL above is your pipeline control cloud rule, run the following queries: ```sql SELECT count(targetAttr) FROM Metric WHERE metricName = 'some.metric' TIMESERIES SINCE 2 hours ago SELECT count(targetAttr) FROM MetricRaw WHERE metricName = 'some.metric' TIMESERIES SINCE 2 hours ago ``` The first query retrieves metric rollups and should drop to 0 since `containerId` has been dropped per the new drop rule. The second query retrieves metric raws using the `MetricRaw` event type and should continue to hold steady since raw data is not impacted by the new drop rule. For more information on how to see the impact this will have on your cardinality, check out [Understand and query high cardinality metrics](https://docs.newrelic.com/docs/data-apis/ingest-apis/metric-api/NRQL-high-cardinality-metrics). ### Restrictions All restrictions that apply to drop attribute rules apply to drop attributes from dimensional metric rollup rules with the additional restriction that you can only target the `MetricAggregate` data type. They also do not work on `Metric` queries targeting data created by an [events to metrics](https://docs.newrelic.com/docs/data-ingest-apis/get-data-new-relic/metric-api/events-metrics-service-create-metrics) rule or on `Metric` queries targeting [timeslice data](https://docs.newrelic.com/docs/data-apis/understand-data/metric-data/query-apm-metric-timeslice-data-nrql). ## Learn more Recommendations for learning more: - [NerdGraph basics and terminology](https://docs.newrelic.com/docs/apis/nerdgraph/get-started/introduction-new-relic-nerdgraph#terminology) - [NRQL basics](https://docs.newrelic.com/docs/query-data/nrql-new-relic-query-language/getting-started/introduction-nrql) - Browse the [Support Forum](https://support.newrelic.com/s/hubtopic/Topic__c/Default?c__categories=%5B%7B%22icon%22%3A%22standard%3Adefault%22%2C%22id%22%3A%22a6c8W000000EesuQAC%22%2C%22sObjectType%22%3A%22Category__c%22%2C%22title%22%3A%22Dashboards%22%2C%22titleFormatted%22%3A%22Dashboards%22%7D%5D) for community discussions about cloud rules. - For a deep dive into managing data ingest for a complex organization, see [Data ingest governance](https://docs.newrelic.com/docs/new-relic-solutions/observability-maturity/operational-efficiency/intro-data-governance).