Data partitions are a way to group or organize log data for faster and more efficient querying. When a query targets a single partition, our logs UI:
- Scans less unrelated data
- Returns results faster
Accounts can have multiple partitions, and multiple partitions can be queried at the same time.
Data partitions also allow data to be mapped to an alternative, or “secondary” namespace with a fixed 30-day retention. This is useful for maintaining compliance with privacy-centric regulations and standards like the General Data Protection Regulation (GDPR).
Before you start creating partitions, make sure you have the required permissions and a plan for how to implement the partitions.
Logs are routed to partitions during the ingestion process, before data is written to NRDB. Partition rules won't affect logs that were ingested before the rule was created.
You can gain significant performance improvements with proper use of data partitions. Organizing your data into discrete partitions enables you to query them separately or all together. The goals of partitioning your data should be:
- Create data partitions that align with concepts in your environment or organization that are static or change infrequently (for example, by business unit, team, environment, service, etc.).
- Ensure each partition remains below 1 TB of daily ingest for optimal performance.
Having more partitions allows for more targeted searches, but creating too many partitions can make logs hard to find and increase administrative overhead. Finding the right balance is important. We support 100 partitions maximum per account, but the optimal number for most accounts is 10 to 15 partitions.
A partition's namespace determines its retention period. We offer two retention options:
- Standard: The account's default retention determined by your New Relic subscription. This is the maximum retention period available in your account and is the namespace you'll select for most of your partitions.
- Secondary: 30-day retention. All logs sent to a partition that's a member of the Secondary namespace will be purged on a rolling basis 30 days after having been ingested.
Secondary retention is not a cost control mechanism; data is billed on ingest.
If you want to manage your data partitions programmatically, you can use NerdGraph API explorer: one.newrelic.com > All capabilities > Apps > NerdGraph API explorer. The NerdGraph data partitions tutorial shows how to query, create, and delete data partitions using this API.
To the left of the logs query bar, click Data partitions, then create a partition name with the retention namespace, optional description, and matching criteria.
To create a new partition rule:
- Go to one.newrelic.com > All capabilities > Logs.
- To the left of the logs query bar, click Partition, then click Create new.
- Define a Partition name as an alphanumeric string that begins with
- Add an optional description.
- Select the retention namespace for the partition.
- Set your rule's Matching criteria: Enter a valid NRQL
WHEREclause to match the logs to store in this partition.
To view your partitions: click the Partition dropdown.
The default partition is
Log. Any log that isn't affected by a partition rule will be stored in the
Log partition by default.
You can query multiple partitions at the same time. For the best performance, select the smallest number of partitions possible.
To search partitions: To the left of the logs query bar, click Partition, and use the partition search bar.