Live archives allows you to store logs for as long as you need. New Relic has a thirty-day retention for logs by default, but with live archives, you can extend that period for up to seven years. Like old tax documents, sometimes you don't need your data close at hand, but it's safe to keep the information available for reference if needed.
Is live archives for you? Here are some example use cases:
- Internal retention requirements: Have an internal retention period where you would like to keep logs and be able to query them only when necessary? Live archives can help with extended storage and querying capability.
- Trends and long-term debugging: Long term log storage can help improve year-over-year performance. For example, if you manage a retail company, comparing logs from one holiday season to the next can improve performance.
How live archives works
The live archives feature offers long-term log data storage that you intend to query sparingly for a lower storage cost. With live archives, you can have peace of mind knowing your logs are available for a longer period for potential audits or long-term debugging, but you will only be charged for querying when you query old data.
Get started
Enable live archives
First, you need to request live archives capabilities.
- On the Data retention UI page, located in the data management hub.
- Scroll down to Logging live archives.
- Hit Request. A member of our team will reach out to you.
Once you've enabled live archives, you need to edit your data retention.
- On the Data retention UI page, located in the data management hub select Edit data retention.
- Scroll down to Logging live archives and enter the total number of days you'd like your data stored. Keep in mind that this includes the standard retention days.
- Click Apply changes.
Keep in mind that you will need Admin capabilities to edit data retention.
Go to one.newrelic.com > Data management hub > Data retention > Select logging live archives > Edit data retention: Select your archive duration to store your logs for as long as you need.
Create a partition using live archives
A partition is a way to group logs based on specific criteria. Creating partitions can help you filter out the logs you don't need and keep the ones you do with live archives.
- Go to one.newrelic.com > All capabilities > Logs.
- To the left of the logs query bar, click Data partitions, then click Create data partition.
- Define a Partition name as an alphanumeric string that begins with Log_.
- Select the field Use live archives on this partition.
- The Total effective retention days will be updated to reflect the regular retention and the live archives retention.
Go to one.newrelic.com > All capabilities > Logs > Data partitions to create a partition using live archives.
You can also edit an existing partition clicking the icon on the Data partitions page.
Query live archives with logs syntax
There are two ways to query live archives: with Lucene or NRQL. If you want to use logs syntax that you're familiar with outside of New Relic, we suggest querying in the Logs UI. You'll need the Live archives query capability to query live archives.
- Go to one.newrelic.com > All capabilities > Logs.
- Using the search bar, run your desired query. Practicing your query first is best to ensure it works as expected.
- Select the dates you'd like to query.
- A screen will show you that you're about to query live archives. Querying live archives is more expensive, so only select Query live archives if you're ready.
Go to one.newrelic.com > All capabilities > Logs to query data stored with live archives.
Query live archives with NRQL
If you want to query using the same language you use for errors inbox or alerts, we suggest querying in NRQL. You'll need the Live archives query capability to query live archives.
- Go to one.newrelic.com > All capabilities > Logs.
- Enter the NRQL query you want to run.
- Practice the query first to make sure it works as expected.
- Toggle Use live archives.
- A screen will show you that you're about to query live archives. Querying live archives is more expensive, so only select Query live archives if you're ready.
Go to one.newrelic.com > All capabilities > Logs, then select NRQL to query data stored with live archives using the query builder.
Query live archives using Nerdgraph API
You can also query your live archives data using the Nerdgraph API. You'll need the Live archives query capability to query live archives. You must include the {eventNamespaces: "Logging:Archive"} in the NRQL options to specify the live archives eventNamespace.
Here is an example query:
query { actor { account(id: 1234567) { nrql( query: "SELECT count(*) FROM Log SINCE '2024-02-01 12:00:00' UNTIL '2024-02-01 13:00:00'" options: {eventNamespaces: "Logging:Archive"} async: true ) { results } } }}See your data consumption
Live archives is billed differently than the rest of your logs. You can read more about it in our Live archives storage: Billing and rules documentation. Live archives has three billing pillars.
- Data: You will be charged for when you originally ingest the regular logs.
- Storage: Live archives charges a fee for storing your data with live archives.
- Compute: Queries of live archives data will be billed in Compute Capacity Units. See our Usage plan documentation for more information about Compute Capacity Units.
To see your data storage on the Plan & Usage page:
- Go to one.newrelic.com, click the user menu, and then click Administration to open the Plan & usage page.
- Scroll down to Retention (GB-month).
- Click View more details for more information about the storage breakdown.
Once you query live archives, you can explore your compute consumption in the compute management hub.
- Go to one.newrelic.com, click the user menu, and then click Administration.
- Select Compute management.
- Scroll down to Product capabilities.
- Select Logs.
- Facet by Data category to view compute consumption for live archives.
Example
Let's say you need to check if a user logged into a machine from their computer to fulfill a ticket on September 1st. You can use live archives to help confirm that this took place by completing the following steps:
- Check the time frame that the user logged into their machine.
- Test your query in Logs to ensure it's working as expected.
Example query:
"logtype":"linux_audit" "type":"USER_LOGIN" "hostname":"apache_svr01"- Change the time selector to search for the dates the user logged into their machine.
- Select Query live archives.
Data deletion
If at any point you want to reduce your data retention, please note:
- You will need Admin privileges to edit data retention.
- You can't recover data if you reduce data retention later. Please choose and configure your retention periods carefully.
- Your data deletion is effective on the day you reduce your retention.
- You can request to delete your data for any given set of days. We'll delete all live archives data for the selected dates.
- Days start and end on UTC boundaries, so when making a request, ensure it aligns with UTC. This can mean adding or subtracting a day.
Important
For the avoidance of doubt, New Relic has no obligation to retain data in live archives after account downgrade or lapse in contract, or otherwise if you do not meet applicable commercial requirements.