Are you looking for a quick way to visually extract attributes from your logs after they've been ingested into New Relic? Query time parsing lets you parse your logs directly in the UI without needing to write complex regular expressions or Grok patterns. You can use query time parsing to temporarily extract values from your logs and quickly perform a query on these variables. The results are shown instantly since parsing is performed at query time.
To get started, watch this five-minute video or see the tips below:
How query time parsing differs from ingest parsing
While both types of parsing make it easier for you to query logs, they have some significant differences:
Ingest parsing: Parsing during log ingestion is where you create parsing rules using Grok or regular expressions (or both). As log records are ingested at New Relic, your parsing rules are applied to create permanent attributes that are stored with your log data in NRDB. These attributes make it easier for you to query log data.
Query time parsing: In contrast to ingest parsing, query time parsing lets you create temporary attributes that will be used as query variables. You can then use these variables in NRQL queries to populate your log table. We automatically create the queries as you make selections in the UI.
You may also choose to use a combination of both parsing approaches. Review the table below to decide if query time parsing is right for you:
Description | Ingest parsing | Query time parsing |
|---|---|---|
Recommended usage | Best for creating permanent attributes you can query in the future | Best for doing quick queries on attributes that aren't permanent |
Parsing language | You create Grok patterns and regular expressions | New Relic creates queries for you using the NRQL aparse function |
Timing | Applied at ingest | Applied when you query |
Results | Makes permanent changes to stored log | Temporarily alters your logs in the UI |
Live tail logs | Live tail logs include any extracted values from ingest parsing | Live tail logs don't include any extracted values from query time parsing |
Exported logs | Exported logs include any extracted values from ingest parsing | Exported logs don't include any extracted values from query time parsing |
Number of attributes | A maximum of 255 attributes is available at ingest (the actual number attributes you can parse at ingest depends on the nature of your logs) | You can parse a maximum of 32 temporary attributes across all rules for query time parsing |
How to create a query time parsing rule
Here's a guide to creating query time parsing rules. The example shows how to extract the log level and scripted message values from the message attribute.
Select attribute value to parse
You can start creating a query time parsing rule by selecting an attribute value to parse.
In the log table or in the Log details view, highlight an anchor string that contains the values you want to extract. In this case, you'd highlight
level=info msg="Running script". It looks like this in the logs table:Keep the following in mind:
- Your highlighted text should include the value(s) you want to extract and the surrounding string characters that will help identify the location of the extracted value(s).
- If your initial anchor string is anywhere in the middle of the original attribute value, include at least one character before and after the values you want to extract.
- If you're highlighting the entire attribute value, you don't need to worry about characters before and after the values you're extracting.
- You cannot highlight blob values to parse.
Click the Create query time parsing rule option.
Log table:
Log details view:
Highlight and extract values
After you've clicked Create query time parsing rule, the editor displays the string you selected for parsing:
To extract values:
- Within the string, highlight the value(s) you want to extract (see Tips for extracting values).
- Under Parse as, enter a name for this temporary attribute that will be used as a query variable.
- Click Save, which replaces the value you highlighted with the variable you created.
Finish creating your query time parsing rule
After you've selected values, complete the following:
In the editor, review the preview of the table showing how your rule will be applied to the log you selected.
If you're interested in the NRQL function used in the query to get your logs, click on Query.
The pattern string, which is used to find and extract value(s) from the selected attribute, is updated whenever a value is extracted.
If you need to rename any variables before you create your rule, click on the name, provide a new name, and then click Save. You can also delete variables by clicking on the variable you wish to delete and clicking Delete.
You can edit or delete variables by clicking on them. After you've finished reviewing and editing your variable names, click Create rule to finish creating your rule.
If you started creating a query time parsing rule from the Log details view, you need to close that view to see your results in the log table.
Log details
After you view your newly extracted values, close the Log details view.
Log table
The log table automatically refreshes and applies the generated query to parse your logs.
Manage your rules
While your rules are temporary and apply to your current user session, you can still perform a variety of tasks during your session.