• EnglishEspañol日本語한국어Português
  • Log inStart now

Query time parsing in logs

Are you looking for a quick way to visually extract attributes from your logs after they've been ingested into New Relic? Query time parsing lets you parse your logs directly in the UI without needing to write complex regular expressions or Grok patterns. You can use query time parsing to temporarily extract values from your logs and quickly perform a query on these variables. The results are shown instantly since parsing is performed at query time.

How query time parsing differs from ingest parsing

While both types of parsing make it easier for you to query logs, they have some significant differences:

  • Ingest parsing: Parsing during log ingestion is where you create parsing rules using Grok or regular expressions (or both). As log records are ingested at New Relic, your parsing rules are applied to create permanent attributes that are stored with your log data in NRDB. These attributes make it easier for you to query log data.

  • Query time parsing: In contrast to ingest parsing, query time parsing lets you create temporary attributes that will be used as query variables. You can then use these variables in NRQL queries to populate your log table. We automatically create the queries as you make selections in the UI.

You may also choose to use a combination of both parsing approaches. Review the table below to decide if query time parsing is right for you:

Description

Ingest parsing

Query time parsing

Recommended usage

Best for creating permanent attributes you can query in the future

Best for doing quick queries on attributes that aren't permanent

Parsing language

You create Grok patterns and regular expressions

New Relic creates queries for you using the NRQL aparse function

Timing

Applied at ingest

Applied when you query

Results

Makes permanent changes to stored log

Temporarily alters your logs in the UI

Live tail logs

Live tail logs include any extracted values from ingest parsing

Live tail logs don't include any extracted values from query time parsing

Exported logs

Exported logs include any extracted values from ingest parsing

Exported logs don't include any extracted values from query time parsing

Number of attributes

A maximum of 255 attributes is available at ingest (the actual number attributes you can parse at ingest depends on the nature of your logs)

You can parse a maximum of 32 temporary attributes across all rules for query time parsing

How to create a query time parsing rule

Here's a guide to creating query time parsing rules. The example shows how to extract the log level and scripted message values from the message attribute.

Select attribute value to parse

You can start creating a query time parsing rule by selecting an attribute value to parse.

  1. In the log table or in the Formatted tab of the log details view, highlight an anchor string that contains the values you want to extract. In this case, you'd highlight level=info msg="Running script". It looks like this in the logs table:

    Keep the following in mind:

    • Your highlighted text should include the value(s) you want to extract and the surrounding string characters that will help identify the location of the extracted value(s).
    • If your initial anchor string is anywhere in the middle of the original attribute value, include at least one character before and after the values you want to extract.
    • If you're highlighting the entire attribute value, you don't need to worry about characters before and after the values you're extracting.
    • You cannot highlight blob values to parse.
  2. Click the Create query time parsing rule option.

    Log table:

    Log details view:

Highlight and extract values

After you've clicked Create query time parsing rule, the editor displays the string you selected for parsing:

To extract values:

  1. Within the string, highlight the value(s) you want to extract (see Tips for extracting values).
  2. Under Parse as, enter a name for this temporary attribute that will be used as a query variable.
  3. Click Save, which replaces the value you highlighted with the variable you created.

Finish creating your query time parsing rule

After you've selected values, complete the following:

  1. In the editor, review the preview of the table showing how your rule will be applied to the log you selected.

  2. If you're interested in the NRQL function used in the query to get your logs, click on Query.

    The pattern string, which is used to find and extract value(s) from the selected attribute, is updated whenever a value is extracted.

  3. If you need to rename any variables before you create your rule, click on the name, provide a new name, and then click Save. You can also delete variables by clicking on the variable you wish to delete and clicking Delete.

    You can edit or delete variables by clicking on them.

  4. After you've finished reviewing and editing your variable names, click Create rule to finish creating your rule.

If you started creating a query time parsing rule from the Log details view, you need to close that view to see your results in the log table.

Log details

After you view your newly extracted values, close the Log details view.

Log table

The log table automatically refreshes and applies the generated query to parse your logs.

Manage your rules

While your rules are temporary and apply to your current user session, you can still perform a variety of tasks during your session.

Copyright © 2024 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.