• /
  • Log in
  • Free account

Parsing

New Relic parses log data according to rules. Learn how logs parsing works, how to use built-in rules, and create custom rules.

Why it matters

Parsing is the process of splitting unstructured data into attribute/value pairs. You can use these attributes to facet or filter logs in useful ways. This in turn helps you build better charts and alerts.

A good example is a default NGINX access log containing unstructured text. It is useful for searching but not much else. Here's an example of a typical line:

127.180.71.3 - - [10/May/1997:08:05:32 +0000] "GET /downloads/product_1 HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)"

In an unparsed format, you would need to do a full text search to answer most questions. After parsing, the log is organized into attributes, like response code and request URL:

{
"remote_addr":"93.180.71.3",
"time":"1586514731",
"method":"GET",
"path":"/downloads/product_1",
"version":"HTTP/1.1",
"response":"304",
"bytesSent": 0,
"user_agent": "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)"
}

Parsing makes it easier to create custom queries that facet on those values. This helps you understand the distribution of response codes per request URL and quickly find problematic pages.

How does our log parsing work?

Here's an overview of how New Relic implements parsing of logs:

Log parsing

How it works

What

  • All parsing takes place against the message field; no other fields can be parsed.
  • Each parsing rule has a matching criteria. We recommend using the logtype attribute name for matching parsing rules to logs.

When

  • Parsing will only be applied once to each log message. If multiple parsing rules match the log, only the first that succeeds will be applied.
  • Parsing takes place during log ingestion, before data is written to NRDB. Once data has been written to storage, it can no longer be parsed.

How

  • Rules can be written in Grok, regex, or a mixture of the two. Grok is a collection of patterns that abstract away complicated regular expressions.
  • If the content of the message field is JSON, it will be parsed automatically.

New Relic's log ingestion pipeline can parse data by matching a log event to a rule that describes how the log should be parsed. There are two ways log events can be parsed:

Rules are a combination of matching logic and parsing logic. Matching is done by defining a query match on an attribute of the logs. Rules are not applied retroactively. Logs collected before a rule is created are not parsed by that rule.

The simplest way to organize your logs and how they are parsed is to include the logtype field in your log event. This tells New Relic what built-in ruleset to apply to the logs.

Important

Once a parsing rule is active, data parsed by the rule is permanently changed. This cannot be reverted.

Limits

Parsing is computationally expensive, which introduces risk. Poorly defined rules can consume a huge amount of memory and CPU resource while also taking a very long time to complete. In order to prevent problems there are two parsing limits: a per-message-per-rule and per-account limit.

Limit

Description

Per-message-per-rule

The per-message-per-rule limit prevents the time spent parsing any single message from being greater than 100 ms. If that limit is reached, the system will cease attempting to parse the log message with that rule.

The ingestion pipeline will attempt to run any other applicable on that message, and the message will still be passed through the ingestion pipeline and stored in NRDB. The log message will be in its original, unparsed format.

Per-account

The per-account limit exists to prevent accounts from using more than their fair share of resources. The limit considers the total time spent processing all log messages for an account per-minute.

The limit is not a fixed value; it scales up or down proportionally to the volume of data stored daily by the account and the environment size that is subsequently allocated to support that customer.

Tip

To easily check if your rate limits have been reached, go to your system Limits page in the New Relic UI.

Built-in parsing rulesets

Common log formats have well-established parsing rules already created for them. To get the benefit of built-in parsing rules, add the logtype attribute when forwarding logs. Set the value to something listed in the following table, and the rules for that type of log will be applied automatically.

List of built-in rulesets

The following logtype attribute values map to a standard parsing rulesets. See Built-in parsing rules to learn what fields are parsed for each rules.

logtype

Example matching query

alb

AWS Application Load Balancer

logtype:alb

apache

Apache Access

logtype:apache

cloudfront-web

CloudFront Web

logtype:cloudfront-web

elb

Amazon Elastic Load Balancer

logtype:elb

iis_w3c

IIS server logs - W3C format

logtype:iis_w3c

monit

Monit logs

logtype:monit

mysql-error

MySQL Error

logtype:mysql-error

nginx

NGINX access logs

logtype:nginx

nginx-error

NGINX error logs

logtype:nginx-error

route-53

Amazon Route 53 logs

logtype:route-53

syslog-rfc5424

Syslog

logtype:syslog-rfc5424

Add the logtype attribute

When aggregating logs, it's important to provide metadata that makes it easy to organize, search, and parse those logs. One simple way of doing this is to add the attribute logtype to the log messages when they are shipped. Built-in parsing rules are applied by default to certain logtype values.

Here are some examples of how to add logtype to logs sent by some of our supported shipping methods.

Create custom parsing rules

Many logs are formatted or structured in a unique way. In order to parse them, custom logic must be built and applied. New Relic's log management lets you create and manage custom parsing rules.

  1. Choose Manage parsing from the action menu dropdown.

    manage-parsing.png

  2. Click Create parsing rule.

  3. Choose an attribute and value to match on.

  4. Write your Grok pattern and test the rule.

  5. Enable and save the custom parsing rule.

To learn about Grok and custom parsing rules, read our blog post about how to parse logs with Grok patterns.

For more help

If you need more help, check out these support and learning resources:

Create issueEdit page
Copyright © 2021 New Relic Inc.