• EnglishEspañol日本語한국어Português
  • Log inStart now

Discover value in log data with patterns

Log patterns are the fastest way to discover value in log data without searching.

Log data is high volume telemetry with a low value per individual record. Searching can quickly lead to logs that provide a root cause explanation, but most data is repetitive and hard to contextualize when browsing. Patterns can make log data discoverable without spending a lot of time reading through low value data.

one.newrelic.com > All capabilities > Logs > Patterns: Log patterns help you spot suspicious spikes or drops in log messages without having to spend extra time searching.

Technical overview

Log pattern functionality applies machine learning to normalize and group log messages that are consistent in format but variable in content. These grouped messages can be sorted, making it easy to find the most frequent or rarest sets of logs in your environment.

Use patterns as the basis for alerts when the frequency of important data changes, or to configure drop rules to get rid of unnecessary repetitive data.

Log patterns use advanced clustering algorithms to group together similar log messages automatically. With patterns, you can:

  • Orient more quickly through millions of logs.
  • Reduce the time it takes to identify unusual behavior in your log estate.
  • Monitor the frequency of known patterns over time to focus your energy on what matters, and exclude what's irrelevant.

Availability

If you see Patterns are turned off in your logs UI, click Configure patterns and enable it. If you don't see patterns within 30 minutes of enabling the feature, there may be a lack of data with a message attribute for the system to create a pattern from it.

Log patterns

Limitations and considerations

Pricing

There is no separate pricing for log patterns. The only cost is for additional data generated and added to your log records.

A pattern attribute will be added to all logs that match a pattern. Attributes also may be added when common values are discovered, such as GUIDs, IP addresses, URL, or email addresses. These attributes are automatically extracted from the log message as part of the pattern process.

HITRUST accounts

The log patterns feature is not FedRAMP compliant. FedRAMP or other HITRUST accounts are not eligible to use patterns.

Log pattern matching limits

We have a system of safety limits on memory and CPU resources when matching logs to patterns. These matching limits can have an impact on the percentage of logs that are able to be grouped into log patterns. However, log pattern matching is a "best effort" process: it is not an error if not all of your logs have the opportunity to be grouped by patterns. You will still get value from the grouping that is able to be done within resource limits.

Get started

To start examining patterns:

  1. Go to one.newrelic.com > All capabilities > Logs, and use the account picker dropdown to select the target account where you want to explore patterns.
  2. In the left navigation of the logs UI, click Patterns.

The main log UI changes to show patterns that match the query in the query bar. To look at logs for a specific time period, click that point (or click and drag an area) on the chart, or use the time picker.

Explore log patterns

By default the log patterns UI first shows the most frequent occurrence of patterns. To sort to show the rarest patterns first, click the Count column. You can also use the query bar or attributes bar to filter your log patterns.

If you want to...

Do this...

Understand the rate of change in patterns

Look at the line chart. The color-coded patterns correspond to the plot column in the table. You can toggle individual plot patterns to narrow your focus.

See the individual log messages that match each pattern

Click pattern to expand the row and see a table of individual log records.

  • To see additional records, scroll up or down.
  • To explore an individual log in more detail, click it to open the details panel.

Group and filter patterns by their attributes

Use the query bar and time picker. As you apply different filters and time windows, the log patterns adjust to your new target data.

Create an alert from a pattern

Add the pattern to the query bar and run the query. Then click Create alert condition in the left nav.

Troubleshoot log messages that haven't been clustered into a pattern

Use the Logs with no pattern tab in the Log patterns UI.

Clicking a specific log message will open the log message details panel you're familiar with from the logs UI.

Explore logs with no pattern

The Logs with no pattern tab groups all recent log messages in your account that were not clustered into a known pattern yet. These log messages don't represent any problem or flaw in the system; they have no pattern because they are too new to have been processed by the machine learning system. This makes them valuable to explore when you want to understand what has recently changed in your environment.

For example:

  • Are any of these logs tied to a recent problem? This is a quick way to discover unique log data that is appearing for the first time in your environment.
  • Does your log data have a new format? Sometimes the logs don't represent a problem, but a new format of log data that deviates from the data model you expect your applications to follow.

Catching these logs early gives you the opportunity to ask developers to correct any deviations in their log output. The more consistent people are in the way log data is generated, the easier it becomes to use logs across a diverse set of teams.

Masked attributes and wildcards

Parts of the log messages in patterns are classified as variables and are substituted by masked attributes. The masking process supports and improves the clustering phase by allowing the algorithm to ignore changing details and focus on the repetitive structure.

Masked attributes include:

  • date_time
  • ip
  • url
  • uuid

Masked attributes are highlighted and are easy to identify, as shown in the following example.

Log patterns extract other less trivial variables that don't belong to any masked attribute. These variables are indicated as wildcards *.

Troubleshooting

Here are a few reasons why you have patterns enabled but do not see any pattern data. If you're sure none of these items are true, get help from support.newrelic.com.

  • No data has arrived in the timeframe you're observing. Try expanding the time range you're viewing with the time picker.
  • It's been less than 24 hours since patterns were enabled in the account. This means the ML model may not be generated for the account yet.
  • None of the data coming in has a message field. Patterns will only be generated for values in the message field of a log record. If your logs don't contain message, there will be no data.

Put the platform to work with patterns

Patterns are a value that is enriched onto the existing log message as a new attribute named newrelic.logPattern. Anything you can do with logs generally can be done with log patterns, such as:

  • Build your own dashboards with patterns, to monitor a specific pattern or group of patterns you care about.
  • Create for patterns by adding NRQL alerts.
  • Use anomaly alert conditions to detect anomalies in known log patterns.
Copyright © 2024 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.