• /
  • ログイン

Discover value in log data with patterns

Log patterns are the fastest way to discover value in log data without searching.

Log data is high volume telemetry with a low value per individual record. Searching can quickly lead to logs that provide a root cause explanation, but most data is repetitive and hard to contextualize when browsing. Patterns can make log data discoverable without spending a lot of time reading through low value data.

Log patterns main UI

Logs > Patterns: Use patterns as the basis for alerts when the frequency of important data changes, or for configuring drop rules to get rid of unnecessary repetitive data.

Technical overview

Log patterns functionality applies machine learning to normalize and group log messages that are consistent in format but variable in content. These grouped messages can be sorted, making it easy to find the most frequent or rarest sets of logs in your environment.

Use patterns as the basis for alerts when the frequency of important data changes, or to configure drop rules to get rid of unnecessary repetitive data.

Log patterns use advanced clustering algorithms to group together similar log messages automatically. With patterns, you can:

  • Orient more quickly through millions of logs.
  • Reduce the time it takes to identify unusual behavior in your log estate.
  • Monitor the frequency of known patterns over time to focus your energy on what matters, and exclude what's irrelevant.

Availability

The ability to configure this feature is dependent on role-based permissions. For example, accounts still using our original pricing model require an Owner or Admin role.

If you see Patterns are turned off in your Log management Patterns UI, click the Configure Patterns button and enable it. If you don't see patterns within 30 minutes of enabling the feature, there may be a lack of data with a message attribute for the system to create a pattern from it.

Log patterns

Limitations and considerations

Pricing

There is no separate pricing for log patterns. The only cost is for additional data generated and added to your log records.

A pattern attribute will be added to all logs that match a pattern. Attributes also may be added when common values are discovered, such as GUIDs, IP addresses, URL, or email addresses. These attributes are automatically extracted from the log message as part of the pattern process.

HITRUST accounts

The log patterns feature is not FedRAMP compliant. FedRAMP or other HITRUST accounts are not eligible to use patterns.

Parsing limits

We have a system of safety limits on memory and CPU resources when processing logs and their patterns. These parsing limits can have an impact on the data you get. For more information, see our documentation about parsing limits.

Get started

To start examining patterns:

  1. Go to one.newrelic.com > Logs, and use the account picker dropdown to select the target account where you want to explore patterns.
  2. In the left navigation of the Logs UI, click Patterns.

The main log UI changes to show patterns that match the query in the query bar.

Log patterns UI

Logs > Log patterns: The line chart shows the top 5 patterns over time. Use the time picker and query bar to adjust the results.

Explore log patterns

By default the log patterns UI first shows the most frequent occurrence of patterns. To sort to show the rarest patterns first, click the Count column. You can also use the query bar or attributes bar to filter your log patterns.

If you want to...

Do this...

Understand the rate of change in patterns

Look at the line chart. The color-coded patterns correspond to the plot column in the table. You can toggle individual plot patterns to narrow your focus.

See the individual log messages that match each pattern

Click pattern to expand the row and see a table of individual log records.

  • To see additional records, scroll up or down.
  • To explore an individual log in more detail, click it to open the details panel.

Group and filter patterns by their attributes

Use the query bar and time picker. As you apply different filters and time windows, the log patterns adjust to your new target data.

Create an alert from a pattern

Add the pattern to the query bar and run the query. Then click Create alert condition in the left nav.

Troubleshoot log messages that haven't been clustered into a pattern

Use the Logs with no pattern tab in the Log patterns UI.

Clicking a specific log message will open the log message details panel you're familiar with from the Logs management page.

Explore logs with no pattern

The Logs with no pattern tab groups all recent log messages in your account that were not clustered into a known pattern yet. These log messages don't represent any problem or flaw in the system; they have no pattern because they are too new to have been processed by the machine learning system. This makes them valuable to explore when you want to understand what has recently changed in your environment.

Log patterns UI: Logs With No Pattern

Logs > Log patterns: New Relic's log patterns feature automatically groups logs without a matching pattern.

For example:

  • Are any of these logs tied to a recent problem? This is a quick way to discover unique log data that is appearing for the first time in your environment.
  • Does your log data have a new format? Sometimes the logs don't represent a problem, but a new format of log data that deviates from the data model you expect your applications to follow.

Catching these logs early gives you the opportunity to ask developers to correct any deviations in their log output. The more consistent people are in the way log data is generated, the easier it becomes to use logs across a diverse set of teams.

Masked attributes and wildcards

Parts of the log messages in patterns are classified as variables and are substituted by masked attributes. The masking process supports and improves the clustering phase by allowing the algorithm to ignore changing details and focus on the repetitive structure.

Masked attributes include:

  • date_time
  • ip
  • url
  • uuid

Masked attributes are highlighted and are easy to identify, as shown in the following example.

Log patterns UI: example log pattern

Here is an example of a log pattern that has masked attributes.

Log patterns extract other less trivial variables that don't belong to any masked attribute. These variables are indicated as wildcards *.

Log patterns UI: wildcard example

Here is an example of how wildcards * group variables in the Log patterns UI.

Troubleshooting

Here are a few reasons why you might have patterns enabled but not see any pattern data. If you're sure none of the items below are true, get help from support.newrelic.com.

  • No data has arrived in the timeframe you're observing. Try expanding the time range you're viewing with the time picker.
  • It's been less than 24 hours since patterns were enabled in the account. This means the ML model may not be generated for the account yet.
  • None of the data coming in has a message field. Patterns will only be generated for values in the message field of a log record. If your logs don't contain message, there will be no data.

Put the platform to work with patterns

Patterns are a value that is enriched onto the existing log message as a new attribute named newrelic.logPattern. Anything you can do with logs generally can be done with log patterns, such as:

  • Build your own dashboards with patterns, to monitor a specific pattern or group of patterns you care about.
  • Create alerts for patterns by adding NRQL alerts.
  • Use baseline alert conditions to detect anomalies in known log patterns.

その他のヘルプ

さらに支援が必要な場合は、これらのサポートと学習リソースを確認してください:

問題を作成するこのページを編集する
Copyright © 2020 New Relic Inc.