Log patterns are the fastest way to discover value in log data without searching.
Log data is high volume telemetry with a low value per individual record. Searching can quickly lead to logs that provide a root cause explanation, but most data is repetitive and hard to contextualize when browsing. Patterns can make log data discoverable without spending a lot of time reading through low value data.
one.newrelic.com > Logs > Patterns: Use patterns as the basis for alerts when the frequency of important data changes, or for configuring drop rules to get rid of unnecessary repetitive data.
Log patterns functionality applies machine learning to normalize and group log messages that are consistent in format but variable in content. These grouped messages can be sorted, making it easy to find the most frequent or rarest sets of logs in your environment.
Use patterns as the basis for alerts when the frequency of important data changes, or to configure drop rules to get rid of unnecessary repetitive data.
Log patterns use advanced clustering algorithms to group together similar log messages automatically. With patterns, you can:
- Orient more quickly through millions of logs.
- Reduce the time it takes to identify unusual behavior in your log estate.
- Monitor the frequency of known patterns over time to focus your energy on what matters, and exclude what's irrelevant.
The ability to configure this feature is dependent on role-based permissions. If you see Patterns are turned off in your Log management Patterns UI, click the Configure Patterns button and enable it. If you don't see patterns within 30 minutes of enabling the feature, there may be a lack of data with a message attribute for the system to create a pattern from.
Limitations and considerations
There is no separate pricing for log patterns. The only cost is for additional data generated and added to your log records.
A pattern attribute will be added to all logs that match a pattern. Attributes also may be added when common values are discovered, such as GUIDs, IP addresses, URL, or email addresses. These attributes are automatically extracted from the log message as part of the pattern process.
The log patterns feature is not FedRAMP compliant. FedRAMP or other HITRUST accounts are not eligible to use patterns.
Availability for this feature depends on whether you have a US or EU region account:
We have a system of safety limits on memory and CPU resources when processing logs and their patterns. These parsing limits can have an impact on the data you get. For more information, see our documentation about parsing limits.
To start examining patterns:
- Go to one.newrelic.com > Log management, and use the account picker dropdown to select the target account where you want to explore patterns.
- In the left navigation of the Log management UI, click Patterns.
The main log UI changes to show patterns that match the query in the query bar.
one.newrelic.com > Log management > Log patterns: The line chart shows the top 5 patterns over time. Use the time picker and query bar to adjust the results.
By default the log patterns UI first shows the most frequent occurrence of patterns. To sort to show the rarest patterns first, click the Count column. You can also use the query bar or attributes bar to filter your log patterns.
If you want to...
Understand the rate of change in patterns
Look at the line chart. The color-coded patterns correspond to the plot column in the table. You can toggle individual plot patterns to narrow your focus.
See the individual log messages that match each pattern
Click pattern to expand the row and see a table of individual log records.
Group and filter patterns by their attributes
Use the query bar and time picker. As you apply different filters and time windows, the log patterns adjust to your new target data.
Create an alert from a pattern
Add the pattern to the query bar and run the query. Then click Create alert condition in the left nav.
Troubleshoot log messages that haven't been clustered into a pattern
Use the Logs with no pattern tab in the Log patterns UI.
Clicking a specific log message will open the log message details panel you're familiar with from the Logs management page.
The Logs with no pattern tab groups all recent log messages in your account that were not clustered into a known pattern yet. These log messages don't represent any problem or flaw in the system; they have no pattern because they are too new to have been processed by the machine learning system. This makes them valuable to explore when you want to understand what has recently changed in your environment.
one.newrelic.com > Log management > Log patterns: New Relic's log patterns feature automatically groups logs without a matching pattern.
- Are any of these logs tied to a recent problem? This is a quick way to discover unique log data that is appearing for the first time in your environment.
- Does your log data have a new format? Sometimes the logs don't represent a problem, but a new format of log data that deviates from the data model you expect your applications to follow.
Catching these logs early gives you the opportunity to ask developers to correct any deviations in their log output. The more consistent people are in the way log data is generated, the easier it becomes to use logs across a diverse set of teams.
Parts of the log messages in patterns are classified as variables and are substituted by masked attributes. The masking process supports and improves the clustering phase by allowing the algorithm to ignore changing details and focus on the repetitive structure.
Masked attributes include:
Masked attributes are highlighted and are easy to identify, as shown in the following example.
one.newrelic.com > Log management > Log patterns: Here is an example of a pattern that has masked attributes.
Log patterns extract other less trivial variables that don't belong to any masked attribute. These variables are indicated as wildcards
one.newrelic.com > Log management > Log patterns: Here is an example of how wildcards
* group variables.
Here are a few reasons why you might have patterns enabled but not see any pattern data. If you're sure none of the items below are true, get help from support.newrelic.com.
- No data has arrived in the timeframe you're observing. Try expanding the time range you're viewing with the time picker.
- It's been less than 24 hours since patterns were enabled in the account. This means the ML model may not be generated for the account yet.
- None of the data coming in has a
messagefield. Patterns will only be generated for values in the
messagefield of a log record. If your logs don't contain
message, there will be no data.
Patterns are a value that is enriched onto the existing log message as a new attribute named
newrelic.logPattern. Anything you can do with logs generally can be done with log patterns, such as:
- Build your own dashboards with patterns, to monitor a specific pattern or group of patterns you care about.
- Create alerts for patterns by adding NRQL alerts.
- Use baseline alert conditions to detect anomalies in known log patterns.
If you need more help, check out these support and learning resources:
- Browse the Explorers Hub to get help from the community and join in discussions.
- Find answers on our sites and learn how to use our support portal.
- Run New Relic Diagnostics, our troubleshooting tool for Linux, Windows, and macOS.
- Review New Relic's data security and licenses documentation.