Exploring through mass amounts of log data takes time. You can spend hours searching for a piece of information required to troubleshoot an issue. Inspecting log messages demands repetitive queries, usually constructed from a combination of time window, entity names, and log message content.
However, in most cases, log messages look the same, with only minor changes in certain variables. This is where log patterns kick in.
Focus on what's unusual
Log patterns employ advanced clustering algorithms to group together similar log messages automatically. With patterns, you can:
- Orient more quickly through millions of logs.
- Reduce the time it takes to identify unusual behavior in your log estate.
- Monitor the frequency of known patterns over time to focus your energy on what matters, and exclude what's irrelevant.
Our model also brings forward all the recent log messages in your account that were not clustered into a known pattern yet. We recommend that you start inspecting these unclustered patterns first.
Log patterns beta
Our log patterns feature is currently in beta, and so its availability is limited to specific participating accounts. If you can't find the Log patterns button in your New Relic account's Log management UI, contact your account representative and ask to participate.
|Log patterns beta||Limitations and considerations|
|Pricing||As part of the beta, log patterns currently report to a new event called
|Events sampling||During the beta, the scope of the collected data required to train the patterns model is limited to 10M events per day. We are sampling your logging estate using NRQL filters such as level =
|Log messages||The number of underlying log messages displayed currently is limited to 200 per pattern. To review a specific subset of logs, simply refine the query as needed.|
|FedRAMP||Our log patterns beta feature is not FedRAMP compliant. This is why FedRAMP accounts currently are not eligible to participate in the beta.|
|Support during beta||If you have questions during the beta phase, please contact your account rep or customer success rep.|
To start examining logs and identifying patterns:
- Go to one.newrelic.com > Log management, and use the account picker dropdown to select the target account where you want to explore patterns.
- At the top right corner of the Log management UI, click Log Patterns.
The log patterns panel shows information relevant for your selected account.
See patterns and recent unclustered logs
The log patterns UI is a different view of the log table. It displays the same logs in another way.
While the log table is ordered by timestamp by default, the log patterns are ordered by the number of their appearances, so that the most repetitive patterns are displayed at the top by default.
|If you want to...||Do this...|
|Identify unusual or different types of patterns over time||
Look at the line chart. The color-coded patterns correspond to the Plot column in the table.
|See the number of log messages that match each pattern||
Click a pattern, then use the expandable table. With each pattern, the varying parts of the log messages are highlighted, so you can easily identify differences across log lines.
The number of underlying log messages displayed is limited to 200 per pattern. To review a specific subset of logs, refine the query.
|Group and filter patterns by their attributes||
Use the query bar and time picker. As you apply different filters and time windows, the log patterns adjust to your new target data.
After clicking on a pattern, an expanded table view appears. On the left side you can see a sample of underlying logs, and on the right you can see a
Clicking each attribute will filter the query and render the line chart accordingly. We filter out some masks such as date, time, and UUID, since those usually don’t characterize the behavior of the pattern.
|Immediately create an alert from a pattern query||
Click the alert button.
|Troubleshoot log messages that haven't been clustered into a pattern||
Use the Uncategorized logs tab in the Log patterns UI.
Clicking on a specific log message will open the log message details panel you're familiar with from the Logs management page.
Troubleshoot uncategorized patterns
To review new log messages that have not yet been clustered into a known pattern, use the Uncategorized logs tab. These messages can also be valuable to detect new problems and troubleshoot incidents.
The table shows the unclustered log message timestamp and its content. To see more details, click the message.
Masked attributes and wildcards
With patterns, parts of the log messages are classified as variables and are substituted by masked attributes. The masking process supports and improves the clustering phase by allowing the algorithm to ignore changing details and focus on the repetitive structure.
Masked attributes include:
Masked attributes are highlighted and are easy to identify, as shown in the following example.
Log patterns extract other less trivial variables that don't belong to any masked attribute. These variables are indicated as wildcards
*. To reveal additional details, click the
Put the platform to work with patterns
Patterns are reported to New Relic as a new event named
LogPatterns. You can use them across the platform like any other event you're familiar with, such as
Log. For example, you can:
- Build your own dashboards with patterns, to monitor a specific pattern or group of patterns you care about.
- Create alerts for patterns by adding NRQL alerts.
- Use baseline alert conditions to detect anomalies in known log patterns.