• English日本語한국어
  • Log inStart now

How to manage large log volume

Modern systems create a large volume of log data. You might be dealing with hundreds of gigabytes to dozens of terabytes today, and the amount will continue to increase as your system scales. When you need to search through your logs, you'll encounter hours of toil trying to uncover valuable and relevant logs. Sending all your logs to a log management tool can help reduce this toil, but you'll quickly encounter organizational hurdles and rising costs as you ingest more logs. New Relic solves this problem by providing tools to ingest only valuable logs to reduce cost, a unified UI to correlate your logs to your services, and various ways to organize your logs before your drown in them.

Whether you're setting up a log management platform for the first time or you're migrating to New Relic, this tutorial will walk you through how to use New Relic to manage a large amount of log data. You'll start by forwarding your logs to New Relic, which means sending your log data to New Relic automatically. You'll then identify what logs to ingest and which to drop. Finally you'll organize your logs through partitions and parsing.

Choosing a log management platform

Once you've identified you have a problem with managing logs, it's time to choose a log management platform. There are many platforms out there. Some focus on quick automation but sacrifice ease-of-use. Others focus on complex features, but obscure their pricing.

New Relic's philosphy when it comes to focuses on three things: we want our logs solution to be flexible, transparent, and usage-based. Let's quickly talk about what these mean:

  • Flexible: Everyone needs different things from their logs. Some may need to ingest a large amount for record keeping while some may need to ingest a small amount. Some may need to heavily parse their logs while other may barely parse their logs at all. Our log management platform gives you tools to manage what you send us.
  • Transparent: There are no surprises in billing. New Relic charges you only for the data you ingest at a fixed price per gigabyte.
  • Usage-based: Only pay for logs you ingest. Not all logs are valuable, so there's no use in ingesting and paying for logs you will never use. In this tutorial we'll explore how to selectively ingest logs in an affordable and effective manner.

Let's begin: forward your logs

To forward your log data to New Relic, choose one or more of these options:

Log forwarding option

When to use

Install

APM agent

By default, our APM agents do three things:

  • Add metadata to your logs, which gives you logs in context (ability to see logs data in various relevant places in our platform)

  • Forward your logs to New Relic.

  • Report performance metrics for your application Read more about our APM capabilities

    This is a popular option for DevOps teams and smaller organizations because it lets you easily report application logs, with no additional third-party solutions required. Learn more about APM logs.

Go agent
Java agent
.NET agent
Node.js agent

Infrastructure agent

With our infrastructure agent, you can capture any logs present on your host, including your app logs.

Compared to using an APM agent to report logs, this can take a little more setting up but gives you much more powerful options (for example, ability to collect custom attributes, which you can't do with agents).

Third-party log services

We have a wide range of integrations for other log services, including Amazon, Microsoft, Fluentd, Fluent Bit, Kubernetes, Logstash, and more.

Third-party log solutions

Log API or TCP endpoint

When you want more precision control about what and how logs are sent to New Relic.

Log API or TCP endpoint

OpenTelemetry SDK

Forward logs from your apps to an OpenTelemetry collector, which can forward them to New Relic via OTLP.

OpenTelemetry collector

For more on log forwarding options and specific use cases, see Forward logs.

1Get started

You are here

2Filter and reduce your log ingest

3Organize your logs

Copyright © 2024 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.