• EnglishEspañol日本語한국어Português
  • Log inStart now

Forward your logs using the infrastructure agent

Forwarding your logs to New Relic makes all of your logging data available in one location, providing deeper visibility into both your application and your platform performance data. With your logs in one spot, you can collect, process, explore, query, and alert on errors or anomalies found in your log data.

From your host's UI, your logs are placed in context of events for the selected time period. You can drill down into detailed data for any of the highlighted attributes.

How you forward your logs depends on how you installed the infrastructure agent, since the infrastructure agent enables the log forwarding feature. You can install the infrastructure agent via:

  • Guided install (recommended for most users)
  • Manual installation
  • Linux tarball

Important

The Linux version of infrastructure agent, specifically version 1.42.0, has transitioned from using the td-agent-bit package to the fluent-bit package. This change was necessitated by the fact that fluent-bit is no longer distributed in the td-agent-bit flavour after the major version 2.x update.

To ensure smooth operations and provide the option to revert to td-agent-bit in the event of any issues with the fluent-bit package, the infrastructure agent now installs both packages (td-agent-bit and fluent-bit). By default, the agent is configured to use fluent-bit.

See Rollback to Fluent Bit 1.9 for more information about how to rollback.

Tip

Got lots of logs? Check out our tutorial on how to optimize and manage them.

System requirements

Automatically forward logs with guided install

When you use our guided install to install the infrastructure agent, the log forwarding feature is automatically configured during the install process.

To initiate your install, choose your deployment method:

Amazon Linux
Amazon Linux
Ansible
Ansible
CentOS
CentOS
Debian
Debian
Red Hat
RHEL
SLES
SLES
Ubuntu
Ubuntu
Windows
Windows

Important

To forward logs from a Docker container, use this Docker image as the base image to be used with kubernetes logs integration, or you can build your own container with custom configurations for different environments.

Enable log forwarding on agent installed manually

To install the infrastructure agent manually, follow our tutorial to install the package manager, or check out our MSI installer (Windows).

Step 1. Configure the infrastructure agent

Configuration files direct your system to forward the log sources you want to appear in New Relic. You can add as many config files as you want. Our infrastructure agent uses .yml files to configure logging. If you install the infrastructure agent via Integrations & Agents in the UI, the file logging.yml is created automatically.

To add a new configuration file for the log forwarding feature:

  1. Navigate to the log forwarder configuration folder:

    • Linux: /etc/newrelic-infra/logging.d/
    • Windows: C:\Program Files\New Relic\newrelic-infra\logging.d\
  2. Create a logging.yml configuration file, and add the parameters you need. The logging.d directory has various .yml.example files you can use as a reference or starting point. For Windows examples, see our Github repo.

    # Log forwarder configuration file example
    # Source: file
    # Available customization parameters: attributes, max_line_kb, pattern
    logs:
    # Basic tailing of a single file
    - name: basic-file
    file: /var/log/logFile.log
    # File with spaces in its path. No need to use quotes.
    - name: file-with-spaces-in-path
    file: /var/log/folder with spaces/logFile.log
    # Specify a list of custom attributes, as key-value pairs, to be included
    # in each log record
    - name: file-with-attributes
    file: /var/log/logFile.log
    attributes:
    application: tomcat
    department: sales
    maintainer: example@mailprovider.com
    # Use wildcards to refer to multiple files having a common extension or
    # prefix. Newly generated files will be automatically detected every 60
    # seconds.
    #
    # WARNING: avoid using wildcards that include the file extension, since
    # it'll cause logs to be forwarded repeatedly if log rotation is enabled.
    - name: log-files-in-folder
    file: /var/log/logF*.log
    # Lines longer than 128 KB will be automatically skipped. Use 'max_line_kb'
    # to increase this limit.
    - name: log-file-with-long-lines
    file: /var/log/logFile.log
    max_line_kb: 256
    # Use 'pattern' to filter records using a regular expression
    - name: only-records-with-warn-and-error
    file: /var/log/logFile.log
    pattern: WARN|ERROR

The agent automatically processes new configuration files without having to restart the infrastructure monitoring service. The only exception to this is when configuring a custom Fluent Bit configuration.

Step 2. Set log forwarding parameters

You're required to set up a name and log source parameter in the log forwarding .yml config file. To start, define a name of the log or logs you want to forward to New Relic.

What you use for the log source depends on the location your logs are sourced from. Available options for the log source include:

Step 3. Define key attributes

Although these configuration parameters aren't required, we still recommend you apply these configurations to your logging.yml file so you get the most out of log forwarding.

Sample configuration file

Here is an example of a logging.d configuration file in YAML format. For more configuration examples, see the infrastructure agent repository.

Step 4. View your log data

If everything is configured correctly and your data is being collected, you should see logs and related telemetry data in these places:

  • Our logs UI
  • In the infrastructure UI, from the host table, click the icon for a specific host, and then click View logs.
  • Our tools for running NRQL queries. For example, you can execute a query like this:
SELECT * FROM Log

Enable logging for your on-host integrations

With the infrastructure agent installed, you can enable automatic log parsing and forwarding for our most popular on-host integrations with one step. To enable this feature, rename the on-host-log.yml.example file to on-host-log.yml. Once done, your integration's logs are automatically parsed and sent to New Relic.

This option is available for our supported Linux platforms.

To enable the on-host integration log forwarding feature:

Enable log forwarding on agent installed using Linux tarball

Our custom Linux installation process for infrastructure monitoring allows you to tailor all aspects of the installation process, and to place files and folders on your machine. If you choose the assisted or manual tarball installation process, follow these steps to implement the log forwarder feature:

  1. Create the following directories:

    • /var/db/newrelic-infra/newrelic-integrations/logging
    • /etc/newrelic-infra/logging.d
  2. Download and install New Relic's fluent-bit-package (RPM) by running a command similar to:

    bash
    $
    yum localinstall fluent-bit-<some-version>.rpm
  3. Download New Relic's fluentbit plugin and save it as /var/db/newrelic-infra/newrelic-integrations/logging/out_newrelic.so.

  4. Download or copy the parsers.conf file from this Github repository, and save it as /var/db/newrelic-infra/newrelic-integrations/logging/parsers.conf.

Did this doc help with your installation?

Troubleshooting

If you encounter problems with configuring your log forwarder, try these troubleshooting tips.

What's next?

Explore logging data across your platform with our logs UI.

Disable log forwarding

To disable log forwarding capabilities, go to your logging.d directory, and remove files with the .yml extension that were originally added during the configuration process.

  • Linux: /etc/newrelic-infra/logging.d/
  • Windows: C:\Program Files\New Relic\newrelic-infra\logging.d\
Copyright © 2024 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.