Forward your logs using the infrastructure agent

You can forward your logs to New Relic using our infrastructure agent. This makes all of your logging data available in one location, providing deeper visibility both into application and platform performance data.

Enable log forwarding using the infrastructure agent

To enable log forwarding through the infrastructure agent:

1. Start by checking the what you need.
2. Install the infrastructure agent, version 1.11.4 or higher.
3. Configure your log sources and other parameters in the logging.d directory.
4. Generate some traffic and wait a few minutes, then check your account for data.
5. Explore your log data in the Logs UI.

What you need

The log forwarding feature is compatible with the following operating systems:

Operating system	Supported version
Amazon Linux	Amazon Linux 2
CentOS	Version 7 or higher
Debian	Version 9 ("Stretch") or higher
Red Hat Enterprise Linux (RHEL)	Version 7 or higher
SUSE Linux Enterprise Server (SLES)	Version 12
Ubuntu	Versions 16.04.x and 18.04.x (LTS versions)
Windows	Windows Server 2012, 2016, and 2019, and their service packs. Windows 10

To use the log forwarder of the infrastructure agent, make sure you meet the following requirements:

• Infrastructure agent version 1.11.4 or higher

• Windows: Install the Microsoft Visual C++ Redistributable: x64 or x86.

Install the infrastructure agent

Starting with version 1.11.4, the infrastructure agent can forward logs to New Relic. To install and run the agent, use a package manager (Linux) or the MSI installer (Windows).

If you don't have a New Relic account yet, or prefer to follow the procedure manually, see our tutorial.

Test log forwarding

To quickly check that the log forwarding feature works, follow these steps:

1. Navigate to the logging forwarder configuration folder:

   • Linux: /etc/newrelic-infra/logging.d/

   • Windows: C:\Program Files\New Relic\newrelic-infra\logging.d\

2. Create a configuration file (for example, logs.yml) with this content:

   • Linux example:

     # Remember to only use spaces for indentation
     logs:
       - name: "test_log"
         file: /var/log/test.log
     

   • Windows example:

     # Remember to only use spaces for indentation
     logs:
       - name: "test_log"
         file: 'C:\ProgramData\New Relic\newrelic-infra\newrelic-infra.log'

3. Run the following command to append a test log message to your log file:

   echo "This is a test message." >> /PATH/TO/YOUR/LOG/FILE
   

   For example:

   echo "This is a test message." >> /var/log/test.log

4. Search New Relic Logs for test message.

Configure the infrastructure agent

Configuration files describe which log sources are forwarded. You can add as many config files as you want, and set as many sources as you need per config file.

To add a new configuration file for the log forwarding feature:

1. Navigate to the logging forwarder configuration folder:

   • Linux: /etc/newrelic-infra/logging.d/

   • Windows: C:\Program Files\New Relic\newrelic-infra\logging.d\

2. Create a configuration file with the parameters you need. Use our sample config file as reference.

The agent automatically processes new configuration files without having to restart it.

Log forwarding parameters

The log forwarder config supports the following parameters (for examples, see the sample configuration).

name

Name of the log or logs.

attributes

List of custom attributes, as key-value pairs, that can be used to send additional data with the logs which you can then query. For example, you can enable built-in parsing rules by setting the logtype attribute.

Example:

 ...
 - name: tcp-simple-test 
    tcp:
      uri: tcp://0.0.0.0:1234
      format: none
      separator: \t
    attributes: # You can add custom attributes to any source of logs
      tcpFormat: none
      logtype: nginx # See https://docs.newrelic.com/docs/logs/log-management/ui-data/new-relic-logs-parsing-built-rules-custom-parsing
      someOtherAttribute: associatedValue

file

Path to the log file or files. The agent tracks changes on the log files in a way similar to tail -f shell.

Your file can point to a specific log file or multiple ones by using wildcards applied to names and extensions; for example, /logs/*.log

Example:

logs:
  - name: file-with-attributes
    file: /var/log/test.log # Path to a single file or pattern
    pattern: Error # Regular expression to filter log entries

Wildcards can also be used in place of directories in a file path, which can be used to tail files located in different directories.

Example:

logs:
  - name: docker-logs
    file: /var/lib/docker/containers/*/*.log # Path to multiple folders and files
    pattern: redis # Regular expression to filter log entries

We recommend increasing the file descriptor limit on Linux hosts running Fluent Bit by adding the following to the host's /etc/security/limits.conf file:

root soft nofile 65536
root hard nofile 65536
*soft nofile 65536
*hard nofile 65536

Once you add the changes above, reboot the host to ensure your changes are applied.

systemd (Linux only)

Service name. Once the systemd input is activated, log messages are collected from the journald daemon in Linux environments.

Example:

logs:
  - name: systemd-example
    systemd: cupsd

syslog (Linux only)

Syslog data source.

Parameters:

• uri: Syslog socket. Format varies depending on the protocol:
  • TCP/UDP network sockets: [tcp/udp]://LISTEN_ADDRESS:PORT
  • Unix domain sockets: unix_[tcp/udp]:// + /socket/path
• parser: Syslog parser. Default is rfc3164. Use rfc5424 if your messages include fractional seconds. Note: rfc3164 currently does not work on SuSE.
• unix_permissions: default is 0644 for domain sockets; this limits entries to processes running as root. You can use 0666 to listen for non-root processes, at your own risk.

When running the agent in privileged mode, ports and sockets must be available or owned by nri-agent, with 0666 file permissions, so that other processes can write logs to the sockets.

logs:
  # TCP network socket
  - name: syslog-tcp-test
    syslog:
      uri: tcp://0.0.0.0:5140 # Use the tcp://LISTEN_ADDRESS:PORT format
      parser: rfc5424 # Default syslog parser is rfc3164
  # UDP network socket
  - name: syslog-udp-test
    syslog:
      uri: udp://0.0.0.0:6140 # Use the udp://LISTEN_ADDRESS:PORT format
    max_line_kb: 35
  # Unix TCP domain socket
  - name: syslog-unix-tcp-test
    syslog:
      uri: unix_tcp:///var/unix-tcp-socket-test
      unix_permissions: 0666 # Default is 0644. Change at your own risk
  # Unix UDP domain socket
  - name: syslog-unix-udp-test
    syslog:
      uri: unix_udp:///var/unix-udp-socket-test
      parser: rfc5424

tcp

Logs retrieved over TCP connections.

Parameters:

• uri: TCP/IP socket to listen for incoming data. The URI format is tcp://LISTEN_ADDRESS:PORT
• format: format of the data. It can be json or none.
• separator: If format: none is used, you can define a separator string for splitting records (default: \n).

logs:
  - name: tcp-simple-test 
    tcp:
      uri: tcp://0.0.0.0:1234 # Use the tcp://LISTEN_ADDRESS:PORT format
      format: none # Raw text - this is default for 'tcp'
      separator: \t # String for separating raw text entries
    max_line_kb: 32
  - name: tcp-json-test 
    tcp:
      uri: tcp://0.0.0.0:2345 # Use the tcp://LISTEN_ADDRESS:PORT format
      format: json

max_line_kb

Maximum size of log entries/lines in KB. If log entries exceed the limit, they are skipped. Default is 128.

winlog

Collect events from Windows log channels.

Parameters:

• channel: name of the channel logs will be collected from.
• collect-eventids: a list of Windows Event IDs to be collected and forwarded to New Relic. Event ID ranges are supported.
• exclude-eventids: a list of Windows Event IDs to be excluded from collection. Event ID ranges are supported.

Add event IDs or ranges to collect-eventids or exclude-eventids to forward or drop specific events. exclude-eventids takes precedence over collect-eventids if the same event ID is present in both sections.

Example:

logs:
  - name: windows-security
    winlog: 
       channel: Security
       collect-eventids:
        - 4624
        - 4265
        - 4700-4800
       exclude-eventids:
        - 4735

pattern

Regular expression for filtering records. Only supported for the tail, systemd, syslog and tcp (only with format none) sources.

This field works in a way similar to grep -E in Unix systems. For example, for a given file being captured, you can filter for records containing either WARN or ERROR using:

- name: only-records-with-warn-and-error
  file: /var/log/logFile.log
  pattern: WARN|ERROR

No filtering is applied by default.

fluentbit

External Fluent Bit configuration and parser files. If defined, they are merged with the existing configuration and parser files generated by the Infrastructure agent.

The infrastructure agent processes the configuration files located in the logging.d directory and will generate a run-time Fluent Bit configuration file that contains the appropriate [INPUT], [FILTER] and [OUTPUT] sections. Optionally, it will also declare an @INCLUDE in case you provided an external Fluent Bit configuration file via the fluentbit option. The run-time file does not define a [SERVICE] section, leaving all default Fluent Bit configuration values. You can still override Fluent Bit's default settings by defining your own [SERVICE] section in your external Fluent Bit configuration file and include it via the fluentbit option.

Parameters:

config_file: path to an existing Fluent Bit configuration file. Note that any overlapping source results in duplicate messages in New Relic Logs.

parsers_file: path to an existing Fluent Bit parsers file. The following parser names are reserved: rfc3164, rfc3164-local and rfc5424.

Sample configuration file

Here is an example of a logging.d/ configuration file in YAML format. For more configuration examples, see the infrastructure agent repository.

logging.d/sample.yaml

 # Remember to only use spaces for indentation

logs:
  # Example of 'file' source
  - name: file-with-attributes
    file: /var/log/test.log # Path to a single file or pattern
    attributes: # You can use custom attributes to enrich your data
      logtype: nginx
      team: The A Team
    pattern: Error # Regular expression to filter log entries

  # Example of 'systemd' source (Linux only)
  - name: systemd-example
    systemd: cupsd

  # Examples of 'syslog' source, one per protocol
  # TCP network socket
  - name: syslog-tcp-test
    syslog:
      uri: tcp://0.0.0.0:5140 # Use the tcp://LISTEN_ADDRESS:PORT format
      parser: rfc5424 # Default syslog parser is rfc3164
  # UDP network socket
  - name: syslog-udp-test
    syslog:
      uri: udp://0.0.0.0:6140 # Use the udp://LISTEN_ADDRESS:PORT format
    max_line_kb: 35

  # Paths for Unix sockets are defined by combining protocol and path:
  # unix_udp:// + /path/socket - for example, unix_udp:///tmp/socket
  # Unix TCP domain socket
  - name: syslog-unix-tcp-test
    syslog:
      uri: unix_tcp:///var/unix-tcp-socket-test
      unix_permissions: 0666 # Default is 0644. Change at your own risk
  # Unix UDP domain socket
  - name: syslog-unix-udp-test
    syslog:
      uri: unix_udp:///var/unix-udp-socket-test
      parser: rfc5424

  # Examples of 'tcp' source for formats 'none' and 'json'
  - name: tcp-simple-test 
    tcp:
      uri: tcp://0.0.0.0:1234 # Use the tcp://LISTEN_ADDRESS:PORT format
      format: none # Raw text - this is default for 'tcp'
      separator: \t # String for separating raw text entries
    attributes: # You can add custom attributes to any source of logs
      tcpFormat: none
      someOtherAttribute: associatedValue
    max_line_kb: 32
  - name: tcp-json-test 
    tcp:
      uri: tcp://0.0.0.0:2345 # Use the tcp://LISTEN_ADDRESS:PORT format
      format: json
    attributes:
      tcpFormat: json
      yetAnotherAttribute: 12345

  # Example of Fluent Bit configuration import 
  - name: fluentbit-import
    fluentbit: 
      config_file: /path/to/fluentbit.config
      parsers_file: /path/to/fluentbit/parsers.conf

View your log data

If everything is configured correctly and data is being collected, you should see data in both of these places:

• New Relic Logs UI

• New Relic tools for running NRQL queries; for example, you can execute a query like this:

  SELECT * FROM Log

Troubleshoot log forwarding

No data appears when tailing a file

The log forwarding feature requires that the agent has permission to read the data sources. When running the infrastructure agent in privileged or non-privileged modes, make sure that the log files you want to forward (and any intermediary directory in its path) are readable by the user running nri-agent.

Example: checking file access under Linux

Let's check whether the file /var/log/restrictedLogs/logFile.log can be monitored by the nri-agent user. In Linux, you can do a quick check with the namei command:

sudo -u nri-agent namei -ml /var/log/restrictedLogs/logFile.log
f: /var/log/restrictedLogs/logFile.log
drwxr-xr-x root root /
drwxr-xr-x root root var
drwxrwxr-x root syslog log
drwxr--r-- root root restrictedLogs
logFile.log - No such file or directory

The above command failed since the file is not visible to the nri-agent user. By inspecting the previous output, we can detect that the restrictedLogs directory is missing the execution flag for "others".

We can fix this by executing:

sudo chmod 755 /var/log/restrictedLogs

And then check for file access again:

# sudo -u nri-agent namei -ml /var/log/restrictedLogs/logFile.log
f: /var/log/restrictedLogs/logFile.log
drwxr-xr-x root root /
drwxr-xr-x root root var
drwxrwxr-x root syslog log
drwxr-xr-x root root restrictedLogs
-rw-r----- vagrant vagrant logFile.log

The file is now visible to the nri-agent user. You must ensure that the file is also readable by the nri-agent user. To check this, you can use:

# sudo -u nri-agent head /var/log/restrictedLogs/logFile.log
head: cannot open '/var/log/restrictedLogs/logFile.log' for reading: Permission denied

In the above example, the file is missing the read rights for the "others" group (that is, users other than vagrant and the vagrant user group). A possible fix would be to grant read permissions to "others", but the application might change these permissions upon restart.

To avoid this, a better approach would be to add the nri-agent user to the vagrant user group.

If no data appears after you enable log management, follow standard troubleshooting procedures.

No data appears when capturing via a Syslog socket

The log forwarding feature requires that the agent has permission to read the data sources. When running the Infrastructure agent in privileged or non-privileged modes:

• If you're using Unix domain socket files, make sure that the nri-agent user can access these files (please refer to the previous section) and that they have read and write permissions (666) so that other users than nri-agent can write to them.
• If you're using IP sockets, ensure that the port that you are using is not a system reserved one (like port 80, for example).

If no data appears after you enable log management, follow standard log management troubleshooting procedures.

Send the agent's logs to New Relic

The infrastructure agent can be configured to send its own logs to New Relic. This can be useful for troubleshooting issues with log forwarding, the agent, or when contacting support.

To forward the infrastructure agent logs to New Relic:

1. Edit your newrelic-infra.yml file.

2. Enable agent logging in troubleshooting mode by adding verbose: 3

   On Windows and systems that don't use systemd or where journald is inaccessible, verbose:3 causes the agent to write the logs on the disk. Revert to verbose:0 to prevent this.

3. (Recommended): Enable agent logging in JSON format to log_format: json.

4. Restart the agent so that the new settings can be loaded.

The above configuration sets up the agent in troubleshooting mode, but the Log Forwarder (based on Fluent Bit) will continue in a non-verbose mode. Sometimes you can have issues with the Log Forwarder itself, for instance, problems when accessing a specific channel when shipping Windows log events or when accessing a particular log file. If that's the case, you can also enable the verbose mode for the Log Forwarder by setting verbose to a value different from 0 and by adding the following configuration option: trace: ["log.fw"].

Runtime error on Windows

One of the following error messages may appear when enabling log forwarding on Windows:

The code execution cannot proceed because VCRUNTIME140.dll was not found.

or:

error="exit status 3221225781" process=log-forwarder

This is caused by a missing DLL.

To solve the issue, install the Microsoft Visual C++ Redistributable: x64 or x86.

What's next?

Now that you've enabled Logs, here are some potential next steps:

• Explore your data using the Logs UI.
• Configure your agent to see contextual log data, such as distributed tracing, stack traces, application logs, and more.
• Query your data and create custom dashboards or alerts.

For more help