Forward your logs using the infrastructure agent

Copy (or rename) the elasticsearch-log.yml.example file to elasticsearch-log.yml to enable automatic Elasticsearch JSON formatted log parsing and forwarding to New Relic. No need to restart the agent.

Example:

sudo cp /etc/newrelic-infra/logging.d/elasticsearch-log.yml.example /etc/newrelic-infra/logging.d/elasticsearch-log.yml

Copy (or rename) the mysql-log.yml.example file to mysql-log.yml to enable automatic MySQL error log parsing and forwarding to New Relic. No need to restart the agent.

Example:

sudo cp /etc/newrelic-infra/logging.d/mysql-log.yml.example /etc/newrelic-infra/logging.d/mysql-log.yml

Copy (or rename) the nginx-log.yml.example file to nginx-log.yml to enable automatic NGINX access and error log parsing and forwarding to New Relic. No need to restart the agent.

Example:

sudo cp /etc/newrelic-infra/logging.d/nginx-log.yml.example /etc/newrelic-infra/logging.d/nginx-log.yml

Copy (or rename) the rabbitmq-log.yml.example file to rabbitmq-log.yml to enable automatic Rabbitmq error log parsing and forwarding to New Relic. No need to restart the agent.

Example:

sudo cp /etc/newrelic-infra/logging.d/rabbitmq-log.yml.example /etc/newrelic-infra/logging.d/rabbitmq-log.yml

Copy (or rename) the redis-log.yml.example file to redis-log.yml to enable automatic Redis error log parsing and forwarding to New Relic. No need to restart the agent.

Example:

sudo cp /etc/newrelic-infra/logging.d/redis-log.yml.example /etc/newrelic-infra/logging.d/redis-log.yml

Path to the log file or files. The agent tracks changes on the log files in a way similar to tail -f shell.

Example:

logs:
  - name: example-log
    file: /var/log/example.log # Path to a single log file
  - name: example-log-two
    file: /var/log/example-two.log # Path to another single log file

The file parameter can point to a specific log file or multiple files by using wildcards applied to names and extensions; for example, /logs/*.log. You can use wildcards in place of directories in a file path, which can be used to tail files located in different directories.

Example:

logs:
  - name: docker-logs
    file: /var/lib/docker/containers/*/*.log # Path to multiple folders and files

Use the systemd parameter to forward log messages that are collected by the journald daemon in Linux environments. This input type requires the agent to run in root mode.

Example:

logs:
  - name: systemd-example
    systemd: cupsd

Syslog data source.

Parameters:

• uri: Syslog socket. Format varies depending on the protocol:

  • TCP/UDP network sockets: [tcp/udp]://LISTEN_ADDRESS:PORT
  • Unix domain sockets: unix_[tcp/udp]:// + /socket/path

• parser: Syslog parser. Default is rfc3164. Use rfc5424 if your messages include fractional seconds. Note: rfc3164 currently does not work on SuSE.

• unix_permissions: default is 0644 for domain sockets; this limits entries to processes running as root. You can use 0666 to listen for non-root processes, at your own risk.

  When running the agent in privileged mode, ports and sockets must be available or owned by nri-agent, with 0666 file permissions, so that other processes can write logs to the sockets.

  logs:
    # TCP network socket
    - name: syslog-tcp-test
      syslog:
        uri: tcp://0.0.0.0:5140 # Use the tcp://LISTEN_ADDRESS:PORT format
        parser: rfc5424 # Default syslog parser is rfc3164
    # UDP network socket
    - name: syslog-udp-test
      syslog:
        uri: udp://0.0.0.0:6140 # Use the udp://LISTEN_ADDRESS:PORT format
      max_line_kb: 35
    # Unix TCP domain socket
    - name: syslog-unix-tcp-test
      syslog:
        uri: unix_tcp:///var/unix-tcp-socket-test
        unix_permissions: 0666 # Default is 0644. Change at your own risk
    # Unix UDP domain socket
    - name: syslog-unix-udp-test
      syslog:
        uri: unix_udp:///var/unix-udp-socket-test
        parser: rfc5424

  コピー

Logs retrieved over TCP connections.

Parameters:

• uri: TCP/IP socket to listen for incoming data. The URI format is tcp://LISTEN_ADDRESS:PORT

• format: format of the data. It can be json or none.

• separator: If format: none is used, you can define a separator string for splitting records (default: \n).

  logs:
    - name: tcp-simple-test
      tcp:
        uri: tcp://0.0.0.0:1234 # Use the tcp://LISTEN_ADDRESS:PORT format
        format: none # Raw text - this is default for 'tcp'
        separator: \t # String for separating raw text entries
      max_line_kb: 32
    - name: tcp-json-test
      tcp:
        uri: tcp://0.0.0.0:2345 # Use the tcp://LISTEN_ADDRESS:PORT format
        format: json

  コピー

Collect event from Windows log channels using new Windows Event Log API using the winevtlog Fluent Bit plugin.

Parameters:

• channel: name of the channel logs will be collected from.

• collect-eventids: a list of Windows Event IDs to be collected and forwarded to New Relic. Event ID ranges are supported.

• exclude-eventids: a list of Windows Event IDs to be excluded from collection. Event ID ranges are supported.

  All events are collected from the specified channel by default. Configure the collect-eventids and exclude-eventids sections to avoid sending unwanted logs to your New Relic account.

  Add event IDs or ranges to collect-eventids or exclude-eventids to forward or drop specific events. exclude-eventids takes precedence over collect-eventids if the same event ID is present in both sections.

  Example:

  logs:
   # Winevtlog log ingestion with eventId filters.
    - name: windows-security
      winevtlog:
        channel: Security
        collect-eventids:
          - 4624
          - 4265
          - 4700-4800
        exclude-eventids:
          - 4735
  
   # entries for the application, system, powershell, and SCOM channels
    - name: windows-application
      winevtlog:
        channel: Application
    - name: windows-system
      winevtlog:
        channel: System
    - name: windows-pshell
      winevtlog:
        channel: Windows Powershell
    - name: scom
      winevtlog:
        channel: Operations Manager
  
   # Entry for Windows Defender Logs
    - name: windows-defender
      winevtlog:
        channel: Microsoft-Windows-Windows Defender/Operational
  
   # Entry for Windows Clustering Logs
    - name: windows-clustering
      winevtlog:
        channel: Microsoft-Windows-FailoverClustering/Operational
  
   # Entry for IIS logs with logtype attribute for automatic parsing
    - name: iis-log
      file: C:\inetpub\logs\LogFiles\w3svc.log
      attributes:
        logtype: iis_w3c

  コピー

Collect events from Windows log channels.

Parameters:

• channel: name of the channel logs will be collected from. It doesn't work for custom channels.

• collect-eventids: a list of Windows Event IDs to be collected and forwarded to New Relic. Event ID ranges are supported.

• exclude-eventids: a list of Windows Event IDs to be excluded from collection. Event ID ranges are supported.

  All events are collected from the specified channel by default. Configure the collect-eventids and exclude-eventids sections to avoid sending unwanted logs to your New Relic account.

  Add event IDs or ranges to collect-eventids or exclude-eventids to forward or drop specific events. exclude-eventids takes precedence over collect-eventids if the same event ID is present in both sections.

  Example:

  logs:
   # Winlog log ingestion with eventId filters.
    - name: windows-security
      winlog:
        channel: Security
        collect-eventids:
          - 4624
          - 4265
          - 4700-4800
        exclude-eventids:
          - 4735
  
   # entries for the application, system, powershell, and SCOM channels
    - name: windows-application
      winlog:
        channel: Application
    - name: windows-system
      winlog:
        channel: System
    - name: windows-pshell
      winlog:
        channel: Windows Powershell
    - name: scom
      winlog:
        channel: Operations Manager
  
   # Entry for Windows Defender Logs
    - name: windows-defender
      winlog:
        channel: Microsoft-Windows-Windows Defender/Operational
  
   # Entry for Windows Clustering Logs
    - name: windows-clustering
      winlog:
        channel: Microsoft-Windows-FailoverClustering/Operational
  
   # Entry for IIS logs with logtype attribute for automatic parsing
    - name: iis-log
      file: C:\inetpub\logs\LogFiles\w3svc.log
      attributes:
        logtype: iis_w3c

  コピー

List of custom attributes specified as key-value pairs that can be used to send additional data with the logs which you can then query. The attributes configuration parameter can be used with any log source.

One common use of the attributes configuration parameter is to specify the logtype attribute. This attribute allows leveraging one of the built-in parsing rules supported by New Relic's log management capabilities.

Example:

logs:
 - name: example-file-attributes
   file: /var/log/example.log
   attributes:
     logtype: nginx
     region: example-us-02
     team: A-team
 - name: example-tcp-attributes
   tcp:
     uri: tcp://0.0.0.0:2345
     format: json
   attributes:
     logtype: nginx
     region: example-us-02
     team: B-team

The infrastructure agent automatically inserts log attributes for your convenience. Some of them are inserted for any log record, while others depend on the configuration parameters you used while setting up the log forwarder.

Regular expression for filtering records. Only supported for the tail, systemd, syslog, and tcp (only with format none) sources.

This field works in a way similar to grep -E in Unix systems. For example, for a given file being captured, you can filter for records containing either WARN or ERROR using:

- name: only-records-with-warn-and-error
  file: /var/log/logFile.log
  pattern: WARN|ERROR

No filtering is applied by default.

Maximum size of log entries/lines in KB. If log entries exceed the limit, they are skipped. Default is 128.

External Fluent Bit configuration and parser files. If defined, they are merged with the existing configuration and parser files generated by the infrastructure agent.

The infrastructure agent processes the configuration files located in the logging.d directory and will generate a run-time Fluent Bit configuration file that contains the appropriate [INPUT], [FILTER] and [OUTPUT] sections. Optionally, it will also declare an @INCLUDE in case you provided an external Fluent Bit configuration file via the fluentbit option.

The runtime file does not define a [SERVICE] section, leaving all default Fluent Bit configuration values. You can still override Fluent Bit's default settings by defining your own [SERVICE] section in your external Fluent Bit configuration file and include it via the fluentbit option.

Parameters:

config_file: path to an existing Fluent Bit configuration file. Note that any overlapping source results in duplicate messages in New Relic's Logs UI.

parsers_file: path to an existing Fluent Bit parsers file. The following parser names are reserved: rfc3164, rfc3164-local and rfc5424.

If no data appears after you enable our log management capabilities, follow our standard log troubleshooting procedures.

The log forwarding feature requires the agent to have permission to read the data sources. When running the infrastructure agent in privileged or non-privileged modes, make sure that the log files you want to forward (and any intermediary directory in its path) are readable by the user running nri-agent.

Example: Check file access under Linux

Let's check whether the file /var/log/restrictedLogs/logFile.log can be monitored by the nri-agent user. In Linux, you can do a quick check with the namei command:

sudo -u nri-agent namei -ml /var/log/restrictedLogs/logFile.log
f: /var/log/restrictedLogs/logFile.log
drwxr-xr-x root root /
drwxr-xr-x root root var
drwxrwxr-x root syslog log
drwxr--r-- root root restrictedLogs
logFile.log - No such file or directory

This command failed because the file is not visible to the nri-agent user. By inspecting the previous output, we can detect that the restrictedLogs directory is missing the execution flag for others.

To fix this, execute:

sudo chmod 755 /var/log/restrictedLogs

And then check for file access again:

# sudo -u nri-agent namei -ml /var/log/restrictedLogs/logFile.log
f: /var/log/restrictedLogs/logFile.log
drwxr-xr-x root root /
drwxr-xr-x root root var
drwxrwxr-x root syslog log
drwxr-xr-x root root restrictedLogs
-rw-r----- vagrant vagrant logFile.log

The file is now visible to the nri-agent user. You must ensure that the file is also readable by the nri-agent user. To check this, use:

# sudo -u nri-agent head /var/log/restrictedLogs/logFile.log
head: cannot open '/var/log/restrictedLogs/logFile.log' for reading: Permission denied

In this example, the file is missing the read rights for the others group (users other than vagrant and the vagrant user group). You could fix this by granting read permissions to others, but the application could change these permissions upon restart.

To avoid this, a better approach is to add the nri-agent user to the vagrant user group.

The log forwarding feature requires that the agent has permission to read the data sources. When running the infrastructure agent in privileged or non-privileged modes:

• If you're using Unix domain socket files, make sure that the nri-agent user can access these files (please refer to the previous section) and that they have read and write permissions (666) so that other users than nri-agent can write to them.

• If you're using IP sockets, ensure that the port that you are using is not a system reserved one (like port 80, for example).

  If no data appears after you enable log management, follow standard log management troubleshooting procedures.

As explained in the infrastructure agent configuration guidelines, the proxy parameter must use either HTTP or HTTPS and be in the form https://user:password@hostname:port. The agent can parse the parameter without the HTTP or HTTPS, but the log forwarder cannot. You will see an error like the following in the agent verbose logs:

[ERROR] building HTTP transport: parse \"hostname:port\":
first path segment in URL cannot contain colon

To solve this problem, check your newrelic-infra.yml file, and ensure the proxy parameter adheres to this form.

If you're using caBundleFile or caBundleDir in order to specify any certificate, we recommend to follow the below rules for each OS:

Linux For HTTP proxies you don't need to setup any certificates. The plugin loads the system certificates and New Relic sends logs into the logging endpoint. However, you can specify the proxy self-signed certificate (PEM file) using either the caBundleFile or caBundleDir parameters.

Windows

• For HTTP proxies you don't need to setup any certificates. The plugin loads the system certificates.

• For HTTPS, you can configure it in one of the following ways:

• Import the proxy certificate to the system pool (Recommended) Import the proxy self-signed certificate (PEM file) by using the MMC tool. Refer to this link, and in Step 2 ensure to import it in your Trusted Root Certification Authorities, instead of in the Intermediate Certification Authorities.

• Using the caBundleFile and caBundleDir parameters On Windows, we cannot load both the certificates from the system certificate pool and the ones specified with the caBundleFile caBundleDir parameters. So, if you are using caBundleFile or caBundleDir, ensure that the following certificates are placed in the same PEM file (when using caBundleFile) or in the same directory (when using caBundleDir):

• The Proxy certificate (because it's an HTTPS proxy).

• The Logging Endpoint certificate (eg. https://log-api.newrelic.com/log/v1).

• The Infrastructure Agent certificate (eg. https://infra-api.newrelic.com).

  You can check the certificates by running:

  # openssl s_client -connect log-api.newrelic.com:443 -servername log-api.newrelic.com

  コピー

You can configure the infrastructure agent to send its own logs to New Relic. This is useful for troubleshooting issues with log forwarding, the agent, or when contacting support.

To forward the infrastructure agent logs to New Relic:

1. Edit your newrelic-infra.yml file.

2. Enable agent logging in troubleshooting mode by adding verbose: 3.

3. On Windows and systems that don't use systemd or where journald is inaccessible, verbose:3 causes the agent to write the logs on the disk. Revert to verbose:0 to prevent this.

4. (Recommended): Enable agent logging in JSON format to log_format: json.

5. Restart the agent to load the new settings.

   This configuration sets up the agent in troubleshooting mode, but the log forwarder (based on Fluent Bit) will continue in a non-verbose mode.

   Sometimes you can have issues with the log forwarder itself. For example, there may be problems accessing a specific channel when shipping Windows log events or when accessing a particular log file. In these situations, you can also enable the verbose mode for the log forwarder:

6. Set verbose to a value other than 0.

7. Add the configuration option: trace: ["log.fw"].

   注意

   Check whether you are using the [fluentbit] option. When setting verbose: 3 and trace: ["log.fw"], ensure that you don't define any [OUTPUT] section pointing to stdout in an external Fluent Bit configuration file,

For Linux versions prior to 2016, you may need to update the OpenSSL library to 1.1.0 (or higher). To check if you have this problem:

1. See if infra-agent has started Fluent Bit by running:

   ps -aux | grep td-agent-bit

   コピー

2. If it isn't running go to /var/db/newrelic-infra/newrelic-integrations/logging and run:

   ./fluent-bit -i systemd -o stdout

   コピー

3. If you get the following error, update OpenSSL to 1.1.0 or higher:

   error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory

   コピー

One of the following error messages may appear when enabling log forwarding on Windows:

The code execution cannot proceed because VCRUNTIME140.dll was not found.

OR

error="exit status 3221225781" process=log-forwarder

This is caused by a missing DLL.

To solve the issue, install the Microsoft Visual C++ Redistributable as applicable:

• x64
• x86

It's common to face either one of the following error messages when attempting to tail a large amount of files:

• Too many open files

• The user limit on the total number of inotify watches was reached or the kernel failed to allocate a needed resource

  The operating system defines a maximum amount of allocatable file descriptors (typically 1024 by default), and a maximum amount of allocatable inotify watches (typically 8192 by default). Any process attempting to go above these limits will fail, returning one of the errors above.

  The underlying technology we use to forward logs, Fluent Bit, opens one file descriptor and sets an inotify watch for each file you configure to be forwarded. Moreover, at the time of writing this section, Fluent Bit uses an extra set of 32 file descriptors for its normal operation, with another extra file descriptor when it shuts down. Therefore, to capture a large amount of files you need to ensure that both the file descriptor and inotify watch limits are slightly greater than the amount of log files you wish to tail.

  The following instructions summarize how to increase these limits if you want to tail 10,000 log files. Also, it assumes the infrastructure agent is installed in root running mode, and therefore must be run using the root user.

1. Check which is the current hard limit for the amount of file descriptors per process. Typically, this limit should be quite high and should not need to be modified.

   ulimit -Hn

   コピー

2. Add the following line to /etc/security/limits.conf. We specified a limit of 10100 here instead of just 10000 to allow Fluent Bit to allocate the extra file descriptors it may need to work.

   root    soft    nofile  10100  # replace root by nri-agent for non-root (privileged and unprivileged) installations

   コピー

3. Add the following line to /etc/pam.d/common-session so that the previous limit is applied upon restart:

   session required pam_limits.so

   コピー

4. Add the following line to /etc/sysctl.conf to increase the amount of allowed inotify watchers per user. We specified a limit of 18192 here instead of just 10000 so that the root user will still have 8192 available inotify watches (the default value).

   fs.inotify.max_user_watches=18192

   コピー

5. Restart your system.

6. Ensure that the new limits have been enforced by running:

   ulimit -Sn                                    # Should return 10100
   cat /proc/sys/fs/inotify/max_user_watches     # Should return 18192

   コピー

   Learn more on how to increase open file limits, or on how to increase the inotify watches.

Prior to version 1.19.0 (or version 1.20.3 for SLES 12.5), the Linux infrastructure agent came bundled with a Fluent Bit binary. Starting in this version, Fluent Bit is now included as as a separate recommended package dependency.

This means that you can install, update, or downgrade Fluent Bit separately from the agent. For your convenience, we have included several Fluent Bit packages in the same repository where the infrastructure lives, so you don't need to install any additional repositories to upgrade Fluent Bit.

Note that the agent automatically installs Fluent Bit when you first install it, using the latest available version. After the first installation, you can upgrade Fluent Bit as you would normally do with any Linux package.

You can list the available Fluent Bit versions by running:

RPM:

sudo yum check-update
yum list td-agent-bit --showduplicates

DEB:

sudo apt update
apt-cache showpkg td-agent-bit

To upgrade to a particular Fluent Bit version, just run (in the following examples, version 1.8.12 is used):

RPM:

# Remove command only required when downgrading to a previous version
sudo yum remove td-agent-bit
sudo yum install td-agent-bit-1.8.12-1

DEB:

sudo apt install td-agent-bit=1.8.12