Infrastructure agent logging behavior

Infrastructure gathers its own data as well as integrations' logs and consolidates them in a single source. By default, logs are shown both in standard-output and added to a log file. To disable logs in standard output, see the agent's config options.

Logging security levels

Infrastructure uses a subset of the standard Syslog severity levels:

  • ERROR: Error conditions met
  • WARN: Warning conditions met
  • INFO: Informational messages
  • DEBUG: Contains debug-level messages (useful when troubleshooting)

DEBUG level is only shown when the verbose mode is enabled.

Log formatting

Starting on agent v1.4.9, log messages are inlined with context values, which allows for better grouping and filtering:

containerized agent found in container
     containerID: VALUE

Formats

By default, Infrastructure logs are formatted as text:

  • In foreground mode, log output is colored, without a timestamp:
    DEBUG Sending deltas divided in blocks component=PatchSender mentityKey=ohaimaci mnumberOfBlocks=1
  • In background mode, logs are a timestamped black-and-white output, used when running as a service or dumping logs to a file:
     time="2019-07-12T09:54:15+02:00" level=info msg="Agent service manager shutdown completed successfully." component=AgentService service=newrelic-infra

Alternatively logs can be formatted as a JSON file:

{"context":{},"level":"info","msg":"upstart_interval_sec: 0","timestamp":"2019-07-11T18:24:03+02:00"} 
{"context":{},"level":"info","msg":"plugin_dir: ","timestamp":"2019-07-11T18:24:03+02:00"}

To change the log format see the agent configuration settings.

Log rotation

The Infrastructure agent does not provide any native log rotation/compression mechanism. Instead, we encourage you to use consolidated log rotation tools, such as the Linux logrotate tool, which is usually installed by default in most Linux distributions.

Logrotate can be configured as an entry in /etc/logrotate.conf, or as a file in the /etc/logrotate.d directory.

Logrotate config file sample

A sample logrotate config file looks like this:

/var/log/newrelic-infra/newrelic-infra.log { 
     copytruncate 
     compress 
     daily 
     dateext 
     maxage 7 
}

Where:

  • /var/log/newrelic-infra/newrelic-infra.log: The Infrastructure agent log file. It must match the log_file configuration parameter in the /etc/newrelic-infra.yml file.
  • copytruncate: Indicates that the log file is truncated but not deleted when it is rotated. This configuration option is mandatory, otherwise the log file will be deleted and won’t be recreated.
  • compress: Compresses (usually in Gzip format) the rotated log files.
  • daily: The agent rotates logs daily.
  • dateext: Appends a date (by default, in the format YYYYMMDD) to the rotated log file (e.g. newrelic-infra.log-20190708.gz)
  • maxage 7: Makes logrotate remove rotated files after 7 days.

See the Logrotate docs for a complete description of the logrotate configuration options.

Since logrotate is usually executed automatically as a cron job, verify that there is a logrotate entry in cron (for example/etc/cron.daily/logrotate) similar to:

#!/bin/sh 

/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf EXITVALUE=$? 
if [ $EXITVALUE != 0 ]; then 
     /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]" 
fi 
exit 0

Logging before Infrastructure agent v1.4.9

For agent versions prior to v1.4.9:

Agent v1.4.9 and higher Before v1.4.9
Foreground mode logged. The agent couldn't log some entries in foreground mode because the logging service wasn't able to write data until the agent was completely configured.​
Logs in text and JSON formats. Logs in text only.
Logs displayed as inline text. Logs displayed as static literals in a single, decontextualized line.

For more help

Recommendations for learning more: