Python agent logging

Problem

You can configure the Python agent to log output. This configuration allows the agent to track whether it is connecting to New Relic properly and if any errors occur. This information will be useful to New Relic Support if you experience problems.

Solution

Detailed debug logging can help troubleshoot your standard Python agent installation.

Important

Some partners use different procedures:

To enable debug logging:

Open your newrelic.ini, usually located within your app hierarchy.
Uncomment #log_file = /tmp/newrelic-python-agent.log. Ensure you have write permissions to the log location, changing the path and file name if necessary. If there is no suitable file location, you can set log_file to stderr.
Change log_level to debug (from info).
Caution
Logging at debug can generate a lot of data very quickly. Monitor the size of your log file closely, changing log_level back to info as you finish troubleshooting.
Save and close the file. Restart your app for the settings to take affect.
Generate a few minutes of traffic to your app.
If sending your log file to New Relic Support, attach your newrelic.ini to your support ticket, and tell Support your time zone.

Important

When troubleshooting your New Relic Python agent, ensure it has been configured to generate debug level log files, and monitor the size of your log file closely. Using log_level = debug generates a lot of data very quickly. After reproducing your problem, return the log file to log_level = info.

Log to a file

The agent uses the Python logging module to output log messages. Output from the agent should therefore be factored into the overall logging strategy for your application.

If you are not using the logging module, the agent provides a simplified way to enable a log file for the Python agent. To use this, set the log_file and log_level options in the agent configuration file.

Example:

log_file = /tmp/newrelic-python-agent.log
log_level = info

The path provided for log_file should be writable to the user that your application runs as.

Tip

If using Apache/mod_wsgi the Apache user has restricted access to the filesystem. Create a special directory into which the log file can be placed which is writable to the Apache user. Using an absolute path, and not a relative path is recommended.

The log level used can be one of error, warning, info or debug. Under normal circumstances, use info. More verbose debug options are used for debug. Do not use these verbose debug options for an extended period of time. They can generate excessive output and the log file handler from the logging module is the standard file handler and does not perform any log file rotation.

Log to standard error

For some hosting providers, it may not be possible to use a distinct log file for the agent. Configure the logging module to log to standard error output. This output is captured in the normal error log file for your hosting mechanism.

To do this within the agent configuration file:

log_file = stderr
log_level = info

The value stdout may also be used instead of stderr.

Log all data (audit log)

If you need to record and view information about all data being transmitted between the monitored process and the New Relic collector, you can enable audit logging for short periods of time. This is useful, for example, with debugging or auditing, when you need detailed information about what exactly is being transmitted.

Troubleshoot log module conflicts

If no logging is captured, there may be a conflict with the way in which the Python logging module is being initialized and/or configured. Problems may arise using the following functions:

One problem which can arise is when the monitored application is using the logging.config.fileConfig() function to initialize the Python logging module. If the Python logging module is initialized with this call, this will by default disable the agent logging if the agent had already been initialized prior to the call.

If using Python 2.6 or higher and the call to logging.config.fileConfig() is under your control, one option is to modify the call to pass the argument disable_existing_loggers with the value of False. This will prevent the existing loggers being disabled and as such the agent loggers will be left alone.

logging.config.fileConfig('logging.cfg', disable_existing_loggers=False)

If you are unable to modify the call, or it would not be appropriate to do so, the alternative is to reinstate the logger installed by the agent. To do this, a logger section should be added to the logging configuration file being passed to logging.config.fileConfig().

[loggers]
keys = root,newrelic

[logger_newrelic]
qualname = newrelic
level = INFO
handlers =

In this case, with the handlers setting being left unset, how messages will be logged will be dictated by the handler associated with the root logger. You can alternatively refer directly to the appropriate handler which should be used to log the messages.

A similar issue as above can also arise when the monitored application is using logging.config.dictConfig().

When using a dictionary to configure the logging module, there is no function argument disable_existing_loggers. Instead, specify it as a key within the dictionary.

LOGGING = {
  'disable_existing_loggers': False,
  ...
}

Similar to before, to avoid inherited existing loggers, an explicit logger definition should be added to the dictionary used to configure the logging module.

LOGGING = {
  ...,

  'loggers': {
    'newrelic': {
      'level': 'INFO',
    },
    ...
  }
}

For more information, see the Python logging module documentation.

Refer to the documentation of any web framework or application being used. It may provide a specialized mechanism for configuring the Python logging module. For example, when using Django, the dictionary approach to setting up the Python logging module is supported automatically, with the definitions being set in the LOGGING attribute within the Django settings module.

For more information, see the Django logging configuration documentation.

Rotate agent log file

When using the log_file option in the agent configuration, the standard file handler from the logging module is used. This does not do any log file rotation. Log file rotation is not done automatically as we will not know what sort of rotating log file handler you may want to use, plus the standard rotating log file handlers supplied with Python are not necessarily safe for a multi process configuration. As such, it may be necessary to download and use one of the third party rotating log file handlers or use a dedicated logging system.

If your application runs in a single process you can safely use either of the RotatingFileHandler or TimedRotatingFileHandler handlers supplied with the logging module. To use this only for the output from the Python agent, include at the start of your WSGI script file or module, but before the import of the newrelic module, the following:

_LOG_FORMAT = (
    "%(asctime)s (%(process)d/%(threadName)s)" " %(name)s %(levelname)s - %(message)s"
)

_logger = logging.getLogger("newrelic")
_handler = logging.handlers.TimedRotatingFileHandler(
    "agent.log", when="midnight", backupCount=7
)
_formatter = logging.Formatter(_LOG_FORMAT)
_handler.setFormatter(_formatter)
_logger.addHandler(_handler)
_logger.setLevel(logging.INFO)

This code accesses the root newrelic logger instance, attaches the rotating log file handler to it, and sets the log level for messages to be sent to this log file. We also show how to set up the log message format, but that is optional.

If the logging module as a whole is initialized, any log output will propagate up and be logged by any handler associated with the root logger, including possibly to standard error. To avoid duplicates, configure the log handler for the root logger.

The above code could also be simplified by using the logging.fileConfig() function and a configuration file. For further details of using a configuration file see Python logging module documentation on Logging configuration.

Rotate log in multi-process configuration

The rotating log file handlers provided in the standard logging module are not entirely safe to use in a multi-process server configuration. Problems that can occur are intermingling of messages from multiple processes and attempts by multiple processes to perform log file rotation at the same time.

For a more robust log file rotation mechanism, use a third party log handler in conjunction with the Python logging module.

One such implementation available on PyPi is:

Concurrent Log Handler

For a more complex solution you may also consider a logging service such as:

Python Logging Server

Also consult the list of other handlers provided as standard in the Python logging module as those for sending to a socket or posting to a HTTP URL may also be reasonable solutions in some circumstances.

Problem

Solution

Important

Caution

Important

Log to a file

Tip

Log to standard error

Log all data (audit log)

Troubleshoot log module conflicts

`logging.config.fileConfig()` function

`disable_existing_loggers` function argument

Rotate agent log file

Rotate log in multi-process configuration

Python agent logging

Problem.css-21sua1{background:none;border:none;width:0;padding:0;}

Solution

Important

Caution

Important

Log to a file

Tip

Log to standard error

Log all data (audit log)

Troubleshoot log module conflicts

disable_existing_loggers function argument

Rotate agent log file

Rotate log in multi-process configuration

Problem

`disable_existing_loggers` function argument