Python agent and uWSGI

The New Relic Python agent can be used with uWSGI version 1.2.6 or higher. (Versions lower than 1.2.6 have a bug with the way that uWSGI implements the WSGI specification, which can cause significant inaccuracies in data being reported by the Python agent.)

Mandatory options

When using uWSGI you will need to supply certain specific command line options to the uwsgi executable. The additional options you must use are:

Option Purpose
--enable-threads By default uWSGI does not enable threading support within the Python interpreter core. This means it is not possible to create background threads from Python code. As the Python agent relies on being able to create background threads, this option is required. This option will be automatically applied if uWSGI is configured for multiple threads using the --threads option.
--single-interpreter By default uWSGI will execute Python code within a sub interpreter of the process rather than the main Python interpreter created when Python is first initialized. This is done to allow multiple separate Python web applications to be run within the one process but to be sufficiently separated so as to not interfere with each other. Older versions of uWSGI can fail however when using sub interpreters with threading enabled. It is therefore safest to use this option and restrict yourself to a single web application per process. Running a single web application in each process with uWSGI is the normal use case so it would be unlikely that this restriction would be an issue.
--wsgi-env-behavior=holy

In uWSGI versions 2.0.x or lower, by default uWSGI sets the WSGI environment behavior value to cheat. This setting preallocates the WSGI application arguments on uWSGI startup and clears them after each request. However, in certain environments, this setting can cause a SystemError application crash. Users of uWSGI versions 2.0.x or lower are recommended to set the WSGI environment behavior to holy, which corrects the behavior and does not cause a SystemError crash.

Wrapper script

If you are starting your WSGI application under uWSGI using a command of the form:

uwsgi --socket /tmp/uwsgi.sock wsgi.py

all you need to instead do is run:

NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program uwsgi --socket /tmp/uwsgi.sock --single-interpreter --enable-threads --wsgi-env-behavior=holy​ wsgi.py

When doing this, instead of defining the NEW_RELIC_CONFIG_FILE environment variable on the same line as executing the command, it can be separately exported and set in the process environment before running the command.

NEW_RELIC_CONFIG_FILE=newrelic.ini
export NEW_RELIC_CONFIG_FILE

newrelic-admin run-program uwsgi --socket /tmp/uwsgi.sock --single-interpreter --enable-threads --wsgi-env-behavior=holy​ wsgi.py

The path newrelic.ini should be replaced with the appropriate absolute or relative path to the location of your actual Python agent configuration file.

If you are using a framework for which the Python agent is not automatically wrapping the WSGI application entry point, you can still use newrelic-admin but you will at least need to manually wrap the WSGI application entry point.

Agent initialization

When using the newrelic-admin wrapper script with uWSGI, the agent will be automatically initialized when the web application starts. You do not need to make any code changes to your web application.

If you are not able to, or do not wish to use the newrelic-admin wrapper script, you can still manually add in calls to the agent API to initialize it when your web application starts.

As per instructions for integration with your Python application, this should be done as the very first step by your web application, after any module import paths are setup, but before any imports of separate application modules or any modules for frameworks being used.

Configuration styles

In addition to configuring uWSGI using command line options, it also supports alternate configuration mechanisms including environment variables, XML, ini, json or yaml files, ldap or a database.

When using such a mechanism you will still need to include the mandatory additional options listed above. These would instead be included in the appropriate configuration file or database.

For example, if using an ini configuration file you would have:

[uwsgi]
socket = /tmp/uwsgi.sock
enable-threads = true
single-interpreter = true
wsgi-env-behavior = holy
file = wsgi.py

and run:

NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program uwsgi --ini uwsgi.ini

Application mounting

When specifying the WSGI application to be served up by uWSGI, it is possible to give a path to the Python WSGI script file or a direct reference to a module and the contained application. You might for example use the latter if using Django with the configuration:

[uwsgi]
socket = /tmp/uwsgi.sock
enable-threads = true
single-interpreter = true
wsgi-env-behavior = holy​
chdir = /var/www/myapp
pythonpath = /var/www
env = DJANGO_SETTINGS_MODULE=myapp.settings
module = django.core.handlers.wsgi:WSGIHandler()

Using a direct reference in this way to the WSGI application and the module it is contained in will work, but only so long as the framework being used is one which the agent automatically instruments and wraps the WSGI application entry point. If you are using a custom WSGI framework or an unsupported framework you would need to manually wrap the WSGI application entry in your code.

Alternatively, you can use the ability of uWSGI to evaluate a small snippet of code in the configuration in order to construct the WSGI application entry point.

[uwsgi]
socket = /tmp/uwsgi.sock
enable-threads = true
single-interpreter = true
wsgi-env-behavior = holy
eval = import newrelic.agent, wsgi; application = newrelic.agent.wsgi_application()(wsgi.application)

Master process

As part of uWSGI's process management features, one can enable a master process to better manage worker processes. When this is enabled the WSGI script file or module will be preloaded into the parent master process. Worker processes will then be forked from this master process.

This can cause problems if the WSGI script or module when loaded creates a background thread which is supposed to run in each worker process, as that background thread will be killed when the worker process are forked.

The Python agent uses a background thread to report data back to our data collector on a regular interval. Under normal circumstances this will only be created upon the first web request being received. As that would normally be in the worker processes, use of a master process should not cause a problem.

If however you have added instrumentation to track calls to certain functions as background tasks and those functions are called on loading the WSGI script file or module, the agent background thread will be started in the master parent process. With the background thread then being subsequently killed when worker processes are forked, no data will be reported for the actual web application.

If using a master process and you are seeing no data being reported for the web application running in the worker processes, you should also use lazy loading mode:

[uwsgi]
socket = /tmp/uwsgi.sock
enable-threads = true
single-interpreter = true
wsgi-env-behavior = holy
master = true
lazy-apps = true
file = wsgi.py

or use uwsgi.post_fork_hook() to register any startup functions which may have a side effect of starting the agent background thread, to ensure they are called after any worker processes are forked.

Note that if using emperor mode, the option for lazy loading must be in the configuration file for the individual vassals. It cannot be specified at the same point where emperor mode is enabled as it will not be inherited by the vassals.

For more help

Join the discussion about Python in the New Relic Online Technical Community! The Technical Community is a public platform to discuss and troubleshoot your New Relic toolset.

If you need additional help, get support at support.newrelic.com.