Python custom instrumentation via config file

You can extend New Relic for Python's monitoring to trace time spent in additional functions or methods of classes by modifying your configuration file. This form of custom instrumentation is simpler to implement than API calls, and does not require you to modify your code. However, for more complex needs, you may need to implement Python instrumentation by API.

Listing functions in the configuration file

To extend instrumentation to designated functions and class methods, add them to the transaction_tracer.function_trace setting within the newrelic section of the agent configuration file. The identifier for a function should have the form module:function and that of a class method module:class.function.

Using dumbdbm

In this example, you use the Python dumbdbm module and want to instrument the time it took to open a database, and then to write that database back to a file. In this case you would use:

transaction_tracer.function_trace = dumbdbm:open
                                               dumbdbm:_Database._commit

To list more than one item, use either of these methods:

  • One line: Separate each item with a space.
  • Multiple lines: Follow the ini file convention to indent the subsequent lines.

When data is reported for the instrumented function, the metric will have a name in the form Function/module:function or Function/module:class.function. In the performance breakdown for a transaction, the category will show as Function and the segment will be module:function or module:class.function. For slow transaction traces, only the segment name appears. Note that where a function is actually returning a generator, only the time spent in returning the generator will be recorded and not the consumption of the values from the generator.

Restrictions on wrapping extension APIs

When wrapping functions by listing them in the agent configuration file, you cannot designate class methods this way when the method is a member of a class which is implemented in C code by a C extension module. This is because it is not possible to modify the method table of a type implemented using the Python C API.

Overriding the reported function name

When using the transaction_tracer.function_trace setting in the agent configuration file, the name of the function will be used in the metric name, with it being classified as a Function. If it is necessary to override what function name may be used in the metric name or classify it differently, then an alternate means of defining the function trace in the configuration file can be used. The equivalent for:

transaction_tracer.function_trace = dumbdbm:open
                                               dumbdbm:_Database._commit

would be to create two new configuration sections in the agent configuration file, one for each function to be traced:

[function-trace:dumbdbm-1]
enabled = true
function = dumbdbm:open
name = dumbdbm:open
group = Function

[function-trace:dumbdbm-2]
enabled = true
function = dumbdbm:_Database._commit
name = dumbdbm:_Database._commit
group = Function

The ini file section name should start with function-trace:. The name component that follows that prefix can be anything but should be unique across all function trace sections in the configuration file. The group and name settings can be overridden as necessary to arrive at the desired metric name. The enabled setting defaults to false and needs to be set to true to enable the function trace.

Applying function decorators to code

Using the agent configuration file to define additional functions to be traced is convenient for third-party code, as it does not require modifying the original code. However, for your own code, it may be inconvenient to maintain a list within a separate configuration file. In this case it may be easier to denote which functions or class methods to trace in the code itself by using a function decorator.

The decorator name is function_trace and is in the newrelic.agent module. For example:

import newrelic.agent

class _Database(UserDict.DictMixin):

    ...

    @newrelic.agent.function_trace()
    def _commit(self):
        ...

@newrelic.agent.function_trace()
def open(file, flag=None, mode=0666):
...

You can also use the decorator in conjunction with existing decorators, including those for static and class methods. When nesting multiple decorators, have the function_trace decorator as the outermost decorator.

Note: For any decorators that are being wrapped, use functools.wraps() from the Python standard library in their implementation. (Or, otherwise ensure that the decorator propagates the correct name attributes of the innermost wrapped object required to allow correct name introspection.) If this is not done, then when the metric is reported, the name of the nested decorator function (not the innermost wrapped function) will be used.

Wrapping functions dynamically

If you know in advance where the specific functions to be traced are, you can use function decorators. However, if you do not know all the functions that need to be traced, because for example they are being looked up dynamically as part of a routing system, then it will be necessary to wrap the function on the fly at the time of registration or at the time of calling.

For example, URL routing with Werkzeug yields a name that is used first to dynamically look up a method of a class using getattr() and the result then invoked:

def dispatch_request(self, request):
    adapter = self.url_map.bind_to_environ(request.environ)
    try:
        endpoint, values = adapter.match()
        return getattr(self, 'on_' + endpoint)(request, **values)
    except HTTPException as e:
        return e

If you want to trace the endpoint function, you can rewrite this as:

import newrelic.agent

def dispatch_request(self, request):
    adapter = self.url_map.bind_to_environ(request.environ)
    try:
        endpoint, values = adapter.match()
        function = getattr(self, 'on_' + endpoint)
        function = newrelic.agent.FunctionTraceWrapper(function)
        return function(request, **values)
    except HTTPException as e:
        return e

If you want to name the metric after the endpoint name (rather than naming the metric based on the identifier for the function being called), you can supply the name to use plus an alternate metric path prefix when the FunctionTraceWrapper object is created.

import newrelic.agent

def dispatch_request(self, request):
    adapter = self.url_map.bind_to_environ(request.environ)
    try:
        endpoint, values = adapter.match()
        function = getattr(self, 'on_' + endpoint)
        function = newrelic.agent.FunctionTraceWrapper(
                function, name=endpoint, group='Python/EndPoint')
        return function(request, **values)
    except HTTPException as e:
        return e

For this example, if the endpoint were called help, the final metric would be:

Python/EndPoint/help

In the performance breakdown for a transaction, the category would be Python and the segment name EndPoint/help. That segment name would also appear in slow transaction traces as the node name.

Notes:

  • When overriding the use of the function name in the metric, choose names within a finite set that are relatively small in length.
  • Avoid names that are unbounded in value. Otherwise they cannot be grouped in a meaningful way to display in the UI.
  • Do not use a large number of unique names. This can result in excessive memory being used, and the agent will forcibly normalize your names to limit the number of unique names.
  • Always use the Python/ prefix to avoid clashing with names that internally have a predefined meaning.

Monitoring code with context managers

As explained above, FunctionTraceWrapper is useful to dynamically wrap a function where it needs to be passed off to another function in order to invoke it. However, if you want to wrap a function at the place in the code where the function actually executes, a better method than FunctionTraceWrapper is to use a context manager. Also, when using the context manager to name a metric, you must always supply the name and the metric path prefix.

import newrelic.agent

def dispatch_request(self, request):
    adapter = self.url_map.bind_to_environ(request.environ)
    try:
        endpoint, values = adapter.match()
        function = getattr(self, 'on_' + endpoint)
        transaction = newrelic.agent.current_transaction()
        with newrelic.agent.FunctionTrace(
                transaction, endpoint, 'Python/EndPoint'):
            return function(request, **values)
    except HTTPException as e:
        return e

The FunctionTrace class implements the context manager and is used in conjunction with the with statement. The FunctionTrace class is the lowest level primitive available for tracing time against a transaction. When using the FunctionTrace class, the agent must first look up the current transaction so that it can be supplied as the first argument.

The with statement defines the bounds of what is timed and not a single function. Thus, the context manager could be applied to an arbitrary block of code. The block of code could therefore contain calls to multiple functions, or it could be a self-contained block of code implementing a time oriented algorithm that you want to track.

Important: Avoid tracing blocks of code that are called an excessive number of times. For example, do not use it within the context of a loop that executes many times. This is because using the context manager and dealing with the data it generates will incur a performance overhead. A better solution is to enclose the loop.

For more help

Additional documentation resources include:

Join the discussion about Python in the New Relic Online Technical Community! The Technical Community is a public platform to discuss and troubleshoot your New Relic toolset.

If you need additional help, get support at support.newrelic.com.