Cross application tracing

A distributed tracing feature is now available. Distributed tracing improves on cross application tracing and is recommended for monitoring activity in complex distributed systems.

Cross application tracing is supported by the Python agent through built in instrumentation and through the use of APIs for custom instrumentation.

The protocol used to communicate between applications involves attaching metadata to requests and responses. The metadata is processed by each application and the resulting data is reported to New Relic.

Requirements

You must have Python agent version 2.92.0.78 or higher.

Custom client (HTTP)

client_http_transport.png

These APIs are used for custom HTTP communication libraries that are not instrumented as part of the built in instrumentation.

HTTP cross application tracing uses HTTP headers to transport transaction metadata across applications. To generate outbound cross application tracing headers, use the generate_request_headers API on the ExternalTrace class. To process inbound cross application tracing headers, use the process_response_headers API on the ExternalTrace class.

Outgoing requests
import newrelic.agent

transaction = newrelic.agent.current_transaction()
# returns a list of tuples in the form (header_name, header_value)
outgoing_headers = newrelic.agent.ExternalTrace.generate_request_headers(transaction)
for header_name, header_value in outgoing_headers:
    ...request.putheader(header_name, header_value) # code to be instrumented
...make_request(request) # code to be instrumented
Incoming response
import newrelic.agent

transaction = newrelic.agent.current_transaction()
trace = newrelic.agent.ExternalTrace(transaction, 'library', 'http://example.com', 'get')
with trace:
    response = fetch_example_dot_com() # code to be instrumented
    trace.process_response_headers(...response.headers)

Custom client (non-HTTP)

client_custom_transport.png

These APIs are used to instrument libraries that do not use the HTTP transport (and therefore may not be able to use headers as a metadata transport) and are not instrumented as part of the built in instrumentation.

Example Instrumentation

Outgoing requests
import newrelic.agent

transaction = newrelic.agent.current_transaction()
# returns a string value
outgoing_metadata = newrelic.agent.ExternalTrace.get_request_metadata(transaction)
...request.add_metadata(outgoing_metadata) # code to be instrumented
...make_request(request) # code to be instrumented
Incoming response
import newrelic.agent

transaction = newrelic.agent.current_transaction()
trace = newrelic.agent.ExternalTrace(transaction, 'library', 'myproto://service', 'fetch')
with trace:
    ...response = fetch_from_service() # code to be instrumented
    # extract the metadata string sent from the service
    metadata_value = response.metadata # code to be instrumented
    trace.process_response_metadata(metadata_value)

WSGI server

The agent will automatically handle processing and sending responses to cross application trace metadata for all WSGI servers.

For information on instrumenting WSGI servers, see the wsgi_application API documentation for details.

Custom non-HTTP server

server_nonhttp_transport.png

Custom servers that are not WSGI based or do not use HTTP as a transport will have to process incoming cross application tracing metadata and generate cross application trace responses.

The following APIs allow processing of cross application tracing metadata sent on non-HTTP requests and generating response metadata to send back to the caller.

Example instrumentation
import newrelic.agent

def handle_incoming_request(request):
    transaction = newrelic.agent.current_transaction()
    if transaction:
    	transaction.process_request_metadata(...request.metadata)

    response = generate_response() # code to be instrumented

    if transaction:
    	# get_response_metadata returns a string
        ...response.metadata = transaction.get_response_metadata()
    return response

For more help

Recommendations for learning more: