Python message queues

The New Relic Python agent supports the Pika RabbitMQ client library, giving you visibility into the performance of your message processing, for both incoming and outgoing messages.

The New Relic APM UI will show transactions initiated via RabbitMQ message receipt (subscribe/consume messages) as Message background tasks. Message creation via RabbitMQ also appears in transaction traces.

Requirements

Requires New Relic Python agent version 2.88.0.72 or higher.

Message tasks are not started for consumers using TornadoConnection.

Performance improvements with background tasks

One way to increase responsiveness of web applications is to delegate work to background processes. Message queues are commonly used for this inter-process communication.

In the context of message queuing systems, applications typically interact with message brokers to send and receive messages. The RabbitMQ Pika client library allows Python applications to interface with message brokers that implement the Advanced Message Queueing Protocol (AMQP) 0.9 or higher.

New Relic's Python agent shows messages sent and received using the RabbitMQ client library. With this visibility, you can see details including:

  • Number of messages produced by your app
  • Time your app spends publishing messages
  • Time your app spends processing "consumed" messages

New Relic APM conveniently groups and reports operations that interact with queues. By analyzing this information, you can more easily identify bottlenecks and areas for performance improvement in your message passing architecture.

Queue operations

Supported entry points for queue operations appear as Put (publish a message) or Take (receive a message) in New Relic APM's user interface.

Queue operations Publish a message (Put in UI) Receive a message (Take in UI)
RabbitMQ basic_publish

basic_get

basic_consume (callback)

View in New Relic UI

Queue operations appear on New Relic APM's Transactions page for the selected app. The Put and Take metrics appear in the Breakdown table and are categorized as MessageBroker metrics. Here is an example:

New Relic for Python rabbitmq task
rpm.newrelic.com/apm > (select an app) > Monitoring > Transactions > (select a transaction): The transaction’s Breakdown table categorizes queue operations as MessageBroker metrics and labels them as Put (publish a message) or Take (receive a message).

Transaction traces also provide additional details for messages.

You can select transaction traces from the app's Overview or Transactions pages in New Relic APM.

The Transaction trace summary page may show Put and Take operations in the Slowest components section. For example:

New Relic for Python rabbitmq transaction trace summary
rpm.newrelic.com/apm > (select an app) > (select a transaction trace): In this example, the selected transaction trace's Summary shows RabbitMQ operations in the colored bar chart. The Slowest components section also lists the RabbitMQ Put and Take results.

The Transaction trace page includes a dedicated Messages tab that shows a summary of the message activity that occurred as part of the transaction. Here is an example:

New Relic for Python rabbitmq trace Messages
rpm.newrelic.com/apm > (select an app) > (select a transaction trace): In this example, the selected transaction trace's Messages tab lists the RabbitMQ message details.

For more help

Recommendations for learning more: