---
title: Python agent and Celery
source: https://docs.newrelic.com/docs/apm/agents/python-agent/back-end-services/python-agent-celery
---

If you're using Celery as a distributed task queuing system, you can use the Python agent to record Celery processes as [non-web transactions](https://docs.newrelic.com/docs/apm/transactions/intro-transactions/monitor-background-processes-other-non-web-transactions).

To get the Python agent working with Celery, first follow the [agent installation instructions](https://docs.newrelic.com/docs/agents/python-agent/installation-and-configuration/python-agent-installation) to install, configure, and [test](https://docs.newrelic.com/docs/agents/python-agent/installation-and-configuration/testing-python-agent) the agent. Then use these Celery-specific instructions.

## Run Celery [#running-with-celery]

The command you use to run Celery with the agent depends on your Celery version and your specific setup.

**celery**

If you use the `celery` command, include any custom options:

````sh
NEW_RELIC_CONFIG_FILE=/some/path/newrelic.ini newrelic-admin run-program celery YOUR_COMMAND_OPTIONS
```

For example:

```sh
NEW_RELIC_CONFIG_FILE=/some/path/newrelic.ini newrelic-admin run-program celery -A tasks worker --loglevel=info
```

````

**celeryd**

The `celeryd` run command is deprecated in newer Celery releases. If you still use `celeryd`, include any custom options:

````sh
NEW_RELIC_CONFIG_FILE=some/path/newrelic.ini newrelic-admin run-program celeryd YOUR_COMMAND_OPTIONS
```

````

**Django**

If you start Celery with a Django management command, use `manage.py`. Replace the `celery` command with your Django command, and include any custom options:

````sh
NEW_RELIC_CONFIG_FILE=/some/path/newrelic.ini newrelic-admin run-program python manage.py celery YOUR_COMMAND_OPTIONS
```

````

**Init scripts**

If you're using one of Celery's generic init scripts, edit the script and wrap the `celery` command that is called when the script is used. This command usually looks like:

````sh
CELERY_BIN=${CELERY_BIN:-"celery"}
```

To add the agent's wrapper script, change the `CELERY_BIN` variable to the following, where `some/path/` is your actual path to the file or package:

```sh
CELERY_BIN=${CELERY_BIN:-"NEW_RELIC_CONFIG_FILE=/some/path/newrelic.ini /some/path/bin/newrelic-admin run-program /some/path/bin/celery"}
```

With this change, every time `CELERY_BIN` is called, it will be called with the agent wrapper script around the actual `celery` command.

````

## Select the application name [#selecting-the-application-name]

The `app_name` setting in the Python agent configuration file sets the app name displayed in the New Relic UI.

-   If your Python agent monitors Celery tasks and you set `app_name` to the same value used in your application agent's `app_name`, the data from both sources will be combined in the UI under that name.
-   If you set different names, the data appears separately in the UI as two different names.

By [setting multiple app names in the agent config files](https://docs.newrelic.com/docs/agents/manage-apm-agents/app-naming/use-multiple-names-app), you can monitor both the combined data and the segregated data. Here's a common way to do this, using a Django application as an example:

**Using multiple application names**

For the agent monitoring the Django app, set the following:

````ini
app_name = Your_app_name (Django); Your_app_name (Combined)
```

For the agent monitoring Celery, set the following:

```ini
app_name = Your_app_name (Celery); Your_app_name (Combined)
```

This results in three displayed names in the UI:

* `Your_app_name (Combined)`, which combines data from both Celery and the Django application
* `Your_app_name (Django)`, which shows the Django app data
* `Your_app_name (Celery)`, which shows the Celery data

````

## Ignore task retry errors [#ignoring-task-retry-errors]

When using Celery, a task can possibly fail and raise a `celery.exceptions:Retry` or `celery.exceptions:RetryTaskError` exception. The exception depends on which version of Celery is used.

To ignore these errors, set this from the New Relic **Application settings** UI or from the agent config file. UI changes override config file changes per the [configuration precedence rules](https://docs.newrelic.com/docs/agents/python-agent/installation-configuration/python-agent-configuration#options).

To use ignore error settings from the UI:

1.  From **[one.newrelic.com](https://one.newrelic.com/all-capabilities)**, select **APM > (select an app) > Settings > Application**.
2.  Select **Server-side agent configuration**.
3.  From **Error collection**, enter the errors you want to ignore, separated by commas.

To ignore these errors using the agent config file, use the [`ignore_errors`](https://docs.newrelic.com/docs/agents/python-agent/installation-configuration/python-agent-configuration#error-ignore) setting and a space-separated list of exceptions:

```ini
error_collector.ignore_errors = celery.exceptions:Retry celery.exceptions:RetryTaskError
```

## Troubleshooting

When a Celery worker process is killed suddenly, the agent is not able to complete its normal shutdown process, which means its final data payload is not sent. This results in the agent reporting fewer Celery transactions than expected or no transactions at all.

This may occur when using the `CELERYD_MAX_TASKS_PER_CHILD` setting, which defines the maximum number of tasks a pool worker process can execute before it's replaced with a new one. If used, the worker is forcibly shut down when that limit is reached, which means that that data is not recorded by the agent. For example, if `MAX_TASKS_PER_CHILD = 1`, this results in no data being reported.

How to troubleshoot this will depend on why you want to use the `MAX_TASKS_PER_CHILD` limit in your application.

-   To allow the normal shutdown process, return this to the default no-limit setting.
-   To lessen the impact of the problem, raise the `MAX_TASKS_PER_CHILD` limit.

> #### ⚠️ IMPORTANT
>
> For version 3.2.2.94 or higher, the Python agent will shut down when the `MAX_TASKS_PER_CHILD` limit is reached. No data will be lost.

> #### ⚠️ IMPORTANT
>
> Monitoring the main process of Celery isn't possible with the agent, it can only monitor the worker processes. See [Activate application warning](https://docs.newrelic.com/docs/apm/agents/python-agent/troubleshooting/activate-application-warning-python)