How to configure Celery¶
This article assumes that you are already familiar with the basics of using Celery with Django. If not, please see Celery’s documentation.
Add Celery to your project¶
In your project’s subscription, add the number of Celery workers you require. You can start with just one and add more later if required.
Celery will then be provisioned on your project’s Test and Live servers by our infrastructure team. This includes the installation of our Aldryn Celery addon, and configuration of new environment variables your project will need.
About Aldryn Celery¶
You don’t need to use Aldryn Celery to use Celery and Django Celery on Divio Cloud - you can of course install and configure Celery components manually if you prefer, perhaps if you wish to use a version that we haven’t provided support for in Aldryn Celery. You will in that case need to:
- install the Celery components you need in your project’s requirements file
- apply the settings we provide as environment variables.
Configure Celery for the local server¶
For development purposes you will need to set up Celery in the local environment too, in such a way that it reflects the provision made on our Cloud.
In the following steps you will set up a number of local services using the docker-compose.yml file:
rabbitmq- RabbitMQ, a broker service responsible for the creation of task queues, assignment of tasks to queues and delivering tasks to workers
celeryworker; the worker(s) will execute the assigned tasks
celerybeat, a scheduling service for periodic tasks
celerycam, a monitoring service
Your project will already have at least two services,
db, listed in
docker-compose.yml. Each of the
new services will be need to be added in a similar way, so that each runs in its own Docker container locally. (On our
Cloud, they will run on our dedicated clusters, and the docker-compose file is not used.)
Set up the RabbitMQ messaging service, by adding the following lines:
services: web: [...] db: [...] rabbitmq: image: rabbitmq:3.5-management hostname: rabbitmq ports: - "15672:15672" expose: - "15672" environment: RABBITMQ_ERLANG_COOKIE: secret_cookie_value
This uses the official Docker RabbitMQ image (the
rabbitmq:3.5-management image in turn installs
rabbitmq:3.5). It also gives the container a hostname
rabbitmq), maps and exposes the management interface port (
15672) and sets a
environment variable (the actual
secret_cookie_value here doesn’t matter too much - you’re only using this locally).
Next add a Celery worker service in the same way. This service needs to run a Django environment almost identical to
that used by the
web service, as it will use the same codebase, need access to the same database and so on. Its
definition will therefore be very similar, with key changes noted here:
celeryworker: build: "." links: - "db:postgres" - "rabbitmq:rabbitmq" volumes: - ".:/app:rw" - "./data:/data:rw" command: aldryn-celery worker env_file: .env-local
Rather than copying the example above, use the actual
web service in your
docker-compose file as its basis, in
case it contains other values that need to be present. Note that the
ports option is not used.
command option starts the worker process, and
links provides a reference to the
Celery beat needs to be set up in much the same way:
celerybeat: build: "." links: - "db:postgres" - "rabbitmq:rabbitmq" volumes: - ".:/app:rw" - "./data:/data:rw" command: aldryn-celery beat env_file: .env-local
And Celery cam:
celerycam: build: "." links: - "db:postgres" - "rabbitmq:rabbitmq" volumes: - ".:/app:rw" - "./data:/data:rw" command: aldryn-celery cam env_file: .env-local
Finally, to the
links option in
web, you also need to add the link to
web: [...] links: [...] - "rabbitmq:rabbitmq"
Set up local environment variables¶
(Don’t confuse the port
5672 of the RabbitMQ server with the port
15672 of its management interface.)
Run the local project¶
docker-compose up or
divio project up will now also start the services that Celery requires.
Note that although the Django runsever in your
web container will restart automatically to load new code whenever
you make changes, that will not apply to the other services.
These will need to be restarted manually, for example by stopping and restarting the local project or by running
docker-compose restart. (Usually, only the
celeryworker container needs to be restarted, so you can do
docker-compose restart celeryworker.)
It’s not within the scope of this documentation to explain how to get started with or use Celery, but as a quick check
that you have configured your local environment correctly, you can create a small Celery task in your project, in a new
In the root of your project, add the application:
tasks_app/ __init__.py tasks.py
And in the
from celery.task import task from aldryn_celery.celery import app @app.task() def add(x, y): return x + y
Note that we are using Aldryn Celery’s ready configured code here for convenience - otherwise, you would follow the steps as described in the First steps with Django from the Celery documentation.
And finally, add
celeryworker container, and start a new Django shell with:
docker-compose run --rm web python manage.py shell
Then in the shell:
>>> from tasks_app.tasks import add >>> result = add.delay(2, 3)
result is a Celery
AsyncResult instance, so you can get the return value:
>>> result.get(timeout=1) 5
If that works successfully, you have created a task, and been able to use RabbitMQ to send it to a waiting Celery worker.
See the Celery documentation for more information.
When Celery is enabled for your project, two new environment variables will be configured:
The Test and Live servers will have different values for both.
Other environment variables used by Aldryn Celery can be found in its aldryn_config.py.
If you change environment variables locally, the containers will need to be stopped and restarted in order to pick up the changes.