Configuring Celery with Redis as Broker and Backend

Celery is a distributed asynchronous computation framework that can be used independently of a web framework; this article shows how to configure Celery to use Redis as both the message broker and result backend.

Project structure

<code>$ tree your_project
your_project
├── __init__.py
├── main.py
├── celery.py
└── tasks.py

0 directories, 4 files</code>

Celery configuration (celery.py)

<code>from celery import Celery
from .settings import REDIS_URL

APP = Celery(
    main=__package__,
    broker=REDIS_URL,
    backend=REDIS_URL,
    include=[f'{__package__}.tasks'],
)

APP.conf.update(task_track_started=True)</code>

REDIS_URL is imported from settings.py and typically looks like redis://localhost:6379/0 . Using Redis as both broker and backend means it handles message queuing and stores task results (default retention 1 day).

Task definition (tasks.py)

<code>from .celery import APP

@APP.task
def do_sth():
    pass</code>

To trigger the task from business code, use .apply_async() :

<code>from celery.result import AsyncResult
from .tasks import do_sth

result = do_sth.apply_async()
assert isinstance(result, AsyncResult)</code>

Run a worker with:

<code>celery -A your_project worker</code>

Execution flow

The main process sends a message to Redis and receives an AsyncResult containing a task_id. Workers poll Redis, consume the message, execute the task, and write the result and state back to Redis. The main process can query the result using AsyncResult(task_id) .

Task states

Celery tasks can be in states such as PENDING, STARTED, RETRY, SUCCESS, FAILURE, REVOKED. If the result expires in Redis, the state reverts to PENDING.

Common control operations

<code>result = AsyncResult(task_id)
# block until result is ready
result.wait()
# cancel the task
result.revoke()
# delete task record
result.forget()</code>

Use wait to block for a result, revoke to cancel a queued or running task, and forget to remove its record.

References

Mostly based on the official Celery documentation, with additional insights from StackOverflow and community articles.