Getting Started with Celery: Architecture, Installation, and Basic Usage

Celery is a Python‑based distributed task queue that enables task scheduling across multiple machines, processes, or threads.

Architecture

Celery consists of three components: a message broker (e.g., RabbitMQ, Redis, MongoDB, Amazon SQS, CouchDB, SQLAlchemy, Django ORM, IronMQ), workers that execute tasks, and a task result store (e.g., AMQP, Redis, memcached, MongoDB, SQLAlchemy, Django ORM, Apache Cassandra, IronCache).

Concurrency and Serialization

Workers can run using prefork, eventlet, gevent, or threads. Supported serializers include pickle, json, yaml, msgpack, zlib, bzip2, and cryptographic message signing.

Installation and Setup (AWS EC2 Linux example)

First install a broker; the example uses RabbitMQ, which requires Erlang. The following commands illustrate the required YUM and wget steps:

sudo yum -y update

# Add and enable relevant application repositories:

# Note: We are also enabling third‑party remi package repositories.

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm

sudo rpm -Uvh remi-release-6*.rpm epel-release-6*.rpm

# Finally, download and install Erlang:

yum install -y erlang

Then install RabbitMQ:

# Download the latest RabbitMQ package using wget:

wget ...

# Add the necessary keys for verification:

rpm --import ...

# Install the .RPM package using YUM:

yum install rabbitmq-server-3.2.2-1.noarch.rpm

Start the RabbitMQ service: rabbitmq-server start Install Celery with pip: pip install Celery Create a simple task definition (tasks.py):

from celery import Celery
app = Celery('tasks', backend='amqp', broker='amqp://guest@localhost//')
app.conf.CELERY_RESULT_BACKEND = 'db+sqlite:///results.sqlite'

@app.task
def add(x, y):
    return x + y

Run a worker in the project directory: celery -A tasks worker --loglevel=info The -A flag specifies the Celery app name; the example uses SQLAlchemy as the result backend, which must be installed beforehand.

Sample client code to invoke the task:

from tasks import add
import time
result = add.delay(4, 4)
while not result.ready():
    print "not ready yet"
    time.sleep(5)
print result.get()

Running the client prints "not ready" until the worker finishes, then outputs the result (8). Worker logs show the task ID, receipt, and successful completion.

Note: Using the AMQP backend for results can lose the return value because the message is removed from the queue after the worker reads it; storing results in a persistent backend such as SQLAlchemy avoids this issue.