Use SQLite as a Celery broker in Django

Redis and RabbitMQ may be the go-to brokers when using Celery, but when you’re developing locally they can feel like overkill. The documentation for Celery 5.4 mentions that you can use SQLite as an experimental broker for local development. However, when you navigate to Celery’s backends and brokers page, the only mentions of SQL are for the SQLAlchemy backend. Thankfully, the page notes that, “this section is not comprehensive of backends and brokers.”

This post will show you how to implement an SQLite broker (or any SQL!) for Celery in a Django project. This post will not teach you to use Celery: check out the official Celery documentation for that.

Before we begin

For the purposes of this post, we will assume that you already have an existing Django project with Celery installed using the steps from Celery’s official “First steps with Django” guide. A Celery backend is not required, but you may want to follow the guide’s steps for installing and configuring django-celery-results. If you’re unclear on what the difference is between backends and brokers, check out my article “Understanding tasks, brokers, workers, and backends in Celery.”

All links to source documentation will be for current versions of Django, Celery, and SQLAlchemy at the time of publication (July, 2024), except where explicitly stated otherwise. If you’re reading this in the distant future, things may have changed.

Setting up the SQL Broker

While Celery manages tasks and queues, it delegates to another library called Kombu for exchanging messages with the broker. RabbitMQ and Redis are Kombu’s most feature-complete transports (brokers), but it also has virtual transports for Amazon SQS, ZooKeeper, and MongoDB.

Tucked away in the far corners of Kombu’s documentation, there is an SQLAlchemy Transport Model that supports PostgreSQL, MySQL, and SQLite. Once upon a time the SQLAlchemy broker was even documented on Celery’s website, but it has since been removed from the docs in newer versions of the library. Nonetheless, it still works well enough for local development.

To use a sequel database as a Celery broker in your Django app, first install SQLAlchemy:

pip install SQLAlchemy

Within your Django project’s settings.py file, you can set your broker’s backend using the CELERY_BROKER_URL setting:

# BASE_DIR is the directory of your project's main directory.

CELERY_BROKER_URL = f"sqlalchemy+sqlite:////{BASE_DIR}/broker.sqlite3"

The SQLAlchemy broker URL consists of 3 parts:

1. The string sqlalchemy or sqla (they are interchangeable)
2. A + sign
3. A SQLAlchemy connection string

Connection strings are different for SQLite on Mac/Unix and Windows:

# MacOS/Unix
CELERY_BROKER_URL = "sqla+sqlite:////your/project/path/broker.sqlite3" # Windows
CELERY_BROKER_URL = "sqla+sqlite:///C:your\\project\\path\\broker.sqlite3"

You can also use Postgres as a Celery broker, or you can just as easily use MySQL as a Celery broker:

# MySQL
CELERY_BROKER_URL = "sqlalchemy+mysql://scott:tiger@localhost/foo" # PostgreSQL
CELERY_BROKER_URL = "sqla+postgresql://scott:tiger@localhost/mydatabase" # PosgreSQL connecting using pg8000
CELERY_BROKER_URL = "sqla+postgresql+pg8000://scott:tiger@localhost/mydatabase"

You may need to install other libraries to connect to MySQL or PostgreSQL, and what library you install may affect the SQLAlchemy connection string. Check the SQLAlchemy Database URL docs for more details.

Regardless of which database you choose, you may want to consider storing the broker URL in an environment variable to make it easy to change in different environments:

CELERY_BROKER_URL = os.getenv("CELERY_BROKER_URL")

A word of caution

The SQLAlchemy transport is likely considered experimental, and therefore it would not be intended for production usage. Data loss may occur, or messages could be delivered multiple times. Consider switching to a more robust broker that’s listed on Celery’s Brokers and Backends page.

That said, it might be fine for local development, or even small side projects. But I wouldn’t be shocked if the ability to use SQLAlchemy as a broker went away in the future.

Next steps

With Celery running locally, you may begin working on your queue-powered application. However, you may find its lack of automatic reloading to be a point of friction. If you want to set up automatic Celery reloading in your Django application, read my post “Automatically reload Celery workers with a custom Django command.”