r/SQL • u/mochama254 • 2d ago

PostgreSQL walbox: react to PostgreSQL changes from Python

I built this because I wanted to react to PostgreSQL changes from Python without polling, without triggers, and without pulling in a whole CDC platform.

It consumes PostgreSQL logical replication and exposes committed transactions as an async stream in Python.

What it does:

  • Keeps a durable checkpoint. If the process dies, it resumes from the last transaction it actually finished, not the last one it started.
  • Bounded delivery queue, so a slow handler doesn't let memory grow without limit.
  • Reconnects automatically after the connection drops.
  • One dependency: psycopg3.

The transactional outbox is one use case, but it works with any published table.

GitHub: https://github.com/mochams/walbox

Curious to hear where this wouldn't fit your setup, or what's missing if you've solved this problem a different way.

2 Upvotes

67% Upvoted

1 comment sorted by

View all comments

1

u/refaelos 1d ago

One place this wouldn't fit: multiple independent consumers on the same table. Logical replication slots are single-consumer.

If two different services need to react to the same table's changes independently (not just fan-out from one process), you need either separate slots (extra WAL retention pressure per slot) or a broker in front of walbox re-publishing to each. Worth calling out in the README, since it's the first wall people hit once a second team wants in.

Also curious about the checkpoint's failure mode: if a handler has non-idempotent side effects (an email send, a webhook) and the process dies mid-transaction, does replay re-run that handler for the transaction it didn't finish? At-least-once delivery with non-idempotent side effects is the classic footgun with CDC consumers.