I like the Transactional Outbox pattern and use it quite often.
But I was thinking about one problem that can become important in production: database polling.
Of course, polling can be optimized. Good indexes, batching, SKIP LOCKED, partitioning, longer polling intervals — there are many options.
But if we want lower latency, we usually need to poll more often. This means more queries, more DB connections and more work for PostgreSQL.
We can use Debezium/CDC, and for many systems this is probably the right choice. But it also adds Kafka Connect, Debezium and more infrastructure to operate.
So I wanted to try a simpler idea:
Keep PostgreSQL as the source of truth, but don't use it as a queue during normal operation.
I built this flow:
DB transaction → afterCommit → Memory Queue → Batch Publisher → Kafka
The business data and Outbox event are saved in one transaction as usual.
After commit, only the eventId goes to the Memory Queue. The publisher takes IDs in batches, loads events from PostgreSQL and sends them to Kafka.
So there is no continuous polling for new events in the normal flow.
If the application crashes, the event is still safely stored in PostgreSQL. A Recovery Worker finds unpublished events and puts them back into the same queue.
The idea is basically:
Memory Queue for the fast path. PostgreSQL for durability and recovery.
I didn't want to stop at an architecture diagram, so I built a working Spring Boot project and started testing the idea with something closer to production conditions.
It has Kafka, PostgreSQL, batching, idempotency, recovery, Gatling load tests, Grafana metrics, tracing and structured logs.
Now I can run load tests and actually see what happens with PostgreSQL, the queue, publishing latency and recovery.
I'm interested in what other Spring Boot developers think about this approach.
Would you use something like this in production?
Maybe you have already solved the same problem in another way — optimized polling, Debezium, LISTEN/NOTIFY or something else?
Here is the project:
https://github.com/KHolodilin/spring-transactional-outbox-kafka
If you find the idea useful, feel free to ⭐ the repo or fork it. There are also a few open issues for contributors if you want to try something yourself.
Any feedback is welcome. I'm still experimenting with the approach and improving the project. 🚀