r/learnpython • u/Expensive_Break_6163 • 21h ago

Does asyncio have a built-in feature for graceful shutdown (and preventing duplicate task execution)?

Multiple pods, each an asyncio worker pulling jobs from a shared queue (Redis/SQS-shaped). On scale-down/deploy I want to stop pulling new jobs and let running ones finish - is there anything in asyncio itself for this?

2 Upvotes
  • permalink
  • reddit

76% Upvoted

5 comments sorted by

3

u/Kevdog824_ 20h ago

Assuming your shutdown is graceful: I would probably use something like atexit.register to register a shutdown function that sends a signal to the worker to stop pulling new jobs and then await the completion of all the existing jobs

1

u/Mathie1729 14h ago

atexit is awkward for this. By the time it fires, the event loop is usually in the middle of teardown, so awaiting in-flight coroutines there is unreliable. I'd use loop.add_signal_handler for SIGTERM/SIGINT to set an asyncio.Event, have workers check that event before pulling new jobs, then await the current batch after the signal. Duplicate execution is more about idempotency keys or lease coordination than anything built into asyncio.

1

u/Kevdog824_ 14h ago

Definitely better approach

2

u/_squik 20h ago

I think the closest thing to what you want here is asyncio.TaskGroup

https://docs.python.org/3/library/asyncio-task.html#task-groups

2

u/itlogicpartnersllc 19h ago

asyncio can help with cancellation and taskgroup but graceful shutdown across multiple pods is really a queue/worker coordination problem so you will want to stop consuming first and let in fight jobs finish.