What is the issue?
The asynchronous worker manager creates an asyncio.Task for every queued work item before applying the configured concurrency limit.
What is the impact?
During a burst of work with slow handlers, the number of pending tasks can grow with the queue depth rather than the configured concurrency. This increases memory use and event-loop scheduling overhead. The manager also repeatedly scans the full set of running tasks to remove completed tasks.
Details about the issue including code reference
Relevant code:
|
async def _consume_queue(self, queue: asyncio.Queue[_WorkItem], semaphore: asyncio.Semaphore) -> None: |
|
# List to track running tasks |
|
running_tasks: set[asyncio.Task[Any]] = set() |
|
|
|
while True: |
|
# Clean up completed tasks |
|
done_tasks = {task for task in running_tasks if task.done()} |
|
running_tasks -= done_tasks |
|
|
|
# Exit if shutdown is set and the queue is empty and no tasks are running |
|
if self._shutdown and queue.empty() and not running_tasks: |
|
break |
|
|
|
try: |
|
work = await asyncio.wait_for(queue.get(), timeout=1.0) |
|
except asyncio.TimeoutError: |
|
continue |
|
|
|
func, cancellation_func, args, kwargs = work |
|
# Create a concurrent task for processing |
|
task = asyncio.create_task( |
|
self._process_work_item(semaphore, queue, func, cancellation_func, args, kwargs) |
|
) |
|
running_tasks.add(task) |
|
|
|
async def _process_work_item( |
|
self, semaphore: asyncio.Semaphore, queue: asyncio.Queue[_WorkItem], |
|
func: Callable[..., Any], cancellation_func: Callable[..., Any], |
|
args: tuple[Any, ...], kwargs: dict[str, Any] |
|
) -> None: |
|
async with semaphore: |
|
try: |
|
await self._run_func(func, *args, **kwargs) |
|
except Exception as work_exception: |
|
self._logger.error(f"Uncaught error while processing work item, item will be abandoned: {work_exception}") |
|
await self._run_func(cancellation_func, *args, **kwargs) |
|
finally: |
|
queue.task_done() |
_AsyncWorkerManager._consume_queue() obtains a queue item and immediately calls asyncio.create_task(). The semaphore is acquired later inside _process_work_item(). Therefore, the semaphore limits execution but not task allocation. The running_tasks cleanup scans the full task set each loop iteration.
A potential or proposed solution
Use a fixed number of consumers per work type, or acquire capacity before creating a task so the queue remains the bounded backlog. Remove completed tasks through done callbacks rather than repeatedly scanning the task set. Preserve current cancellation behavior and exactly-once queue.task_done() semantics.
What is the issue?
The asynchronous worker manager creates an
asyncio.Taskfor every queued work item before applying the configured concurrency limit.What is the impact?
During a burst of work with slow handlers, the number of pending tasks can grow with the queue depth rather than the configured concurrency. This increases memory use and event-loop scheduling overhead. The manager also repeatedly scans the full set of running tasks to remove completed tasks.
Details about the issue including code reference
Relevant code:
durabletask-python/durabletask/worker.py
Lines 3436 to 3473 in 55d8e0b
_AsyncWorkerManager._consume_queue()obtains a queue item and immediately callsasyncio.create_task(). The semaphore is acquired later inside_process_work_item(). Therefore, the semaphore limits execution but not task allocation. Therunning_taskscleanup scans the full task set each loop iteration.A potential or proposed solution
Use a fixed number of consumers per work type, or acquire capacity before creating a task so the queue remains the bounded backlog. Remove completed tasks through done callbacks rather than repeatedly scanning the task set. Preserve current cancellation behavior and exactly-once
queue.task_done()semantics.