How to debug a stuck asyncio coroutine in Python?

Question

There are lots of coroutines in my production code, which are stuck at unknown position while processing request. I attached gdb with Python support extension to the process, but it doesn't show the exact line in the coroutine where the process is stuck, only primary stack trace. Here is a minimal example:

import asyncio

async def hello():
    await asyncio.sleep(30)
    print('hello world')

asyncio.run(hello())

(gdb) py-bt
Traceback (most recent call first):
  File "/usr/lib/python3.8/selectors.py", line 468, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 2335, in _run_once
  File "/usr/lib/python3.8/asyncio/base_events.py", line 826, in run_forever
    None, getaddr_func, host, port, family, type, proto, flags)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.8/asyncio/runners.py", line 299, in run
  File "main.py", line 7, in <module>

GDB shows a trace that ends on line 7, but the code is obviously stuck on line 4. How to make it show a more complete trace with nested coroutines?

Praveen mundur · Accepted Answer · 2021-05-27 11:44:21Z

3

You can use the aiodebug.log_slow_callbacks.enable(0.05)

Follow for more : https://pypi.org/project/aiodebug/

answered May 27, 2021 at 11:44

Praveen mundur

665 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Roland Pihlakas Over a year ago

The usage is import aiodebug.log_slow_callbacks aiodebug.log_slow_callbacks.enable(0.05).

shubh · Accepted Answer · 2025-05-19 19:00:10Z

With my limited knowledge of asyncio, my guess is the code is indeed stuck at line 7. With the async and await keywords, the code got (pre-)compiled into multiple functions where the code right after your await sleep becomes another function and gets registered on the Event Loop as a call back.
The sleep is implemented in the EventLoop using the _selector.poll which is what it shows as the point where the wait is. Once the poll returns due to an I/O event or timeout, the next set of callbacks in the Event Loop will be called - if eligible. If the 30 seconds you specified have passed, then your callback would be eligible and the print function would get executed.
I know this does not directly help you find the bottleneck, but hopefully my over-simplified description of the Event Loop logic helps in understanding of the stack trace when using a traditional debugger or profiling tool.

I am in the process of looking at some of the asyncio aware profiling tools and will circle back here if I find something useful. Not having much luck with aiodebug so far.

I have played around with scalene and found it useful:
python -m scalene --outfile test.html --html test.py

Collectives™ on Stack Overflow

How to debug a stuck asyncio coroutine in Python?

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related