Skip to content

Drop RuntimeError on closing a connection pool with active connections.#631

Merged
lovelydinosaur merged 6 commits intomasterfrom
drop-runtime-error-on-closing-connection-pool-with-open-connections
Dec 14, 2022
Merged

Drop RuntimeError on closing a connection pool with active connections.#631
lovelydinosaur merged 6 commits intomasterfrom
drop-runtime-error-on-closing-connection-pool-with-open-connections

Conversation

@lovelydinosaur
Copy link
Contributor

@lovelydinosaur lovelydinosaur commented Nov 30, 2022 •
edited
Loading

Closes #564.

Closes #561 (The motivation given there becomes redundant. Errors will be raised against the connections themselves)

With this script...

example.py:

import anyio
import httpcore


async def worker(client, tg):
    r = await client.request('HEAD', 'https://google.com')
    print(r)
    tg.cancel_scope.cancel()  # Cancel our other outstanding requests


async def main():
    async with httpcore.AsyncConnectionPool() as client, anyio.create_task_group() as tg:
        for _ in range(3):
            tg.start_soon(worker, client, tg)


if __name__ == "__main__":
    anyio.run(main)

Before this change...

$ venv/bin/python ./example.py 
<Response [301]>
Traceback (most recent call last):
  File "./example.py", line 18, in <module>
    anyio.run(main)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 583, in run_until_complete
    return future.result()
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "./example.py", line 14, in main
    tg.start_soon(worker, client, tg)
  File "/Users/tomchristie/GitHub/encode/httpcore/httpcore/_async/connection_pool.py", line 326, in __aexit__
    await self.aclose()
  File "/Users/tomchristie/GitHub/encode/httpcore/httpcore/_async/connection_pool.py", line 313, in aclose
    f"The connection pool was closed while {requests_still_in_flight} "
RuntimeError: The connection pool was closed while 2 HTTP requests/responses were still in-flight.

After this change...

$ venv/bin/python ./example.py 
<Response [301]>

But, if we attempt close the connection pool and then later attempt to read from a connection, we will still get sensible behaviour...

example.py:

import anyio
import httpcore


async def worker(client, tg):
    r = await client.request('HEAD', 'https://google.com')
    print(r)
    await client.aclose()  # Yikes, lookout.

async def main():
    async with httpcore.AsyncConnectionPool() as client, anyio.create_task_group() as tg:
        for _ in range(3):
            tg.start_soon(worker, client, tg)


if __name__ == "__main__":
    anyio.run(main)

Output:

$ venv/bin/python ./example.py 
<Response [301]>
Traceback (most recent call last):
  File "./example.py", line 17, in <module>
    anyio.run(main)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 583, in run_until_complete
    return future.result()
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "./example.py", line 13, in main
    tg.start_soon(worker, client, tg)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 702, in _run_wrapped_task
    await coro
  File "./example.py", line 6, in worker
    r = await client.request('HEAD', 'https://google.com')
  File "/Users/tomchristie/GitHub/encode/httpcore/httpcore/_async/interfaces.py", line 43, in request
    response = await self.handle_async_request(request)
  File "/Users/tomchristie/GitHub/encode/httpcore/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/Users/tomchristie/GitHub/encode/httpcore/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/Users/tomchristie/GitHub/encode/httpcore/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/Users/tomchristie/GitHub/encode/httpcore/httpcore/_async/http11.py", line 112, in handle_async_request
    raise exc
  File "/Users/tomchristie/GitHub/encode/httpcore/httpcore/_async/http11.py", line 91, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/Users/tomchristie/GitHub/encode/httpcore/httpcore/_async/http11.py", line 155, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/Users/tomchristie/GitHub/encode/httpcore/httpcore/_async/http11.py", line 192, in _receive_event
    self.READ_NUM_BYTES, timeout=timeout
  File "/Users/tomchristie/GitHub/encode/httpcore/httpcore/backends/asyncio.py", line 36, in read
    return b""
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/tomchristie/GitHub/encode/httpcore/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ReadError

@lovelydinosaur lovelydinosaur mentioned this pull request Nov 30, 2022
Closed
@lovelydinosaur lovelydinosaur requested a review from a team December 1, 2022 10:37
@lovelydinosaur lovelydinosaur mentioned this pull request Dec 8, 2022
Merged
5 tasks
@Kludex
Copy link
Contributor

Kludex commented Dec 8, 2022

Closes #631 (Errors will be raised against the connections themselves)

This PR is #631, is that correct?

@lovelydinosaur
Copy link
Contributor Author

lovelydinosaur commented Dec 8, 2022 •
edited
Loading

This PR is #631, is that correct?

Ooops, updated.
Should have (and now does) read #561.

zanieb
zanieb reviewed Dec 8, 2022
View reviewed changes
@Kludex Kludex requested a review from zanieb December 12, 2022 12:36
@lovelydinosaur lovelydinosaur merged commit 585b6f3 into master Dec 14, 2022
@lovelydinosaur lovelydinosaur deleted the drop-runtime-error-on-closing-connection-pool-with-open-connections branch December 14, 2022 15:58
@lovelydinosaur
Copy link
Contributor Author

lovelydinosaur commented Dec 14, 2022

Thanks @madkinsz 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zanieb zanieb zanieb approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Bug List in-flight connections

3 participants

@lovelydinosaur @Kludex @zanieb