Skip to content

Make Pool.close() wait until all checked out connections are released - #299

Merged
elprans merged 1 commit into
masterfrom
pool
May 31, 2018
Merged

elprans merged 1 commit into
masterfrom
pool

Conversation

@elprans

@elprans elprans commented May 30, 2018

Copy link
Copy Markdown
Member

Currently, pool.close(), despite the "graceful" designation, closes
all connections immediately regardless of whether they are acquired.

With this change, pool will wait for connections to actually be released
before closing.

WARNING: This is a potentially incompatible behavior change, as sloppily
written code which does not release acquired connections will now cause
pool.close() to hang forever.

Also, when conn.close() or conn.terminate() are called directly
on an acquired connection, the associated pool item is released
immediately.

Closes: #290

@elprans
elprans requested a review from 1st1 May 30, 2018 22:23
Comment thread asyncpg/connection.py
def _abort(self):
# Put the connection into the aborted state.
self._aborted = True
self._protocol.abort()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd also add self._protocol = None to catch any use of connection after _abort().

Comment thread asyncpg/connection.py Outdated
self._clean_tasks()
if self._proxy is not None:
# Connection is a member of a pool, but is getting
# aborted directly. Notify the pool about the fact.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"is getting aborted directly." -- outdated comment?

Comment thread asyncpg/pool.py
# Use `close` to close the connection gracefully.
# An exception in `setup` isn't necessarily caused
# by an IO or a protocol error.
await self._con.close()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment that con.close() will cleanup its holder too.

Comment thread asyncpg/pool.py Outdated
assert self._in_use
self._in_use = False
self._timeout = None
assert self._in_use is not None

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's raise a proper exception here.

Comment thread asyncpg/pool.py
self._timeout = None

if self._con._protocol.queries_count >= self._max_queries:
await self._con.close(timeout=timeout)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add an early return here?

try:
    await self._con.close(..)
finally:
    self._release()
    return

Comment thread asyncpg/pool.py Outdated
self._inactive_callback.cancel()
self._inactive_callback = None

def _deactivate_connection(self):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rename this to _deactivate_inactive_connection()

Comment thread asyncpg/pool.py
self._con.terminate()
assert self._in_use is None
if self._con is not None:
self._con.terminate()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment that we terminate the connection instead of just closing it because it's not in use.

Comment thread asyncpg/pool.py
self._in_use = None

# Let go of the connection proxy.
if self._proxy is not None:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make sure we set and unset self._proxy and self._con consistently.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Connection and proxy lifetimes are not the same. A proxy only lives while the connection is acquired. I'll clarify with a comment.

Comment thread asyncpg/pool.py Outdated
self._check_init()

con = connection._con
con._on_release()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

connection._con._on_release() to make it easier for the reader (o/w I need to track if con is used in the code below)

Comment thread asyncpg/pool.py Outdated

con = connection._con
con._on_release()
ch = connection._holder

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you rename ch to holder?

Comment thread asyncpg/pool.py Outdated
# operations on it will fail.
if self._proxy is not None:
if self._proxy._con is not None:
self._proxy._detach()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just make _proxy._detach() idempotent.

Currently, `pool.close()`, despite the "graceful" designation, closes
all connections immediately regardless of whether they are acquired.

With this change, pool will wait for connections to actually be released
before closing.

WARNING: This is a potentially incompatible behavior change, as sloppily
written code which does not release acquired connections will now cause
`pool.close()` to hang forever.

Also, when `conn.close()` or `conn.terminate()` are called directly
on an acquired connection, the associated pool item is released
immediately.

Closes: #290
@elprans
elprans merged commit 7a0585a into master May 31, 2018
@elprans
elprans deleted the pool branch May 31, 2018 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pool.close() does not close the pool gracefully

2 participants