Skip to content
This repository was archived by the owner on Nov 23, 2017. It is now read-only.

Speedup readexactly #395

Closed
socketpair wants to merge 1 commit into from
Closed

Speedup readexactly #395

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 17 additions & 19 deletions asyncio/streams.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -448,6 +448,7 @@ def _wait_for_data(self, func_name):
assert not self._eof, '_wait_for_data after EOF'

# Waiting for data while paused will make deadlock, so prevent it.
# This is essential for readexactly(n) for case when n > self._limit.
if self._paused:
self._paused = False
self._transport.resume_reading()
Expand Down Expand Up @@ -658,25 +659,22 @@ def readexactly(self, n):
if n == 0:
return b''

# There used to be "optimized" code here. It created its own
# Future and waited until self._buffer had at least the n
# bytes, then called read(n). Unfortunately, this could pause
# the transport if the argument was larger than the pause
# limit (which is twice self._limit). So now we just read()
# into a local buffer.

blocks = []
while n > 0:
block = yield from self.read(n)
if not block:
partial = b''.join(blocks)
raise IncompleteReadError(partial, len(partial) + n)
blocks.append(block)
n -= len(block)

assert n == 0

return b''.join(blocks)
while len(self._buffer) < n:
if self._eof:
incomplete = bytes(self._buffer)
self._buffer.clear()
raise IncompleteReadError(incomplete, n)

yield from self._wait_for_data('readexactly')

if len(self._buffer) == n:

@socketpair socketpair Sep 26, 2016 •
edited
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really frequent case for protocols like this: http://www.postfix.org/socketmap_table.5.html where we expect a string with known length, data after which is not expected.

(Did not benchmarked if this really speedup)

@methane methane Sep 27, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, I want to add helper C function for "slicing bytes from bytearray".
And I hope it will be part of Python 3.7, (e.g. bytes.frombuffer(buff, start=0, end=None)).

data = bytes(self._buffer)
self._buffer.clear()
else:
data = bytes(self._buffer[:n])
del self._buffer[:n]
self._maybe_resume_transport()
return data

if compat.PY35:
@coroutine
Expand Down