Speedup readexactly#395
Conversation
|
I still not tested performance. Please be patient, or help me with that :) |
| current_len = len(self._buffer) | ||
|
|
||
| if current_len >= n: | ||
| break |
|
#394 should be fixed with that. Will benchmark after some time. |
|
@socketpair I can merge this PR in 3.6 today/tomorrow if you can show that it improves performance. |
| yield from self._wait_for_data('readexactly') | ||
|
|
||
| data = bytes(memoryview(self._buffer)[:n]) | ||
| del self._buffer[:n] |
There was a problem hiding this comment.
In non-cpython implementations, destructing of memoryview may be postponed, so cutting buffer may fail. In order to fix that, we should call memoryview_obj.release()...
Not sure if it is actual.
There was a problem hiding this comment.
Ah, you're right, I think.
I wonder if there is more simple and safe API.
There was a problem hiding this comment.
To overcome that, it is require to make two separate variables: i.e.
memview1 = memoryview(self._buffer)
memview2 = memview1[:n]
data = bytes(memview2)
memview2.release()
memview1.release()So, I decide to eliminate memoryview...Yes it should be faster, but....
There was a problem hiding this comment.
I think you should watch your language.
There was a problem hiding this comment.
Sorry, I have fixed comment.
|
I have benchmarked that. program: #!/usr/bin/env python3.5
import asyncio
import os
import time
async def action(rs, chunksize):
try:
while True:
await rs.readexactly(chunksize)
except asyncio.IncompleteReadError:
pass
async def writer(ws, bufsize):
ws.write(b'x' * bufsize)
await ws.drain()
ws.close()
async def amain():
bufsize = 30 * 1024 * 1024
chunksize = 16
handler = lambda rs, ws: writer(ws, bufsize)
server = await asyncio.start_server(handler, '127.0.0.1', 0)
addr = server.sockets[0].getsockname()
(rs, ws) = await asyncio.open_connection(*addr)
server.close()
await server.wait_closed()
# now writing part is running "in background". TODO: wait it completion somehow
await asyncio.sleep(3)
a = time.monotonic()
await action(rs, chunksize)
b = time.monotonic()
ws.close()
print('chunk size={}: {:.2f} seconds.'.format(
chunksize,
b - a,
))
loop = asyncio.get_event_loop()
loop.run_until_complete(amain()) |
|
Adding memoryview slowers down (!) so, it is good not to use |
|
Any feedback ? |
|
|
||
| yield from self._wait_for_data('readexactly') | ||
|
|
||
| if len(self._buffer) == n: |
There was a problem hiding this comment.
Really frequent case for protocols like this: http://www.postfix.org/socketmap_table.5.html where we expect a string with known length, data after which is not expected.
(Did not benchmarked if this really speedup)
There was a problem hiding this comment.
IMO, I want to add helper C function for "slicing bytes from bytearray".
And I hope it will be part of Python 3.7, (e.g. bytes.frombuffer(buff, start=0, end=None)).
|
lgtm |
|
@1st1 @gvanrossum ping |
|
I'll commit this later today. |
|
Committed by hand in b0e7438. Thanks! |
4 participants
See #394