You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've followed the documentation and have compressed a 2GB file using the LZ4StreamCompressor with a page size of 1MB.
The compression speed is as expected, but when I use the LZ4StreamDecompressor I get very slow decompression speed. I've tried using a read-only mmap of the compressed file, and also loading the whole file in a bytes or bytearray object, but the speed is low in all cases at around 200KB/s (yes kilobytes)
I am afraid I don't have time to maintain the streaming protocol code, as I don't use it myself anywhere. It was contributed by @tSed who has subsequently not responded to problems. I'll be disabling building of it for the next release if noone steps up to maintain it.
Release 4.0.0 has relegated the streams bindings to experimental status, and the stream bindings are no longer built by default. The stream bindings and their test suite need a significant rethink. If anyone is interested in helping out with this, I am happy to collaborate on making them robust and sustainable.
Activity
jonathanunderwood commentedon Feb 8, 2021
@tSed any thoughts on this?
danielhrisca commentedon Feb 8, 2021
jonathanunderwood commentedon Nov 21, 2021
I am afraid I don't have time to maintain the streaming protocol code, as I don't use it myself anywhere. It was contributed by @tSed who has subsequently not responded to problems. I'll be disabling building of it for the next release if noone steps up to maintain it.
jonathanunderwood commentedon Feb 17, 2022
Release 4.0.0 has relegated the streams bindings to experimental status, and the stream bindings are no longer built by default. The stream bindings and their test suite need a significant rethink. If anyone is interested in helping out with this, I am happy to collaborate on making them robust and sustainable.
jmg-duarte commentedon Sep 18, 2024
I think the issue here is thismemcpy:python-lz4/lz4/stream/_stream.c
Lines 1389 to 1391 in 65b563a
The small buffer size implies that it's constantly being called just to copy a measly 8192 bytes.EDIT: the issue is not the
memcpy, changing the code not to use it has the same issue. I suspect it may instead have to do with the block sizes.If I change the top parameters from the example to:
I get a 4x performance improvement for a 1GB file using the script from the docs.
Details
I removed the second
.extendto make sure reallocation was not the issue here.from lz4.stream import LZ4StreamCompressor, LZ4StreamDecompressor import os block_size_length = 4 # LZ4 compressed block size stored on 2 bytes page_size = 65536 # LZ4 context double buffer page size origin_stream = 1000 * 1024 * os.urandom(1024)
import time
start = time.perf_counter()
LZ4 stream compression of origin_stream into compressed_stream:
compressed_stream = bytearray()
with LZ4StreamCompressor("double_buffer", page_size, store_comp_size=block_size_length) as proc:
offset = 0
while offset < len(origin_stream):
chunk = origin_stream[offset:offset + page_size]
block = proc.compress(chunk)
compressed_stream.extend(block)
offset += page_size
end = time.perf_counter() - start
print(end)
start = time.perf_counter()
LZ4 stream decompression of compressed_stream into decompressed_stream:
decompressed_stream = bytearray( len(origin_stream))
with LZ4StreamDecompressor("double_buffer", page_size, store_comp_size=block_size_length) as proc:
offset = 0
while offset < len(compressed_stream):
block = proc.get_block(compressed_stream[offset:])
new_offset = offset + block_size_length + len(block)
# chunk = proc.decompress(block)
end = time.perf_counter() - start
print(end)