Skip to content

slow stream decompression #218

Open
Open

Description

@danielhrisca

Hello,

I've followed the documentation and have compressed a 2GB file using the LZ4StreamCompressor with a page size of 1MB.

The compression speed is as expected, but when I use the LZ4StreamDecompressor I get very slow decompression speed. I've tried using a read-only mmap of the compressed file, and also loading the whole file in a bytes or bytearray object, but the speed is low in all cases at around 200KB/s (yes kilobytes)

Any ideas what I might try?

Activity

  1. jonathanunderwood commented on Feb 8, 2021

    @jonathanunderwood
    Member

    @tSed any thoughts on this?

  2. danielhrisca commented on Feb 8, 2021

    @danielhrisca
    Author
    • windows 10 x64
    • python 3.7.7 x64
    • python-lz4 3.1.3
  3. jonathanunderwood commented on Nov 21, 2021

    @jonathanunderwood
    Member

    I am afraid I don't have time to maintain the streaming protocol code, as I don't use it myself anywhere. It was contributed by @tSed who has subsequently not responded to problems. I'll be disabling building of it for the next release if noone steps up to maintain it.

  4. jonathanunderwood commented on Feb 17, 2022

    @jonathanunderwood
    Member

    Release 4.0.0 has relegated the streams bindings to experimental status, and the stream bindings are no longer built by default. The stream bindings and their test suite need a significant rethink. If anyone is interested in helping out with this, I am happy to collaborate on making them robust and sustainable.

  5. jmg-duarte commented on Sep 18, 2024

    @jmg-duarte
    Contributor

    I think the issue here is this memcpy:

    memcpy (context->output.buf,
    context->strategy.ops->get_work_buffer (context),
    output_size);

    The small buffer size implies that it's constantly being called just to copy a measly 8192 bytes.

    EDIT: the issue is not the memcpy, changing the code not to use it has the same issue. I suspect it may instead have to do with the block sizes.

    If I change the top parameters from the example to:

    block_size_length = 4 # LZ4 compressed block size stored on 2 bytes
    page_size = 65536 # LZ4 context double buffer page size
    

    I get a 4x performance improvement for a 1GB file using the script from the docs.

    Details

    I removed the second .extend to make sure reallocation was not the issue here.

    from lz4.stream import LZ4StreamCompressor, LZ4StreamDecompressor import os block_size_length = 4 # LZ4 compressed block size stored on 2 bytes page_size = 65536 # LZ4 context double buffer page size origin_stream = 1000 * 1024 * os.urandom(1024)

    import time

    start = time.perf_counter()

    LZ4 stream compression of origin_stream into compressed_stream:

    compressed_stream = bytearray()
    with LZ4StreamCompressor("double_buffer", page_size, store_comp_size=block_size_length) as proc:
    offset = 0
    while offset < len(origin_stream):
    chunk = origin_stream[offset:offset + page_size]
    block = proc.compress(chunk)
    compressed_stream.extend(block)
    offset += page_size

    end = time.perf_counter() - start
    print(end)

    start = time.perf_counter()

    LZ4 stream decompression of compressed_stream into decompressed_stream:

    decompressed_stream = bytearray( len(origin_stream))
    with LZ4StreamDecompressor("double_buffer", page_size, store_comp_size=block_size_length) as proc:
    offset = 0
    while offset < len(compressed_stream):
    block = proc.get_block(compressed_stream[offset:])
    new_offset = offset + block_size_length + len(block)
    # chunk = proc.decompress(block)

        decompressed_stream[new_offset:] = proc.decompress(block)
    
        # decompressed_stream.extend(chunk)
        # offset += block_size_length + len(block)
        offset = new_offset
    

    end = time.perf_counter() - start
    print(end)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedstreamRelated to stream compression bindings

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions