Skip to content

Low decompression speed in multithreading #269

Open

Description

@Dalbasar

As far as i understand, the GIL should be dropped when calling the underlying LZ4 C library during compression and decompression, see lz4.block.decompress

I only see a minor speedup when using multiple threads for decompression using python-lz4 4.3.2 on a 6 core Intel i5-8400 on Debian 11 and also not on a Amd Ryzen 5900X on Windows 10, neither with Python 3.11.1 nor 3.8.10.

Compression speed seems to increase almost linearly with the number of threads.

The following code gives me about 4500MB/s decompression speed (slight underestimation due to some overhead from starting the threads) when using 6 threads and ~4300MB/s when using 1 thread on an AMD Ryzen 5900X on Windows 10. Using lz4.frame yields similar results. Using py-lz4framed instead gives me about 13000MB/s using 6 threads and ~8300MB/s on 1 thread (not sure the compression settings are the same, but at the very least there is some speedup for multithreading).

import os
import threading
import time
import lz4.block

size_mb = 2000
n_threads = 6

input_data = size_mb * 1024 * os.urandom(1024)
compressed = lz4.block.compress(input_data)
input_data = None


def decompress(data):
    start_time = time.perf_counter()
    thread_start_time = time.thread_time()
    lz4.block.decompress(data)
    stop_time = time.perf_counter()
    thread_stop_time = time.thread_time()
    print(f"{threading.current_thread()} Decompression took {(stop_time-start_time)*1000:.3f}ms "
          f"(Thread time: {(thread_stop_time-thread_start_time)*1000:.3f}ms)\n")


threads = [threading.Thread(name=str(i), target=decompress, args=(compressed,)) for i in range(n_threads)]

start_thread_time = time.perf_counter()
[t.start() for t in threads]
[t.join() for t in threads]
done_thread_time = time.perf_counter()
duration = done_thread_time - start_thread_time
MBs = n_threads*size_mb/duration
print(f"Total time: {duration}s : {MBs}MB/s")

Activity

  1. jonathanunderwood commented on Jan 9, 2023

    @jonathanunderwood
    Member

    Yes, a quick glanced at lz4framed, and wading through the macro definitions, the key difference is that lz4framed is calling
    PyThread_acquire_lock and PyThread_release_lock from pythread.h before/after calling the lz4 library functions, which we're not doing in this library at the moment. The main challenge is knowing when to release the lock - it's not consistently done in lz4framed.

    Shouldn't be too hard to do similarly here. Will have a look when I get time, unless you beat me to it.

  2. jonathanunderwood commented on Jan 9, 2023

    @jonathanunderwood
    Member

    For when I come back to this:

  3. jonathanunderwood commented on Dec 25, 2024

    @jonathanunderwood
    Member

    Looking at this again:

    • @Dalbasar is comparing lz4framed, which uses the LZ4 framing format, to the LZ4 block format which is an apples to oranges comparison.
    • lz4framed does add a lock to the context object, and makes sure to hold that lock when entering the lz4 C library. This will be important if multi-threading is being used to compress/decompress a single lz4 compressed object across multiple threads, but will not be necessary when each thread is (de)compressing separate objects, with each thread having its own context object. It is the latter case that is being tested here, and all that is needed for correct operation in this situation is for the python extension module to drop the GIL, which it does.

    So, not really clear what's going on here, nor what the fix is, but following lz4framed's approach doesn't seem particularly helpful (unless I have missed something).

  4. jonathanunderwood commented on Dec 30, 2024

    @jonathanunderwood
    Member

    Aside: the block bindings don't drop the GIL, and that'll be the root of the absence of speedup in the posted benchmark. It would be trivial to drop the GIL when entering the lz4 library for the block bindings just as we do for the frame bindings - I'll put that on the list.

    More interesting though is performance of the frame bindings in comparison to lz4framed - that's more of an apples to apples comparison.

  5. Dalbasar commented on Dec 31, 2024

    @Dalbasar
    Author

    Sounds good. It seems like the GIL is already dropped for compressing data in lz4 blocks, but not for decompressing. The compression speed is doubled for 2 threads.
    To me it looks the same in _block.c:
    LZ4 block Compression Allow Threads
    vs
    LZ4 block Decompression Allow Threads

    Or is there anything else that needs to be done to release the GIL properly for decompression?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions