You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As far as i understand, the GIL should be dropped when calling the underlying LZ4 C library during compression and decompression, see lz4.block.decompress
I only see a minor speedup when using multiple threads for decompression using python-lz4 4.3.2 on a 6 core Intel i5-8400 on Debian 11 and also not on a Amd Ryzen 5900X on Windows 10, neither with Python 3.11.1 nor 3.8.10.
Compression speed seems to increase almost linearly with the number of threads.
The following code gives me about 4500MB/s decompression speed (slight underestimation due to some overhead from starting the threads) when using 6 threads and ~4300MB/s when using 1 thread on an AMD Ryzen 5900X on Windows 10. Using lz4.frame yields similar results. Using py-lz4framed instead gives me about 13000MB/s using 6 threads and ~8300MB/s on 1 thread (not sure the compression settings are the same, but at the very least there is some speedup for multithreading).
Yes, a quick glanced at lz4framed, and wading through the macro definitions, the key difference is that lz4framed is calling PyThread_acquire_lock and PyThread_release_lock from pythread.h before/after calling the lz4 library functions, which we're not doing in this library at the moment. The main challenge is knowing when to release the lock - it's not consistently done in lz4framed.
Shouldn't be too hard to do similarly here. Will have a look when I get time, unless you beat me to it.
@Dalbasar is comparing lz4framed, which uses the LZ4 framing format, to the LZ4 block format which is an apples to oranges comparison.
lz4framed does add a lock to the context object, and makes sure to hold that lock when entering the lz4 C library. This will be important if multi-threading is being used to compress/decompress a single lz4 compressed object across multiple threads, but will not be necessary when each thread is (de)compressing separate objects, with each thread having its own context object. It is the latter case that is being tested here, and all that is needed for correct operation in this situation is for the python extension module to drop the GIL, which it does.
So, not really clear what's going on here, nor what the fix is, but following lz4framed's approach doesn't seem particularly helpful (unless I have missed something).
Aside: the block bindings don't drop the GIL, and that'll be the root of the absence of speedup in the posted benchmark. It would be trivial to drop the GIL when entering the lz4 library for the block bindings just as we do for the frame bindings - I'll put that on the list.
More interesting though is performance of the frame bindings in comparison to lz4framed - that's more of an apples to apples comparison.
Activity
jonathanunderwood commentedon Jan 9, 2023
Yes, a quick glanced at lz4framed, and wading through the macro definitions, the key difference is that lz4framed is calling
PyThread_acquire_lockandPyThread_release_lockfrompythread.hbefore/after calling the lz4 library functions, which we're not doing in this library at the moment. The main challenge is knowing when to release the lock - it's not consistently done in lz4framed.Shouldn't be too hard to do similarly here. Will have a look when I get time, unless you beat me to it.
jonathanunderwood commentedon Jan 9, 2023
For when I come back to this:
jonathanunderwood commentedon Dec 25, 2024
Looking at this again:
So, not really clear what's going on here, nor what the fix is, but following lz4framed's approach doesn't seem particularly helpful (unless I have missed something).
jonathanunderwood commentedon Dec 30, 2024
Aside: the block bindings don't drop the GIL, and that'll be the root of the absence of speedup in the posted benchmark. It would be trivial to drop the GIL when entering the lz4 library for the block bindings just as we do for the frame bindings - I'll put that on the list.
More interesting though is performance of the frame bindings in comparison to lz4framed - that's more of an apples to apples comparison.
Dalbasar commentedon Dec 31, 2024
Sounds good. It seems like the GIL is already dropped for compressing data in lz4 blocks, but not for decompressing. The compression speed is doubled for 2 threads.
To me it looks the same in _block.c:
LZ4 block Compression Allow Threads
vs
LZ4 block Decompression Allow Threads
Or is there anything else that needs to be done to release the GIL properly for decompression?