You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The streams module currently only supports the double buffer approach, and not the ring buffer approach. The scaffolding for supporting the double buffer approach is in the code, however. This just needs a willing volunteer to implement it.
The streams module was authored by @tSed who may be able to give guidance to a volunteer.
Hey,
i believe need this feature...
I try to reproduce a cpp application which calls LZ4_decompress_safe_continue() ->LZ4_decompress_safe() -> LZ4_decompress_generic().
It seems that it uses the ring buffer approach.
Can you @jonathanunderwood or @tSed give hints on what to do?
I am willing to contribute somehow, nfortunately i never dealt with implementing python bindings. Until today i was only on the user-side of them 😕.
My understanding of the lz4 API (from liblz4) is that it is not that aware of the stream compression strategy.
At this level, the stream compression strategy is hidden behind the way the input buffers, output buffers, dictionaries and context are managed between each call to the {de,}compress functions.
The examples explain this quite well, with schemes helping getting whole flow:
In the next release of the python-lz4 package I will be removing the streams functionality. The tests are too long running and the code too fragile, and migrating away from Travis to Github actions is throwing up too many problems. I'll reinstate these bindings when there is an active maintainer for them, but I do not have the time to fight with keeping these bindings.
Activity
fwarmuth commentedon Apr 22, 2021
Hey,
i believe need this feature...
I try to reproduce a cpp application which calls
LZ4_decompress_safe_continue()->LZ4_decompress_safe()->LZ4_decompress_generic().It seems that it uses the ring buffer approach.
Can you @jonathanunderwood or @tSed give hints on what to do?
I am willing to contribute somehow, nfortunately i never dealt with implementing python bindings. Until today i was only on the user-side of them 😕.
tSed commentedon May 15, 2021
Hi @fwarmuth, @jonathanunderwood, all,
Apologizes for the later answer :s
My understanding of the lz4 API (from liblz4) is that it is not that aware of the stream compression strategy.
At this level, the stream compression strategy is hidden behind the way the input buffers, output buffers, dictionaries and context are managed between each call to the {de,}compress functions.
The examples explain this quite well, with schemes helping getting whole flow:
Thus, the stream compression strategy is implemented in the binding wrapper.
Most of the work will be understanding what should be done in each stream strategy helpers (here for ring buffer) WRT the
_compressand_decompressfunctions.I think that most of the work regarding python binding implementation is already done.
The big piece of this development will be getting what should be done (in term of buffer and context management) on each iteration.
jonathanunderwood commentedon Jun 20, 2021
In the next release of the python-lz4 package I will be removing the streams functionality. The tests are too long running and the code too fragile, and migrating away from Travis to Github actions is throwing up too many problems. I'll reinstate these bindings when there is an active maintainer for them, but I do not have the time to fight with keeping these bindings.