Skip to content

Streams module does not support ring buffer #187

Open

Description

@jonathanunderwood

The streams module currently only supports the double buffer approach, and not the ring buffer approach. The scaffolding for supporting the double buffer approach is in the code, however. This just needs a willing volunteer to implement it.

The streams module was authored by @tSed who may be able to give guidance to a volunteer.

Activity

  1. fwarmuth commented on Apr 22, 2021

    @fwarmuth

    Hey,
    i believe need this feature...
    I try to reproduce a cpp application which calls LZ4_decompress_safe_continue() ->LZ4_decompress_safe() -> LZ4_decompress_generic().
    It seems that it uses the ring buffer approach.
    Can you @jonathanunderwood or @tSed give hints on what to do?
    I am willing to contribute somehow, nfortunately i never dealt with implementing python bindings. Until today i was only on the user-side of them 😕.

  2. tSed commented on May 15, 2021

    @tSed
    Contributor

    Hi @fwarmuth, @jonathanunderwood, all,

    Apologizes for the later answer :s

    My understanding of the lz4 API (from liblz4) is that it is not that aware of the stream compression strategy.

    At this level, the stream compression strategy is hidden behind the way the input buffers, output buffers, dictionaries and context are managed between each call to the {de,}compress functions.

    The examples explain this quite well, with schemes helping getting whole flow:

    Thus, the stream compression strategy is implemented in the binding wrapper.

    Most of the work will be understanding what should be done in each stream strategy helpers (here for ring buffer) WRT the _compress and _decompress functions.

    I think that most of the work regarding python binding implementation is already done.

    The big piece of this development will be getting what should be done (in term of buffer and context management) on each iteration.

  3. jonathanunderwood commented on Jun 20, 2021

    @jonathanunderwood
    MemberAuthor

    In the next release of the python-lz4 package I will be removing the streams functionality. The tests are too long running and the code too fragile, and migrating away from Travis to Github actions is throwing up too many problems. I'll reinstate these bindings when there is an active maintainer for them, but I do not have the time to fight with keeping these bindings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions