Skip to content

Using Compressed with a separate length field #1123

@marksteward

Description

@marksteward

I've hit a few issues trying to get Compressed to work, which makes me think it's not very well used or supported. It was called out in #355 but there aren't any examples in the gallery.

  1. Compressed pretty much only makes sense when used with GreedyBytes for its subcon. The documentation instead focuses on the input, saying it should be used with Prefixed, but not why.

    The reason for GreedyBytes is that this:

    Compressed(Bytes(10), 'zlib')

    will read an unlimited amount of compressed data, and then return the first 10 bytes. While this:

    FixedSized(10, Compressed(GreedyBytes, 'zlib'))

    will read 10 bytes, and decompress it into the right size. I think FixedSized should be included in the documentation, along with an explanation that Prefixed is suitable when creating your own structure. Perhaps GreedyBytes should be the default subcon?

  2. While the above is the only way it could work when looking closely, I was misled by the signature for the preceding Checksum, and assumed the subcon was the output type. I used the first example, and as the compressed output was always larger than the input, there was no error. I didn't realise I'd got it wrong until I started fully parsing the output. I feel like there's some footgun-avoiding that could be added here, perhaps warning if the output isn't consumed, or changing the interface to match Checksum, or vice versa.

  3. For the format I'm looking at, I have a compressed field with a separate length field, and while I can now parse it, I want to be able to populate the compressed length when building. I basically want Prefixed, but where the prefix is in a different location. If I use FixedSized(this.compressed_size, Compressed(GreedyBytes, 'zlib')), I need to somehow provide a value for compressed_size. I can't work out how to do that without doing the compression a second time myself. I can pull it out of the ctx if the field follows the data, but Rebuild doesn't seem to work if it's before.

    Am I trying to make Compressed do something it's just not suited for? Do I need to create an adapter instead? Is FixedSized the wrong thing to use?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions