Skip to content

zlib._ZlibDecompressor.decompress() drops input when given more than 4 GiB with max_length #162153

Open

Description

@dommyrock

Bug report

Bug description:

zlib._ZlibDecompressor.decompress(data, max_length) loses input when data is larger than UINT_MAX bytes and the output limit is reached in the same call. About 4 GiB of input is dropped without an error: eof stays False, needs_input becomes True, and the output comes out short. If the caller then feeds the rest of the stream, inflate continues at the wrong position, which shows up later as e.g. zlib.error: Error -3 while decompressing data: invalid stored block lengths.

Reproducer. It writes a ~4.3 GB file to the given directory (default: current); the data is memory-mapped, not loaded into RAM:

"""zlib._ZlibDecompressor drops input when one decompress() call gets more than
4 GiB of input together with max_length. Needs ~4.3 GB of free disk; the data
is memory-mapped, not loaded into RAM.
usage: python3 cpython_zlib_4gib.py [work dir]"""
import mmap, os, sys, zlib

SIZE = 4 * 2**30 + 256 * 2**20  # 4.25 GiB: just over UINT_MAX
path = os.path.join(sys.argv[1] if len(sys.argv) > 1 else ".", "zlib_4gib.deflate")

# A raw deflate stream of stored blocks, so the compressed input is > 4 GiB too.
c = zlib.compressobj(0, zlib.DEFLATED, -15)
block = bytes(64 << 20)
with open(path, "wb") as f:
    left = SIZE
    while left:
        n = min(len(block), left)
        f.write(c.compress(block[:n]))
        left -= n
    f.write(c.flush())

try:
    with open(path, "rb") as f:
        data = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
    d = zlib._ZlibDecompressor(wbits=-15)
    out = len(d.decompress(data, 65536))   # all input at once, output capped
    while not d.eof and not d.needs_input:  # drain what the decompressor kept
        out += len(d.decompress(b"", 65536))
    print(f"Python {sys.version.split()[0]}, zlib {zlib.ZLIB_RUNTIME_VERSION}")
    print(f"input {len(data)} B; expected output {SIZE} B; got {out} B; "
          f"lost {SIZE - out} B; eof={d.eof} needs_input={d.needs_input}")
finally:
    os.remove(path)

Output:

Python 3.13.9, zlib 1.3.1.zlib-ng
input 4563751257 B; expected output 4563402752 B; got 268828973 B; lost 4294573779 B; eof=False needs_input=True

Cause, in decompress_buf() in Modules/zlibmodule.c (main):

  • Each pass of the outer loop calls arrange_input_buffer(&self->zst, &self->avail_in_real), which sets zst.avail_in = min(avail_in_real, UINT_MAX) and subtracts it from avail_in_real.
  • When the output buffer is full (obuflen == -2) the inner loop breaks, but the outer loop runs again while avail_in_real != 0. arrange_input_buffer() then overwrites zst.avail_in without adding back the input inflate() did not consume.
  • After the loop, self->avail_in_real += self->zst.avail_in; adds back only the last chunk.

So the unconsumed part of the earlier UINT_MAX chunk drops out of the count, while next_in still points at it.

BZ2Decompressor in Modules/_bz2module.c has the same structure but adds bzs->avail_in back to bzs_avail_in_real after every BZ2_bzDecompress() call. zlib.Decompress computes the leftover from pointers in save_unconsumed_input(). Neither is affected. Doing what bz2 does (add zst.avail_in back to avail_in_real after each inflate()), or leaving the outer loop once the output limit is reached, should fix it.

How it came up: unblob's gzip reader kept passing more compressed input to the decompressor while it still had buffered input. A 12 GB .tar.gz firmware image hit this and was rejected as an invalid gzip. unblob has since fixed its reader (onekey-sec/unblob#1699). The gzip module itself is not affected, because it reads in small chunks and respects needs_input.

Seen on 3.13.9 (above) and 3.12.14 (same loss with a gzip member over 4 GiB). The same code is on main; I checked that by reading, not by running it.

Possibly related: #159999 (needs_input while output is still buffered), #158169.

CPython versions tested on:

3.12, 3.13

Operating systems tested on:

Linux

Activity

  1. added
    type-bugAn unexpected behavior, bug, or error
    on Nov 17, 2026
  2. aisk commented on Nov 17, 2026

    @aisk
    Member

    I can reproduce this on main (4497fe9) on macOS 15.7.7 (x86_64) with the system zlib, using the reproducer above:

    Python 3.16.0a2+dev, zlib 1.2.12
    input 4563751257 B; expected output 4563402752 B; got 268828973 B; lost 4294573779 B; eof=False needs_input=True
    
  3. pawangopallavudya-commits commented on Nov 17, 2026

    @pawangopallavudya-commits

    Hi, I’d be interested in investigating this zlib input-loss bug and, if I can confirm a fix with regression coverage, submitting a PR. I see it reproduces on main and haven’t found an open PR for it. I’ll follow CPython’s Developer Guide for the contribution and testing workflow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    extension-modulesC modules in the Modules dirtype-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions