Repository navigation
Description
Bug report
Bug description:
zlib._ZlibDecompressor.decompress(data, max_length) loses input when data is larger than UINT_MAX bytes and the output limit is reached in the same call. About 4 GiB of input is dropped without an error: eof stays False, needs_input becomes True, and the output comes out short. If the caller then feeds the rest of the stream, inflate continues at the wrong position, which shows up later as e.g. zlib.error: Error -3 while decompressing data: invalid stored block lengths.
Reproducer. It writes a ~4.3 GB file to the given directory (default: current); the data is memory-mapped, not loaded into RAM:
"""zlib._ZlibDecompressor drops input when one decompress() call gets more than
4 GiB of input together with max_length. Needs ~4.3 GB of free disk; the data
is memory-mapped, not loaded into RAM.
usage: python3 cpython_zlib_4gib.py [work dir]"""
import mmap, os, sys, zlib
SIZE = 4 * 2**30 + 256 * 2**20 # 4.25 GiB: just over UINT_MAX
path = os.path.join(sys.argv[1] if len(sys.argv) > 1 else ".", "zlib_4gib.deflate")
# A raw deflate stream of stored blocks, so the compressed input is > 4 GiB too.
c = zlib.compressobj(0, zlib.DEFLATED, -15)
block = bytes(64 << 20)
with open(path, "wb") as f:
left = SIZE
while left:
n = min(len(block), left)
f.write(c.compress(block[:n]))
left -= n
f.write(c.flush())
try:
with open(path, "rb") as f:
data = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
d = zlib._ZlibDecompressor(wbits=-15)
out = len(d.decompress(data, 65536)) # all input at once, output capped
while not d.eof and not d.needs_input: # drain what the decompressor kept
out += len(d.decompress(b"", 65536))
print(f"Python {sys.version.split()[0]}, zlib {zlib.ZLIB_RUNTIME_VERSION}")
print(f"input {len(data)} B; expected output {SIZE} B; got {out} B; "
f"lost {SIZE - out} B; eof={d.eof} needs_input={d.needs_input}")
finally:
os.remove(path)Output:
Python 3.13.9, zlib 1.3.1.zlib-ng
input 4563751257 B; expected output 4563402752 B; got 268828973 B; lost 4294573779 B; eof=False needs_input=True
Cause, in decompress_buf() in Modules/zlibmodule.c (main):
- Each pass of the outer loop calls
arrange_input_buffer(&self->zst, &self->avail_in_real), which setszst.avail_in = min(avail_in_real, UINT_MAX)and subtracts it fromavail_in_real. - When the output buffer is full (
obuflen == -2) the inner loop breaks, but the outer loop runs again whileavail_in_real != 0.arrange_input_buffer()then overwriteszst.avail_inwithout adding back the inputinflate()did not consume. - After the loop,
self->avail_in_real += self->zst.avail_in;adds back only the last chunk.
So the unconsumed part of the earlier UINT_MAX chunk drops out of the count, while next_in still points at it.
BZ2Decompressor in Modules/_bz2module.c has the same structure but adds bzs->avail_in back to bzs_avail_in_real after every BZ2_bzDecompress() call. zlib.Decompress computes the leftover from pointers in save_unconsumed_input(). Neither is affected. Doing what bz2 does (add zst.avail_in back to avail_in_real after each inflate()), or leaving the outer loop once the output limit is reached, should fix it.
How it came up: unblob's gzip reader kept passing more compressed input to the decompressor while it still had buffered input. A 12 GB .tar.gz firmware image hit this and was rejected as an invalid gzip. unblob has since fixed its reader (onekey-sec/unblob#1699). The gzip module itself is not affected, because it reads in small chunks and respects needs_input.
Seen on 3.13.9 (above) and 3.12.14 (same loss with a gzip member over 4 GiB). The same code is on main; I checked that by reading, not by running it.
Possibly related: #159999 (needs_input while output is still buffered), #158169.
CPython versions tested on:
3.12, 3.13
Operating systems tested on:
Linux
Metadata
Metadata
Assignees
Labels
Projects
- StatusShow more project fieldsNo status
Activity
aisk commentedon Nov 17, 2026
I can reproduce this on main (4497fe9) on macOS 15.7.7 (x86_64) with the system zlib, using the reproducer above:
pawangopallavudya-commits commentedon Nov 17, 2026
Hi, I’d be interested in investigating this zlib input-loss bug and, if I can confirm a fix with regression coverage, submitting a PR. I see it reproduces on main and haven’t found an open PR for it. I’ll follow CPython’s Developer Guide for the contribution and testing workflow.