Bug report

Bug description:

gzip.decompress(), bz2.decompress(), lzma.decompress() and compression.zstd.decompress() take time quadratic in the number of concatenated members (streams, frames) in their input. For every member a new decompressor is given all of the remaining input, and the rest is then copied back out through unused_data (and sliced twice more in gzip), so each member costs a copy of everything after it. Files made of many small members, such as concatenated logs or blocked gzip variants, decompress far slower than through GzipFile and the other file objects, which read in bounded chunks.

import bz2, gzip, lzma, time
from compression import zstd

for mod in gzip, bz2, lzma, zstd:
    member = mod.compress(b"")
    for n in 10_000, 20_000, 40_000:
        data = member * n
        t = time.perf_counter()
        assert mod.decompress(data) == b""
        print(f"{mod.__name__:17} {n:6} members {len(data):8} bytes "
              f"{time.perf_counter() - t:6.2f} s")

On a debug build of main, doubling the input takes about four times as long:

gzip               10000 members   200000 bytes   0.34 s
gzip               20000 members   400000 bytes   1.15 s
gzip               40000 members   800000 bytes   4.57 s
bz2                10000 members   140000 bytes   0.08 s
bz2                20000 members   280000 bytes   0.24 s
bz2                40000 members   560000 bytes   0.88 s
lzma               10000 members   320000 bytes   0.15 s
lzma               20000 members   640000 bytes   0.53 s
lzma               40000 members  1280000 bytes   2.21 s
compression.zstd   10000 members    90000 bytes   0.12 s
compression.zstd   20000 members   180000 bytes   0.35 s
compression.zstd   40000 members   360000 bytes   1.24 s

The gzip loop dates from the rewrite in GH-27941 (bpo-43613, 3.11); gh-90033 discussed its slicing cost. gh-155152 is the same pattern in zipimport._zstd_decompress().

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-160310: Fix quadratic one-shot decompress() with many members #160311
• [3.15] gh-160310: Fix quadratic one-shot decompress() with many members (GH-160311) #161037
• [3.14] gh-160310: Fix quadratic one-shot decompress() with many members (GH-160311) #161038
• gh-160310: Decompress a single stream directly in _decompress_all() #161737