Skip to content

bpo-32728: Add compresslevel support for zipfile and LZMA #5534

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bbayles wants to merge 11 commits into from
Closed

bpo-32728: Add compresslevel support for zipfile and LZMA #5534

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
63a9350
Add compresslevel support for LZMA
bbayles Jan 30, 2018
4ca4ba6
Add NEWS entry
bbayles Feb 4, 2018
47d7dbc
Add docs entry for lzma compresslevel
bbayles Feb 4, 2018
c97397a
Merge branch 'master' into zipfile-lzma-presets
bbayles Sep 1, 2018
3218ba6
Merge remote-tracking branch 'upstream/master' into zipfile-lzma-presets
bbayles Aug 5, 2019
916be26
Merge remote-tracking branch 'upstream/master' into zipfile-lzma-presets
bbayles Aug 5, 2019
37b8945
Update Doc/library/zipfile.rst
bbayles Aug 5, 2019
3ed3f24
Update Misc/NEWS.d/next/Library/2018-02-04-13-18-14.bpo-32728.2u0pLO.rst
bbayles Aug 5, 2019
b7499b0
Adjust punctuation
bbayles Aug 5, 2019
978bf3f
Remove collections import
bbayles Aug 6, 2019
538e956
Prettify dict_size
bbayles Aug 6, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion Doc/library/zipfile.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,11 +172,14 @@ ZipFile Objects

The *compresslevel* parameter controls the compression level to use when
writing files to the archive.
When using :const:`ZIP_STORED` or :const:`ZIP_LZMA` it has no effect.
When using :const:`ZIP_STORED` it has no effect.
When using :const:`ZIP_DEFLATED` integers ``0`` through ``9`` are accepted
(see :class:`zlib <zlib.compressobj>` for more information).
When using :const:`ZIP_BZIP2` integers ``1`` through ``9`` are accepted
(see :class:`bz2 <bz2.BZ2File>` for more information).
When using :const:`ZIP_LZMA` integers ``0`` through ``9`` are accepted,
as are the values ``0 | lzma.PRESET_EXTREME`` through ``9 | lzma.PRESET_EXTREME``
(see :class:`lzma <lzma.LZMACompressor>` for more information).

The *strict_timestamps* argument, when set to ``False``, allows to
zip files older than 1980-01-01 at the cost of setting the
Expand Down Expand Up @@ -221,6 +224,9 @@ ZipFile Objects
.. versionadded:: 3.8
The *strict_timestamps* keyword-only argument

.. versionchanged:: 3.8

@epicfaace epicfaace Aug 27, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
.. versionchanged:: 3.8
.. versionchanged:: 3.9

Add support for using LZMA presets with the *compresslevel* parameter.


.. method:: ZipFile.close()

Expand Down
14 changes: 14 additions & 0 deletions Lib/test/test_zipfile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -633,6 +633,20 @@ class LzmaTestsWithSourceFile(AbstractTestsWithSourceFile,
unittest.TestCase):
compression = zipfile.ZIP_LZMA

def test_write_preset(self):
# Initialize an LZMA compressor with a preset and get the generated
# property header.
props_header = zipfile.LZMACompressor(preset=1).compress(b'')

# Write a ZIP archive with that LZMA compression preset and ensure
# that the property header above is written to that archive.
with io.BytesIO() as fp:
kwargs = {'compression': self.compression, 'compresslevel': 1}

@ZackerySpytz ZackerySpytz Aug 5, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's a need for this kwargs variable.

@bbayles bbayles Aug 5, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite right. I put it there to avoid a long line.

with zipfile.ZipFile(fp, 'w', **kwargs) as zipfp:
zipfp.writestr('dummy_file', b'dummy_contents')
written_bytes = fp.getvalue()
self.assertIn(props_header, written_bytes)


class AbstractTestZip64InSmallFiles:
# These tests test the ZIP64 functionality without using large files,
Expand Down
69 changes: 61 additions & 8 deletions Lib/zipfile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -596,15 +596,69 @@ def decrypter(data):


class LZMACompressor:

def __init__(self):
# LZMA uses "filter chains" to control compression settings (e.g., how
# much memory and CPU to use) in details. liblzma defines certain filter
# chain configurations as "preset levels," akin to the compression levels
# in zlib or bzip2.
_PRESET_OPTIONS_MAP = {
Comment thread
bbayles marked this conversation as resolved.
# Levels 0 through 9, without the "extreme" setting
0x00000000: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 18,
'mode': 1, 'mf': 3, 'nice_len': 128, 'depth': 4},
0x00000001: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 20,
'mode': 1, 'mf': 4, 'nice_len': 128, 'depth': 8},
0x00000002: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 21,
'mode': 1, 'mf': 4, 'nice_len': 273, 'depth': 24},
0x00000003: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 22,
'mode': 1, 'mf': 4, 'nice_len': 273, 'depth': 48},
0x00000004: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 22,
'mode': 2, 'mf': 20, 'nice_len': 16, 'depth': 0},
0x00000005: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 23,
'mode': 2, 'mf': 20, 'nice_len': 32, 'depth': 0},
0x00000006: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 23,
'mode': 2, 'mf': 20, 'nice_len': 64, 'depth': 0},
0x00000007: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 24,
'mode': 2, 'mf': 20, 'nice_len': 64, 'depth': 0},
0x00000008: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 25,
'mode': 2, 'mf': 20, 'nice_len': 64, 'depth': 0},
0x00000009: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 26,
'mode': 2, 'mf': 20, 'nice_len': 64, 'depth': 0},
# Levels 0 through 9, OR-ed with the "extreme" setting (PRESET_EXTREME)
0x80000000: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 18,
'mode': 2, 'mf': 20, 'nice_len': 273, 'depth': 512},
0x80000001: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 20,
'mode': 2, 'mf': 20, 'nice_len': 273, 'depth': 512},
0x80000002: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 21,
'mode': 2, 'mf': 20, 'nice_len': 273, 'depth': 512},
0x80000003: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 22,
'mode': 2, 'mf': 20, 'nice_len': 192, 'depth': 0},
0x80000004: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 22,
'mode': 2, 'mf': 20, 'nice_len': 273, 'depth': 512},
0x80000005: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 23,
'mode': 2, 'mf': 20, 'nice_len': 192, 'depth': 0},
0x80000006: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 23,
'mode': 2, 'mf': 20, 'nice_len': 273, 'depth': 512},
0x80000007: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 24,
'mode': 2, 'mf': 20, 'nice_len': 273, 'depth': 512},
0x80000008: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 25,
'mode': 2, 'mf': 20, 'nice_len': 273, 'depth': 512},
0x80000009: {'lc': 3, 'lp': 0, 'pb': 2, 'dict_size': 1 << 26,
'mode': 2, 'mf': 20, 'nice_len': 273, 'depth': 512},
}

def __init__(self, preset=None):
self._comp = None
if (preset is not None) and (preset not in self._PRESET_OPTIONS_MAP):

@ZackerySpytz ZackerySpytz Aug 5, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the parentheses here are needed.

@bbayles bbayles Aug 5, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed. My thought was that it helped with readability.

raise ValueError(f'invalid preset {preset}')
self._preset = preset

def _init(self):
props = lzma._encode_filter_properties({'id': lzma.FILTER_LZMA1})
self._comp = lzma.LZMACompressor(lzma.FORMAT_RAW, filters=[
lzma._decode_filter_properties(lzma.FILTER_LZMA1, props)
])
# Translate the encoder settings into the value to be written to
# the LZMA Properties Header in the ZIP file.
opts = {'id': lzma.FILTER_LZMA1}
if self._preset is not None:
opts.update(self._PRESET_OPTIONS_MAP[self._preset])
props = lzma._encode_filter_properties(opts)
self._comp = lzma.LZMACompressor(lzma.FORMAT_RAW, filters=[opts])
return struct.pack('<BBH', 9, 4, len(props)) + props

def compress(self, data):
Expand Down Expand Up @@ -694,9 +748,8 @@ def _get_compressor(compress_type, compresslevel=None):
if compresslevel is not None:
return bz2.BZ2Compressor(compresslevel)
return bz2.BZ2Compressor()
# compresslevel is ignored for ZIP_LZMA
elif compress_type == ZIP_LZMA:
return LZMACompressor()
return LZMACompressor(preset=compresslevel)
else:
return None

Expand Down
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Library/2018-02-04-13-18-14.bpo-32728.2u0pLO.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Archives created by the :mod:`zipfile` module can now use LZMA compression
presets.