Skip to content

Commit b804443

Browse files
jimmodpgeorge
jimmo
authored and
dpgeorge
committed
docs/library/deflate: Add docs for deflate.DeflateIO.
Also update zlib & gzip docs to describe the micropython-lib modules. This work was funded through GitHub Sponsors. Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
1 parent 8b315ef commit b804443

4 files changed

Lines changed: 357 additions & 28 deletions

File tree

‎docs/library/deflate.rst‎

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
:mod:`deflate` -- deflate compression & decompression
2+
=====================================================
3+
4+
.. module:: deflate
5+
:synopsis: deflate compression & decompression
6+
7+
This module allows compression and decompression of binary data with the
8+
`DEFLATE algorithm <https://en.wikipedia.org/wiki/DEFLATE>`_
9+
(commonly used in the zlib library and gzip archiver).
10+
11+
**Availability:**
12+
13+
* Added in MicroPython v1.21.
14+
15+
* Decompression: Enabled via the ``MICROPY_PY_DEFLATE`` build option, on by default
16+
on ports with the "extra features" level or higher (which is most boards).
17+
18+
* Compression: Enabled via the ``MICROPY_PY_DEFLATE_COMPRESS`` build option, on
19+
by default on ports with the "full features" level or higher (generally this means
20+
you need to build your own firmware to enable this).
21+
22+
Classes
23+
-------
24+
25+
.. class:: DeflateIO(stream, format=AUTO, wbits=0, close=False, /)
26+
27+
This class can be used to wrap a *stream* which is any
28+
:term:`stream-like <stream>` object such as a file, socket, or stream
29+
(including :class:`io.BytesIO`). It is itself a stream and implements the
30+
standard read/readinto/write/close methods.
31+
32+
The *stream* must be a blocking stream. Non-blocking streams are currently
33+
not supported.
34+
35+
The *format* can be set to any of the constants defined below, and defaults
36+
to ``AUTO`` which for decompressing will auto-detect gzip or zlib streams,
37+
and for compressing it will generate a raw stream.
38+
39+
The *wbits* parameter sets the base-2 logarithm of the DEFLATE dictionary
40+
window size. So for example, setting *wbits* to ``10`` sets the window size
41+
to 1024 bytes. Valid values are ``5`` to ``15`` inclusive (corresponding to
42+
window sizes of 32 to 32k bytes).
43+
44+
If *wbits* is set to ``0`` (the default), then a window size of 256 bytes
45+
will be used (corresponding to *wbits* set to ``8``), except when
46+
:ref:`decompressing a zlib stream <deflate_wbits_zlib>`.
47+
48+
See the :ref:`window size <deflate_wbits>` notes below for more information
49+
about the window size, zlib, and gzip streams.
50+
51+
If *close* is set to ``True`` then the underlying stream will be closed
52+
automatically when the :class:`deflate.DeflateIO` stream is closed. This is
53+
useful if you want to return a :class:`deflate.DeflateIO` stream that wraps
54+
another stream and not have the caller need to know about managing the
55+
underlying stream.
56+
57+
If compression is enabled, a given :class:`deflate.DeflateIO` instance
58+
supports both reading and writing. For example, a bidirectional stream like
59+
a socket can be wrapped, which allows for compression/decompression in both
60+
directions.
61+
62+
Constants
63+
---------
64+
65+
.. data:: deflate.AUTO
66+
deflate.RAW
67+
deflate.ZLIB
68+
deflate.GZIP
69+
70+
Supported values for the *format* parameter.
71+
72+
Examples
73+
--------
74+
75+
A typical use case for :class:`deflate.DeflateIO` is to read or write a compressed
76+
file from storage:
77+
78+
.. code:: python
79+
80+
import deflate
81+
82+
# Writing a zlib-compressed stream (uses the default window size of 256 bytes).
83+
with open("data.gz", "wb") as f:
84+
with deflate.DeflateIO(f, deflate.ZLIB) as d:
85+
# Use d.write(...) etc
86+
87+
# Reading a zlib-compressed stream (auto-detect window size).
88+
with open("data.z", "rb") as f:
89+
with deflate.DeflateIO(f, deflate.ZLIB) as d:
90+
# Use d.read(), d.readinto(), etc.
91+
92+
Because :class:`deflate.DeflateIO` is a stream, it can be used for example
93+
with :meth:`json.dump` and :meth:`json.load` (and any other places streams can
94+
be used):
95+
96+
.. code:: python
97+
98+
import deflate, json
99+
100+
# Write a dictionary as JSON in gzip format, with a
101+
# small (64 byte) window size.
102+
config = { ... }
103+
with open("config.gz", "wb") as f:
104+
with deflate.DeflateIO(f, deflate.GZIP, 6) as f:
105+
json.dump(config, f)
106+
107+
# Read back that dictionary.
108+
with open("config.gz", "rb") as f:
109+
with deflate.DeflateIO(f, deflate.GZIP, 6) as f:
110+
config = json.load(f)
111+
112+
If your source data is not in a stream format, you can use :class:`io.BytesIO`
113+
to turn it into a stream suitable for use with :class:`deflate.DeflateIO`:
114+
115+
.. code:: python
116+
117+
import deflate, io
118+
119+
# Decompress a bytes/bytearray value.
120+
compressed_data = get_data_z()
121+
with deflate.DeflateIO(io.BytesIO(compressed_data), deflate.ZLIB) as d:
122+
decompressed_data = d.read()
123+
124+
# Compress a bytes/bytearray value.
125+
uncompressed_data = get_data()
126+
stream = io.BytesIO()
127+
with deflate.DeflateIO(stream, deflate.ZLIB) as d:
128+
d.write(uncompressed_data)
129+
compressed_data = stream.getvalue()
130+
131+
.. _deflate_wbits:
132+
133+
Deflate window size
134+
-------------------
135+
136+
The window size limits how far back in the stream the (de)compressor can
137+
reference. Increasing the window size will improve compression, but will
138+
require more memory.
139+
140+
However, just because a given window size is used for compression, this does not
141+
mean that the stream will require the same size window for decompression, as
142+
the stream may not reference data as far back as the window allows (for example,
143+
if the length of the input is smaller than the window size).
144+
145+
If the decompressor uses a smaller window size than necessary for the input data
146+
stream, it will fail mid-way through decompression with :exc:`OSError`.
147+
148+
.. _deflate_wbits_zlib:
149+
150+
The zlib format includes a header which specifies the window size used to
151+
compress the data (which due to the above, may be larger than the size required
152+
for the decompressor).
153+
154+
If this header value is lower than the specified *wbits* value, then the header
155+
value will be used instead in order to reduce the memory allocation size. If
156+
the *wbits* parameter is zero (the default), then the header value will only be
157+
used if it is less than the maximum value of ``15`` (which is default value
158+
used by most compressors [#f1]_).
159+
160+
In other words, if the source zlib stream has been compressed with a custom window
161+
size (i.e. less than ``15``), then using the default *wbits* parameter of zero
162+
will decompress any such stream.
163+
164+
The gzip file format does not include the window size in the header.
165+
Additionally, most compressor libraries (including CPython's implementation
166+
of :class:`gzip.GzipFile`) will default to the maximum possible window size.
167+
This makes it difficult to decompress most gzip streams on MicroPython unless
168+
your board has a lot of free RAM.
169+
170+
If you control the source of the compressed data, then prefer to use the zlib
171+
format, with a window size that is suitable for your target device.
172+
173+
.. rubric:: Footnotes
174+
175+
.. [#f1] The assumption here is that if the header value is the default used by
176+
most compressors, then nothing is known about the likely required window
177+
size and we should ignore it.

‎docs/library/gzip.rst‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
:mod:`gzip` -- gzip compression & decompression
2+
===============================================
3+
4+
.. module:: gzip
5+
:synopsis: gzip compression & decompression
6+
7+
|see_cpython_module| :mod:`python:gzip`.
8+
9+
This module allows compression and decompression of binary data with the
10+
`DEFLATE algorithm <https://en.wikipedia.org/wiki/DEFLATE>`_ used by the gzip
11+
file format.
12+
13+
.. note:: Prefer to use :class:`deflate.DeflateIO` instead of the functions in this
14+
module as it provides a streaming interface to compression and decompression
15+
which is convenient and more memory efficient when working with reading or
16+
writing compressed data to a file, socket, or stream.
17+
18+
**Availability:**
19+
20+
* This module is **not present by default** in official MicroPython firmware
21+
releases as it duplicates functionality available in the :mod:`deflate
22+
<deflate>` module.
23+
24+
* A copy of this module can be installed (or frozen)
25+
from :term:`micropython-lib` (`source <https://github.com/micropython/micropython-lib/blob/master/python-stdlib/gzip/gzip.py>`_).
26+
See :ref:`packages` for more information. This documentation describes that module.
27+
28+
* Compression support will only be available if compression support is enabled
29+
in the built-in :mod:`deflate <deflate>` module.
30+
31+
Functions
32+
---------
33+
34+
.. function:: open(filename, mode, /)
35+
36+
Wrapper around built-in :func:`open` returning a GzipFile instance.
37+
38+
.. function:: decompress(data, /)
39+
40+
Decompresses *data* into a bytes object.
41+
42+
.. function:: compress(data, /)
43+
44+
Compresses *data* into a bytes object.
45+
46+
Classes
47+
-------
48+
49+
.. class:: GzipFile(*, fileobj, mode)
50+
51+
This class can be used to wrap a *fileobj* which is any
52+
:term:`stream-like <stream>` object such as a file, socket, or stream
53+
(including :class:`io.BytesIO`). It is itself a stream and implements the
54+
standard read/readinto/write/close methods.
55+
56+
When the *mode* argument is ``"rb"``, reads from the GzipFile instance will
57+
decompress the data in the underlying stream and return decompressed data.
58+
59+
If compression support is enabled then the *mode* argument can be set to
60+
``"wb"``, and writes to the GzipFile instance will be compressed and written
61+
to the underlying stream.
62+
63+
By default the GzipFile class will read and write data using the gzip file
64+
format, including a header and footer with checksum and a window size of 512
65+
bytes.
66+
67+
The **file**, **compresslevel**, and **mtime** arguments are not
68+
supported. **fileobj** and **mode** must always be specified as keyword
69+
arguments.
70+
71+
Examples
72+
--------
73+
74+
A typical use case for :class:`gzip.GzipFile` is to read or write a compressed
75+
file from storage:
76+
77+
.. code:: python
78+
79+
import gzip
80+
81+
# Reading:
82+
with open("data.gz", "rb") as f:
83+
with gzip.GzipFile(fileobj=f, mode="rb") as g:
84+
# Use g.read(), g.readinto(), etc.
85+
86+
# Same, but using gzip.open:
87+
with gzip.open("data.gz", "rb") as f:
88+
# Use f.read(), f.readinto(), etc.
89+
90+
# Writing:
91+
with open("data.gz", "wb") as f:
92+
with gzip.GzipFile(fileobj=f, mode="wb") as g:
93+
# Use g.write(...) etc
94+
95+
# Same, but using gzip.open:
96+
with gzip.open("data.gz", "wb") as f:
97+
# Use f.write(...) etc
98+
99+
# Write a dictionary as JSON in gzip format, with a
100+
# small (64 byte) window size.
101+
config = { ... }
102+
with gzip.open("config.gz", "wb") as f:
103+
json.dump(config, f)
104+
105+
For guidance on working with gzip sources and choosing the window size see the
106+
note at the :ref:`end of the deflate documentation <deflate_wbits>`.

‎docs/library/index.rst‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@ library.
6464
collections.rst
6565
errno.rst
6666
gc.rst
67+
gzip.rst
6768
hashlib.rst
6869
heapq.rst
6970
io.rst
@@ -95,6 +96,7 @@ the following libraries.
9596
bluetooth.rst
9697
btree.rst
9798
cryptolib.rst
99+
deflate.rst
98100
framebuf.rst
99101
machine.rst
100102
micropython.rst
@@ -194,11 +196,11 @@ Extending built-in libraries from Python
194196
A subset of the built-in modules are able to be extended by Python code by
195197
providing a module of the same name in the filesystem. This extensibility
196198
applies to the following Python standard library modules which are built-in to
197-
the firmware: ``array``, ``binascii``, ``collections``, ``errno``, ``hashlib``,
198-
``heapq``, ``io``, ``json``, ``os``, ``platform``, ``random``, ``re``,
199-
``select``, ``socket``, ``ssl``, ``struct``, ``time`` ``zlib``, as well as the
200-
MicroPython-specific ``machine`` module. All other built-in modules cannot be
201-
extended from the filesystem.
199+
the firmware: ``array``, ``binascii``, ``collections``, ``errno``, ``gzip``,
200+
``hashlib``, ``heapq``, ``io``, ``json``, ``os``, ``platform``, ``random``,
201+
``re``, ``select``, ``socket``, ``ssl``, ``struct``, ``time`` ``zlib``, as well
202+
as the MicroPython-specific ``machine`` module. All other built-in modules
203+
cannot be extended from the filesystem.
202204

203205
This allows the user to provide an extended implementation of a built-in library
204206
(perhaps to provide additional CPython compatibility or missing functionality).

0 commit comments

Comments
 (0)