Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,13 @@ python:
- 3.4
- 3.5
- 3.6
install: pip install tox
install:
- pip install tox
- pip install -r docs/requirements.txt
script:
- tox

- python setup.py install
- make -C docs doctest html
deploy:
- provider: pypi
# server: https://test.pypi.org/legacy/
Expand Down
5 changes: 4 additions & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,10 @@ The API provided by the frame format bindings follows that of the LZMA, zlib,
gzip and bzip2 compression libraries which are provided with the Python standard
library. As such, these LZ4 bindings should provide a drop-in alternative to the
compression libraries shipped with Python. The package provides context managers
and file handlers support.
and file handler support.

The bindings drop the GIL when calling in to the underlying LZ4 library, and is
thread safe. An extensive test suite is included.

Documenation
============
Expand Down
9 changes: 9 additions & 0 deletions docs/intro.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,12 @@ standard container for the compressed data. In the frame format, the data is
compressed into a sequence of blocks. The frame format defines a frame header,
which contains information about the compressed data such as its size, and
defines a standard end of frame marker.

The API provided by the frame format bindings follows that of the LZMA, zlib,
gzip and bzip2 compression libraries which are provided with the Python standard
library. As such, these LZ4 bindings should provide a drop-in alternative to the
compression libraries shipped with Python. The package provides context managers
and file handler support.

The bindings drop the GIL when calling in to the underlying LZ4 library, and is
thread safe. An extensive test suite is included.
11 changes: 8 additions & 3 deletions docs/lz4.block.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,16 @@ Example usage
-------------
To use the lz4 block format bindings is straightforward::

.. doctest::

>>> import lz4.block
>>> compressed_data = lz4.block.compress(data)
>>> data == lz4.block.decompress(compressed_data)
>>> import os
>>> input_data = 20 * 128 * os.urandom(1024) # Read 20 * 128kb
>>> compressed_data = lz4.block.compress(input_data)
>>> output_data = lz4.block.decompress(compressed_data)
>>> input_data == output_data
True
>>>


Contents
----------------
Expand Down
24 changes: 18 additions & 6 deletions docs/quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,15 @@ provides interoperability with other implementations and language bindings.
The simplest way to use the frame bindings is via the :py:func:`compress` and
:py:func:`decompress` functions::

.. doctest::

>>> import os
>>> import lz4.frame
>>> input_data = 20 * 128 * os.urandom(1024) # Read 20 * 128kb
>>> compressed = lz4.frame.compress(input_data)
>>> decompressed = lz4.frame.decompress(compressed)
>>> decompressed == input_data
Out[6]: True
True

The :py:func:`compress` function reads the input data and compresses it and
returns a LZ4 frame. A frame consists of a header, and a sequence of blocks of
Expand All @@ -34,14 +36,16 @@ Working with data in chunks
It's often inconvenient to hold the full data in memory, and so functions are
also provided to compress and decompress data in chunks::

.. doctest::

>>> import lz4.frame
>>> import os
>>> input_data = 20 * 128 * os.urandom(1024)
>>> c_context = lz4.frame.create_compression_context()
>>> compressed = lz4.frame.compress_begin(c_context)
>>> compressed += lz4.frame.compress_chunk(c_context, input_data[:10 * 128 * 1024])
>>> compressed += lz4.frame.compress_chunk(c_context, input_data[10 * 128 * 1024:])
>>> compressed += compress_flush(c_context)
>>> compressed += lz4.frame.compress_flush(c_context)

Here a compression context is first created which is used to maintain state
across calls to the LZ4 library. This is an opaque PyCapsule object.
Expand All @@ -59,11 +63,13 @@ time without ending the frame by calling :py:func:`compress_flush` with

Decompressing data can also be done in a chunked fashion::

.. doctest::

>>> d_context = lz4.frame.create_decompression_context()
>>> d1, b, e = lz4.frame.decompress_chunk(d_context, compressed[:len(compressed)//2])
>>> d2, b, e = lz4.frame.decompress_chunk(d_context, compressed[len(compressed)//2:])
>>> d1 + d2 == input_data
Out[12]: True
True

Note that :py:func:`decompress_chunk` returns a tuple ``(decompressed_data,
bytes_read, end_of_frame_indicator)``. ``decompressed_data`` is the decompressed
Expand All @@ -78,19 +84,21 @@ is more convenient to use the :py:class:`LZ4FrameCompressor` and
:py:class:`LZ4FrameDecompressor` classes which provide context manager
functionality::

.. doctest::

>>> import lz4.frame
>>> import os
>>> input_data = 20 * 128 * os.urandom(1024)
>>> with lz4.frame.LZ4FrameCompressor() as compressor:
... compressed = compressor.begin()
... compressed += compressor.compress(input_data[:10 * 128 * 1024])
... compressed += compressor.compress(input_data[10 * 128 * 1024:])
... compressed += compressor.finalize()
... compressed += compressor.flush()
>>> with lz4.frame.LZ4FrameDecompressor() as decompressor:
... decompressed = decompressor.decompress(compressed[:len(compressed)//2])
... decompressed += decompressor.decompress(compressed[len(compressed)//2:])
>>> decompressed == input_data
Out[13]: True
True


Working with compressed files
Expand All @@ -102,11 +110,15 @@ replacement for that offered in the Python standard library for bz2, gzip and
LZMA compressed files. The :py:func:`lz4.frame.open()` function is the most
convenient way to work with compressed data files::

.. doctest::

>>> import lz4.frame
>>> import os
>>> input_data = 20 * os.urandom(1024)
>>> with lz4.frame.open('testfile', mode='wb') as fp:
... fp.write(input_data)
... bytes_written = fp.write(input_data)
... bytes_written == len(input_data)
True
>>> with lz4.frame.open('testfile', mode='r') as fp:
... output_data = fp.read()
>>> output_data == input_data
Expand Down
1 change: 1 addition & 0 deletions docs/static/.keep
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# This file created to allow us to check an otherwise empty directory in to git
2 changes: 1 addition & 1 deletion lz4/frame/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ def finalize(self):
`LZ4FrameCompressor.flush()`.

"""
result = flush()
result = self.flush()
return result

def reset(self):
Expand Down
4 changes: 4 additions & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
tox
pytest
pytest-runner
setuptools_scm
deprecation
pkgconfig
future