Skip to content

ipc.rst

Latest commit

 

History

History
75 lines (58 loc) · 2.98 KB

ipc.rst

File metadata and controls

75 lines (58 loc) · 2.98 KB
.. default-domain:: cpp
.. cpp:namespace:: arrow::ipc

Reading and writing the Arrow IPC format

.. seealso::
   :ref:`Arrow IPC format specification <format-ipc>`.

   :doc:`API reference for IPC readers and writers <api/ipc>`.

Arrow C++ provides readers and writers for the Arrow IPC format which wrap lower level input/output, handled through the :doc:`IO interfaces <io>`. For reading, there is also an event-driven API that enables feeding arbitrary data into the IPC decoding layer asynchronously.

Reading IPC streams and files

Synchronous reading

For most cases, it is most convenient to use the :class:`RecordBatchStreamReader` or :class:`RecordBatchFileReader` class, depending on which variant of the IPC format you want to read. The former requires a :class:`~arrow::io::InputStream` source, while the latter requires a :class:`~arrow::io::RandomAccessFile`.

Reading Arrow IPC data is inherently zero-copy if the source allows it. For example, a :class:`~arrow::io::BufferReader` or :class:`~arrow::io::MemoryMappedFile` can typically be zero-copy. Exceptions are when the data must be transformed on the fly, e.g. when buffer compression has been enabled on the IPC stream or file.

Event-driven reading

When it is necessary to process the IPC format without blocking (for example to integrate Arrow with an event loop), or if data is coming from an unusual source, use the event-driven :class:`StreamDecoder`. You will need to define a subclass of :class:`Listener` and implement the virtual methods for the desired events (for example, implement :func:`Listener::OnRecordBatchDecoded` to be notified of each incoming :class:`RecordBatch`).

Writing IPC streams and files

Use one of the factory functions, :func:`MakeStreamWriter` or :func:`MakeFileWriter`, to obtain a :class:`RecordBatchWriter` instance for the given IPC format variant.

Configuring

Various aspects of reading and writing the IPC format can be configured using the :class:`IpcReadOptions` and :class:`IpcWriteOptions` classes, respectively.