Skip to content

Keep Unpacker.tell() at the start of an incomplete object - #772

Open
PDGGK wants to merge 1 commit into
msgpack:mainfrom
PDGGK:fix-unpacker-tell-incomplete-object
Open

PDGGK wants to merge 1 commit into
msgpack:mainfrom
PDGGK:fix-unpacker-tell-incomplete-object

Conversation

@PDGGK

@PDGGK PDGGK commented Nov 12, 2026

Copy link
Copy Markdown

Unpacker.tell() is documented as "the starting position of the next object". With the C extension that stops being true once the buffer ends inside an object. The C unpacker decodes the bytes it has, keeps the partial object for the next feed(), and counts those bytes in stream_offset. So tell() points into the middle of the next object.

This breaks the usual way to resume a stream: read what is there, iterate, save tell(), and continue from that offset later. A minimal case is packb([1, 2, 3]) + packb(["x" * 40, "y" * 40]), a 4-byte object followed by an 85-byte one, with only the first 49 bytes available:

unpacker = msgpack.Unpacker(io.BytesIO(data[:49]), read_size=16)
list(unpacker)    # [[1, 2, 3]]
unpacker.tell()   # 49 with the C extension, 4 with the pure Python one

Starting a new Unpacker at offset 49 decodes forty 121 integers from the middle of "y" * 40, with no error. The same happens with feed(). The pure Python implementation rolls back to the start of the incomplete object, so it returns 4.

The C unpacker now records stream_offset when it starts a new object, and tell() returns that value while ctx.execute shows an incomplete object. Nothing changes once the object is complete or after an error, and read_bytes() already refuses to run while an object is incomplete. The pure Python implementation is unchanged.

test_unpack_tell_incomplete_object in test/test_sequnpack.py cuts the stream at three points inside the second object, for both feed() and a file-like object. It checks that tell() is 4 after the first object and 89 once the rest has arrived. With only the tell() change reverted, it fails on the C extension with assert 5 == 4, and it passes on both implementations with the change. The full suite passes with the C extension (179 passed, 1 skipped) and with MSGPACK_PUREPYTHON=1 (176 passed, 4 skipped), and ruff check and ruff format report no changes. I ran this on CPython 3.13 on Linux only. I did not run the free-threaded build or the parallel-thread tests.

When the buffer ends inside an object, the C extension decodes the part it
has and keeps it for the next feed. tell() then included those bytes, so it
pointed into the middle of the next object instead of at its start. Code
that saves tell() to resume a stream later restarted mid-object and decoded
garbage. The pure Python implementation already returns the start.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant