@roeap Thanks, your patch fixed correctly passing through the 102: partition field 🥳

@Fokko This is awesome! thanks for your work on this one.
Do we expect any performance/memory differences with reading it with the rust module?

@yogevyuval Thanks for asking, and yes, I do expect performance impact since just a part of the deserialization is cythonized. With this change, much more is pushed into Rust. This will also reduce the GIL pressure since we don't have to build the readers anymore :)

I left this PR small to focus on the essentials, but once this is in, we can also clean up a LOT of code 👍

@yogevyuval Thanks for asking, and yes, I do expect performance impact since just a part of the deserialization is cythonized. With this change, much more is pushed into Rust. This will also reduce the GIL pressure since we don't have to build the readers anymore :)

I left this PR small to focus on the essentials, but once this is in, we can also clean up a LOT of code 👍

Interesting, we have quite a bit of concurrency going on so curious to see the results. Once apache/iceberg-rust#1328 gets merged and release i'll try and create some basic benchmarks and share the results

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.