You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
register_record_batches panics ("index out of bounds") on an empty partition #1626
Describe the bug
SessionContext.register_record_batches raises an unhandled Rust PanicException instead of a normal Python error when given a partition that contains no record batches — for example, the batches of an empty pyarrow.Table, whose to_batches() returns [].
The schema is read from the first batch with unchecked indexing (crates/core/src/context.rs):
With empty input, partitions.0 is [[]] — one partition holding zero batches — so [0][0] indexes out of bounds and panics across the Python/Rust boundary.
This is the unfixed sibling of #575 / #613, which hit the identical panic on from_arrow_table. That fix added an optional schema to create_dataframe and threads the pyarrow object's schema through, but register_record_batches takes no schema and was left unguarded.
Expected behavior
A catchable Python error, not a panic. An empty pyarrow.Table loses its schema through to_batches(), so the schema genuinely cannot be recovered at this point — a clear error is the right outcome (matching the graceful handling added for from_arrow_table in #613).
Additional context
The same unchecked partitions.0[0][0] also remains in create_dataframe's no-schema else branch, reachable when it is called directly without a schema. (Out of scope here; noted for completeness.)
Activity