Do we need to block LargeListView together to be safe?

Also, I'm wondering if we need to block ArrowType.LargeList.INSTANCE in the same way to be safe.

Both are real gaps, and your "do we also need X" question pointed at the deeper problem: a blocklist of known-bad shapes can never be complete, and its failure direction is stream corruption. I flipped the check to an allowlist of the shapes toArrowField can actually produce (with exact parameter checks -- int widths, decimal bit width, timestamp/time/duration units), so LargeListView, LargeList, and everything else Spark never declares (unions, FixedSizeList, Decimal256, future types) are rejected by construction rather than by enumeration. The failure directions are asymmetric -- a wrongly rejected shape merely takes the row-based fallback (slower, still correct), a wrongly forwarded one corrupts the stream -- so the conservative polarity is the right one; this is now recorded in the doc comment. Added rejection tests for LargeList, ListView, LargeListView, and FixedSizeList.

Since isInterchangeShapedField is a hot-path which is invoked at all field tree every batch, shall we define Set(8, 16, 32, 64) somewhere and reuse this?

Hoisted to an object-level private val interchangeIntBitWidths with a comment noting why (per-field-tree per-batch call site).

Shall we add field.getDictionary == null because a dictionary-encoded vector's record batch carries indices and needs separate dictionary batches this stream never writes, so it must take the fallback even though its index type alone would pass the shape check above? Although Spark doesn't generate those data, we assume DSv2 data source, right?

Added as suggested. You're right that the DSv2 assumption makes this reachable: Spark never produces dictionary-encoded vectors, but an Arrow-backed connector may, and its record batches carry only indices -- the dictionary batches live in messages this stream never writes, so the field must take the fallback even when its type passes the shape check. Documented in the doc comment and covered by a test pairing a dictionary-encoded field (rejected) with the same field undictionaried (passes), isolating the dictionary as the reason.

[P1] Validate the complete declared struct shape before forwarding buffers.

This branch accepts any Struct as long as each child independently passes the allowlist. isGeometryField/isGeographyField (and isVariantField) recognize a tagged struct containing the required canonical two children even when an extra child follows, so ArrowColumnVector still reports the correct Spark logical type and its normal accessor reads the correct first two children. However, the UDF stream schema declares only those two children, while writeArrowDirect serializes the additional child's buffers too. An Arrow reader then silently consumes that extra child as the following selected argument: I reproduced an extra int32 child containing 7 followed by an actual int32 argument containing 123, and the argument decoded as 7 without raising an exception.

Could we compare the complete incoming field tree with the corresponding declared UDF field (including exact child count, ordering, and physical types), or reject noncanonical tagged Variant/Geometry/Geography structs before taking the direct path? A regression with a trailing scalar argument should catch the silent buffer shift.

Reproducing the silent shift settles it -- thank you. You're right that per-child validation cannot be sound when the recognizers tolerate extra children, and rather than special-casing the tagged structs I took your first suggestion in full: the check is now isCompatibleWithDeclaredField(actual, declared), a structural congruence walk (same Arrow type, same child count, congruent children, no dictionary) of each selected vector's field tree against the corresponding field of root.getSchema -- the exact header the stream already sent. That turns "body matches header" from an approximation (the allowlist) into the literal check, and subsumes every case discussed on this PR: the lossless structs, the var-width/largeVarTypes disagreement (the parameter disappears -- the declared side carries it), view/large-list types, dictionary encoding, and your extra-child tagged structs (child-count inequality). The regression test builds your exact scenario: a geometry-tagged struct padded with an extra int32 child, asserted to still read back as GeometryType via the recognizers but to be incongruent with the canonical declared field.

[P2] Do not treat timezone labels as a physical Arrow-layout mismatch.

This is reachable entirely within Spark: a scalar Arrow UDF returning TimestampType emits timestamp[us, tz=UTC], while a following Arrow-optimized regular Python UDF in an America/Los_Angeles session runs as a separate ArrowEvalPythonExec and declares timestamp[us, tz=America/Los_Angeles]. Both vectors represent Spark TimestampType and have identical int64 epoch-microsecond buffers, but ArrowType.Timestamp equality includes the timezone label, so this check rejects every batch and routes the complete input through writeRowByRow. That silently defeats the zero-copy optimization for a normal mixed-UDF pipeline.

ArrowFileReadWrite.checkLayoutMatch already handles the same case by comparing timestamp units while ignoring timezone labels; it also ignores Map.keysSorted because that flag does not affect the physical layout. Could we use equivalent physical-layout comparisons here, while preserving the timezone-present versus timezone-absent distinction between TimestampType and TimestampNTZType, and add a non-UTC mixed-UDF regression?

Fixed following the ArrowFileReadWrite.checkLayoutMatch precedent you pointed at: the type comparison now goes through a physical-layout equivalence -- timestamps compare unit and timezone presence-vs-absence (the label is ignored; it differs across Spark's own paths exactly as you described, while presence distinguishes TimestampType from TimestampNTZType, whose identical-looking values are interpreted differently), Map.keysSorted is ignored, and everything else stays strict equality. Unit tests pin all four directions (label mismatch accepted, presence mismatch rejected both ways, unit mismatch rejected since forwarding one unit under the other reinterprets values by a factor of 1000), and the requested regression runs your exact pipeline -- a pandas UDF feeding an Arrow-optimized Python UDF under an America/Los_Angeles session -- end to end.

[P2] Make the non-UTC regression exercise the columnar Arrow path

createDataFrame(...) is not cached here, so this input plans as RDDScanExec with supportsColumnar = false. Both chained ArrowEvalPythonExec operators therefore take super.doExecute() rather than doExecuteColumnar(), and ColumnarArrowPythonInput.isArrowBacked / ArrowUtils.isCompatibleWithDeclaredField never run. Consequently this test still passes if the timezone-layout fix is reverted, so it does not cover the mixed-UDF false-rejection regression it describes. Please cache and materialize the input (and unpersist it), and assert that the physical plan actually uses the columnar path.

Good catch -- and following it up uncovered two more layers, which reshaped the test.

You're right that the input planned as RDDScanExec, so neither eval node ever took the columnar path. But caching alone doesn't rescue the chained shape either: the two eval types plan as separate ArrowEvalPython nodes with a Project in between, and ProjectExec is row-based, so the second node never takes the columnar path -- the "worker output labeled UTC" pipeline I described cannot reach the gate through chaining at all today.

Digging further with an instrumented run: the cache cannot present a mismatched label either. ArrowCachedBatchToColumnarBatchIterator rebuilds its Arrow schema at read time from the query's session timezone, so a cache materialized under UTC and scanned under America/Los_Angeles hands the gate vectors already relabeled America/Los_Angeles -- actual and declared labels are equal by construction. As far as I can tell, no in-tree columnar producer can hit the timestamp-label clause today; it matters for external DSv2 Arrow sources (the audience of this columnar path), and its polarity stays pinned by the ArrowUtilsSuite cases.

Test changes accordingly: every test in the suite now runs against a cached, materialized (and unpersisted) input, and a shared helper walks the executed plan asserting the expected number of ArrowEvalPythonExec nodes and that each one supportsColumnar -- so the suite cannot silently stop exercising the columnar path again. The chained non-UTC test is replaced by a timestamp test that caches under UTC and runs the UDF under America/Los_Angeles, asserting the columnar plan and that the stored epoch round-trips across the timezone change.

One thing to be upfront about: even with the gate provably executing, reverting the layout fix would still not fail an e2e test -- a false reject falls back to the row path, which is correctness-preserving by design, so accept-vs-fallback is only observable at the unit level. If you think end-to-end observability is worth it, a small follow-up could add a pass-through batch count to the node's SQL metrics; happy to do that separately.

[P1] Preserve the semantic ordering of Arrow map-entry children

Ignoring child names here makes a valid Arrow Map<int,int> with entry children [value: int NOT NULL, key: int NOT NULL] pass against Spark's canonical [key, value] schema: the recursive positional pairs have identical Arrow types and child counts. Arrow 19 MapVector accepts that layout because it checks the two children and first-child nullability but does not validate their names/order, while Spark's ArrowColumnVector.MapAccessor reads key and value by name. I reproduced the full direct-IPC path with actual Arrow 19 vectors: the source map {123: 7} passed this predicate, but VectorUnloader plus the canonical stream header silently decoded it as {7: 123}. Please require canonical semantic key/value names at the corresponding map-entry positions (including nested maps), and add a swapped-entry regression; field names elsewhere can remain ignored.

Nice find -- confirmed and fixed. Map entries are indeed the one place where field names carry semantics rather than decoration: the Arrow spec binds key/value to the entry-struct children (key first, addressed as "key"/"value"), yet Arrow tolerates vectors with the children in the opposite order, and with matching key/value types the swapped tree is positionally indistinguishable from the canonical one.

isCompatibleWithDeclaredField now additionally requires, at every Map node, that the entry-struct children match the declared names positionally (mapEntryNamesMatchDeclared); the recursion covers nested maps. Names stay ignored everywhere else, as you suggested.

Regressions added at the unit level: your exact shape -- Map<int,int> with [value, key] entry children, where only the names reveal the swap -- is rejected, a nested map<int, map<int,int>> with swapped inner entries is rejected, and the canonical tree still passes. Negative-validated by removing the check and watching the test fail. Also added an e2e companion (test_udf_on_cached_map_column) pinning that canonical cached map vectors pass the gate and ride the columnar path.