Repository navigation
Avro: Cache the Avro to Iceberg schema conversion per schema string - #4384
Conversation
|
@Fokko @kevinjqliu could you take a look when you have a moment? Small change for apache/iceberg-python#3662: it caches the Avro schema conversion, and plan_files() on 150 manifests goes from ~56 ms to ~30 ms. Thanks |
|
I'm torn on this one. There is a reasonable speedup, but the tradeoff is on increased memory. Also, hashing the schema is not cheap. Can you maybe profile the memory? |
|
Hey @Fokko!
Hashing: hashing the schema string does cost something, but ~0.9 µs per lookup vs ~244 µs for the conversion it replaces. The numbers use a fresh Memory: the manifest schema depends on the partition spec, not on the table's columns, so a scan only sees a couple of distinct strings. Peak memory of Some examples of what this means:
If you'd prefer a tighter bound, I'm happy to set an explicit |
|
@zn-danielbariudin Thanks for running the numbers, that looks reasonable. If you could cap it at 32 for now, that would be great. In the case of wide schemas |
|
@Fokko Thanks for the review! :) |
Fokko
left a comment
There was a problem hiding this comment.
Thanks, let's move this forward. Thanks for the quick follow up @zn-danielbariudin
Closes #3662
Rationale for this change
Every manifest in a table carries the same Avro schema string in its header, but
AvroFileHeader.get_schema()parsed and converted that string to an IcebergSchemaon every file open. During scan planning this repeats the same conversion once per manifest.This caches the conversion by schema string with
functools.lru_cache, so each distinct schema is converted once per process.Schemais a frozen model, so sharing the cached instance between readers is safe.lru_cachekeeps at most 128 schema strings; a process sees only a few distinct manifest schemas.Builds on #3663 by @vishnuprakaz (picked up with his 👍 on the issue); he is credited as co-author on the commit.
Benchmark
plan_files()on an unpartitioned table with 150 manifests (localSqlCatalog, 7 runs in one process), Python 3.14.7, Apple M5 Pro:plan_files()plan_files()medianplan_files()first callBenchmark script
Run with
uv run python bench_plan_files.py 150.Are these changes tested?
Yes. Added
test_get_schema_reuses_conversion_for_identical_schema_stringsintests/avro/test_file.py: it checks the converted schema, that two headers with the same schema string return the same instance, and that a different string returns a different one. The full unit suite passes locally.Are there any user-facing changes?
No API changes. Scan planning is faster on tables with many manifests.