Skip to content

Avro: Cache the Avro to Iceberg schema conversion per schema string - #4384

Merged
Fokko merged 2 commits into
apache:mainfrom
zn-danielbariudin:avro-cache-schema-conversion
Nov 17, 2026
Merged

Fokko merged 2 commits into
apache:mainfrom
zn-danielbariudin:avro-cache-schema-conversion

Conversation

@zn-danielbariudin

Copy link
Copy Markdown
Contributor

Closes #3662

Rationale for this change

Every manifest in a table carries the same Avro schema string in its header, but AvroFileHeader.get_schema() parsed and converted that string to an Iceberg Schema on every file open. During scan planning this repeats the same conversion once per manifest.

This caches the conversion by schema string with functools.lru_cache, so each distinct schema is converted once per process.

  • Schema is a frozen model, so sharing the cached instance between readers is safe.
  • lru_cache keeps at most 128 schema strings; a process sees only a few distinct manifest schemas.

Builds on #3663 by @vishnuprakaz (picked up with his 👍 on the issue); he is credited as co-author on the commit.

Benchmark

plan_files() on an unpartitioned table with 150 manifests (local SqlCatalog, 7 runs in one process), Python 3.14.7, Apple M5 Pro:

Before After
Schema conversions in one plan_files() 151 1
plan_files() median ~56 ms ~30 ms
plan_files() first call ~96 ms ~67 ms
Benchmark script
import statistics
import sys
import tempfile
import time

import pyarrow as pa
from pyiceberg.avro.file import AvroSchemaConversion
from pyiceberg.catalog.sql import SqlCatalog

manifests = int(sys.argv[1])
repeats = 7

warehouse = tempfile.mkdtemp()
catalog = SqlCatalog("bench", uri=f"sqlite:///{warehouse}/catalog.db", warehouse=f"file://{warehouse}")
catalog.create_namespace("ns")
table = catalog.create_table("ns.t", schema=pa.schema([("id", pa.int64())]))
for i in range(manifests):
    table.append(pa.table({"id": pa.array([i], pa.int64())}))

calls = 0
original = AvroSchemaConversion.avro_to_iceberg


def counting(self, avro_schema):
    global calls
    calls += 1
    return original(self, avro_schema)


AvroSchemaConversion.avro_to_iceberg = counting

timings = []
for run in range(repeats):
    start = time.perf_counter()
    tasks = list(table.scan().plan_files())
    timings.append((time.perf_counter() - start) * 1000)
    if run == 0:
        first_run_calls = calls

print(f"manifests={manifests} data_files={len(tasks)}")
print(f"conversions: first plan_files={first_run_calls}, all {repeats} runs={calls}")
print(f"plan_files ms: first={timings[0]:.1f} median={statistics.median(timings):.1f} min={min(timings):.1f}")

Run with uv run python bench_plan_files.py 150.

Are these changes tested?

Yes. Added test_get_schema_reuses_conversion_for_identical_schema_strings in tests/avro/test_file.py: it checks the converted schema, that two headers with the same schema string return the same instance, and that a different string returns a different one. The full unit suite passes locally.

Are there any user-facing changes?

No API changes. Scan planning is faster on tables with many manifests.

@zn-danielbariudin

Copy link
Copy Markdown
Contributor Author

@Fokko @kevinjqliu could you take a look when you have a moment? Small change for apache/iceberg-python#3662: it caches the Avro schema conversion, and plan_files() on 150 manifests goes from ~56 ms to ~30 ms. Thanks

@Fokko

Fokko commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

I'm torn on this one. There is a reasonable speedup, but the tradeoff is on increased memory. Also, hashing the schema is not cheap. Can you maybe profile the memory?

@zn-danielbariudin

Copy link
Copy Markdown
Contributor Author

Hey @Fokko!
Fair concerns, so I profiled both on a table with 53 columns and 3 partition fields (day, identity, bucket), 150 manifests:

Result
Avro header reads / distinct schema strings 151 / 2
Schema string length 4,256 chars
Hash of a freshly decoded string 0.71 µs
Cache hit (hash + equality check) 0.90 µs
Conversion without the cache (json.loads + avro_to_iceberg) 244 µs
Memory retained per cache entry (Schema + key string) 47 KiB
plan_files() peak memory, without / with cache 2.92 / 2.95 MiB

Hashing: hashing the schema string does cost something, but ~0.9 µs per lookup vs ~244 µs for the conversion it replaces. The numbers use a fresh str per lookup, since each header read decodes a new string with no precomputed hash.

Memory: the manifest schema depends on the partition spec, not on the table's columns, so a scan only sees a couple of distinct strings. Peak memory of plan_files() is unchanged.

Some examples of what this means:

  • This run (150 manifests): without the cache, 151 × 244 µs ≈ 36.8 ms of conversion. With it, 2 conversions + 149 lookups ≈ 0.6 ms, for ~94 KiB retained.
  • 1,000 manifests: ≈ 244 ms → 1.4 ms, same ~94 KiB, since it's still 2 distinct strings.
  • A service reading 10 tables with different partition specs: ~11 distinct strings ≈ 0.5 MiB retained, regardless of how many scans it runs.
  • Worst case: bounded by the lru_cache default of 128 entries ≈ 5.9 MiB, which needs 128+ distinct manifest schemas in one process.

If you'd prefer a tighter bound, I'm happy to set an explicit maxsize, e.g. maxsize=32 caps it at ~1.5 MiB while still covering typical workloads. Let me know what you prefer.

@Fokko

Fokko commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

@zn-danielbariudin Thanks for running the numbers, that looks reasonable. If you could cap it at 32 for now, that would be great. In the case of wide schemas

@zn-danielbariudin

Copy link
Copy Markdown
Contributor Author

@Fokko
capped it at 32 in the latest commit

Thanks for the review! :)

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, let's move this forward. Thanks for the quick follow up @zn-danielbariudin

@Fokko
Fokko added this pull request to the merge queue Nov 17, 2026
Merged via the queue into apache:main with commit ab0afe7 Nov 17, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avro schema is re-converted to an Iceberg schema on every manifest read during scan planning

2 participants