Split out of #1595 (report 8 of 9) — filed by @rrodriguesNutrium, credit to them. Described as the widest-impact finding.
database_embedded_kuzu.py:743-757 raises a fabricated exception specifically to force the per-row fallback path:
# Force loop fallback for relationship writes inside UNWIND to avoid Kuzu query planner bugs
if (re.search(r"UNWIND\s+\$\w+\s+AS\s+\w+", query) ... ):
So every batched relationship write becomes one query per row. A stack sample of a stalled full-repo run showed the main thread almost entirely in kuzu::binder::Expression, planner::LogicalPlan, Schema::copy and PhysicalOperator::initLocalState — query planning, not scanning, consistent with tens of thousands of tiny queries.
At scale (2,861 Kotlin files, ~374k extrapolated queries) a full index ran over 105 minutes without completing, at 1.3-1.8 GB RSS. Serialization was 44 of 47 seconds on a smaller run.
Amplifier, with a fix already measured. write_inheritance_links (writer.py:972) iterates a 12-element label tuple as both child and parent — 144 combinations — issuing a query per pair whether or not any row could match. On a Kotlin repo only 5 of those 12 tables are populated, so ~83% of pairs are pure waste. Skipping empty labels via a cheap existence probe took that writer from 2,748 calls to 562 (-80%) and the whole index from 63.9s to 49.7s (-22%), with INHERITS edge count identical at 120 before and after. The reporter has this ready to open as a small PR.
Two directions for the underlying problem, neither attempted:
- Re-test whether the
unordered_map::at planner bug still reproduces on current Kùzu — the workaround may outlive its cause, and the guard is narrowly scoped enough (UNWIND \$param AS row only) to make an opt-in escape hatch cheap.
- Chunk the fallback rather than going fully per-row — even 50-row sub-batches would cut planning overhead by an order of magnitude if the planner bug is size-dependent.
Note #1302 proposes deprecating KùzuDB as a runtime backend, which would change the calculus here.
Split out of #1595 (report 8 of 9) — filed by @rrodriguesNutrium, credit to them. Described as the widest-impact finding.
database_embedded_kuzu.py:743-757raises a fabricated exception specifically to force the per-row fallback path:So every batched relationship write becomes one query per row. A stack sample of a stalled full-repo run showed the main thread almost entirely in
kuzu::binder::Expression,planner::LogicalPlan,Schema::copyandPhysicalOperator::initLocalState— query planning, not scanning, consistent with tens of thousands of tiny queries.At scale (2,861 Kotlin files, ~374k extrapolated queries) a full index ran over 105 minutes without completing, at 1.3-1.8 GB RSS. Serialization was 44 of 47 seconds on a smaller run.
Amplifier, with a fix already measured.
write_inheritance_links(writer.py:972) iterates a 12-element label tuple as both child and parent — 144 combinations — issuing a query per pair whether or not any row could match. On a Kotlin repo only 5 of those 12 tables are populated, so ~83% of pairs are pure waste. Skipping empty labels via a cheap existence probe took that writer from 2,748 calls to 562 (-80%) and the whole index from 63.9s to 49.7s (-22%), withINHERITSedge count identical at 120 before and after. The reporter has this ready to open as a small PR.Two directions for the underlying problem, neither attempted:
unordered_map::atplanner bug still reproduces on current Kùzu — the workaround may outlive its cause, and the guard is narrowly scoped enough (UNWIND \$param AS rowonly) to make an opt-in escape hatch cheap.Note #1302 proposes deprecating KùzuDB as a runtime backend, which would change the calculus here.