0.40.0 closure: native Q5 kernels + pre-transposed default (closes #170, #184) - #307

Summary

Engine 0.40.0 is tagged and on Maven Central (q5_0_matmul.c / q5_1_matmul.c native kernels confirmed via git ls-tree -r 0.40.0, and sk.ainet.core:skainet-backend-native-cpu-jvm:0.40.0 is live on Central). This lands the step PR #294 explicitly deferred: flipping the gemma/llama converters to emit pre-transposed weights by default now that a native packed kernel is confirmed to exist.

• Bump skainet 0.39.1 → 0.40.0 in gradle/libs.versions.toml.
• GemmaQuantLayout.packGemmaKQuant / LlamaQuantLayout.packLlamaKQuant (commonMain, shared by the JVM MemSeg path and the Kotlin/Native board path) and GemmaMemSegConverter's JVM convertOne now call BlockQuantPacking.packPreTransposed instead of BlockQuantPacking.pack by default, wherever the existing hasPackedMatmulKernel() gate confirms a packed kernel exists (Q4_K/Q5_K/Q6_K/Q8_0 unconditionally — long-standing kernels; Q4_0/Q5_0/Q5_1 gated as before, unchanged gating logic). linearProject then dispatches ops.matmul(x, W) directly and skips its per-forward ops.transpose for these weights.
• Deprecate-don't-delete: both pack*Quant functions keep a preTransposed: Boolean = true parameter so the classic non-transposed BlockQuantPacking.pack path stays reachable for fallback / parity comparison. Nothing was deleted.
• GemmaQuantLayoutTest updated for the new default, plus explicit new coverage of the preTransposed = false fallback path.

Parity evidence (real checkpoint)

functiongemma-physical-ai-v10-Q5_K_M.gguf, 12g heap, -PincludeIntegration:

• GemmaQ5KPackedParityTest.q5kPackedMatchesFp32: greedy decode is byte/token-identical across the FP32 baseline, NATIVE_OPTIMIZED (JVM MemSeg, now pre-transposed), and load(NATIVE_OPTIMIZED) (K/N board path, also now pre-transposed) — all three produce the same 8-token generation [262146, 236769, 3255, 718, 498, 1373, 262152, 106].
• GemmaQ5xPackedParityTest.functionGemma_q5_1_tensors_stay_packed_after_conversion: all 81 Q5_1 tensors in the checkpoint stay packed post-conversion (none fall back to FP32 dequant).
• GemmaQ5xPackedParityTest's synthetic byte-level Q5_0/Q5_1 tests: pass (these exercise the classic non-pre-transposed path directly, unaffected by the default flip — see note below).

Speed: before vs after

Measured on :llm-inference:gemma's own jvmTest classpath, which only wires in skainet-backend-cpu (scalar + Panama Vector API kernels — this module does not depend on skainet-backend-native-cpu, unlike :llm-inference:llama):

That modest delta is the honest, CI-reproducible number for this module as shipped — it reflects only the transpose-wrapper-allocation removal, not the native kernel tier, because the native kernel provider was never on this module's classpath to begin with (confirmed by inspecting build.gradle.kts: only :llm-inference:llama wires skainet.backend.nativeCpu).

Diagnostic-only (not shipped): temporarily adding skainet.backend.nativeCpu to this module's jvmTest dependencies (mirroring llama) and re-running:

• Real-checkpoint decode: 4.39 tok/s (~53% faster than the 2.88 baseline), parity still held (same 8-token greedy decode, byte-identical).
• But GemmaQ5xPackedParityTest's two synthetic Q5_0/Q5_1 byte-level tests failed with all-zero output. Root-caused: those tests construct Q5_0BlockTensorData / Q5_1BlockTensorData directly and drive them through linearProject's classic lazy-ops.transpose branch — code this PR does not touch. This reproduces identically regardless of the pre-transposed default flip, so it's a pre-existing engine 0.40.0 issue (native Q5_0/Q5_1 kernel × lazy transpose dispatch), not something introduced or fixable in this repo.
• Notably, the pre-transposed path does not hit this bug — it skips lazy transpose entirely, and the real-checkpoint decode matched byte-for-byte with the native kernel active. So defaulting converters to pre-transposed isn't only a perf win here; it also routes around this gap.
• :llm-runtime:kgemma (the actual runnable app) already depends on skainet.backend.nativeCpp/jniCpu directly, so production Gemma inference gets both the real native-kernel speedup and automatically avoids the lazy-transpose gap via this PR's new default.
• Documented in llm-inference/gemma/build.gradle.kts (comment, not a dependency change) rather than silently worked around. Flagging for a separate engine-side bug report — happy to file if wanted.

Build

./gradlew build apiCheck green on transformer-core, llm-inference/gemma, llm-inference/llama. No public API surface changed (packGemmaKQuant/packLlamaKQuant are internal).

Test plan

• :transformer-core:jvmTest — BlockQuantPackingTest, LinearProjectionPreTransposedTest green
• :llm-inference:gemma:jvmTest — GemmaQuantLayoutTest green (incl. new pre-transposed-default + fallback coverage)
• :llm-inference:gemma:jvmTest -PincludeIntegration w/ GEMMA_GGUF — GemmaQ5KPackedParityTest, GemmaQ5xPackedParityTest green, parity confirmed
• ./gradlew build apiCheck green on touched modules

Closes #170, closes #184.

🤖 Generated with Claude Code