feat(quant): packed Q5_0/Q5_1 converter path + shared block packer & pre-transpose marker (#170, #184 (2)(3)) - #294

Closes #170 (transformers side); implements #184 items (2) and (3). Pairs with the engine PR SKaiNET-developers/SKaiNET#951 (native Q5_0/Q5_1 kernels, closes engine #708) for the 0.40.0 lock-step train — but everything here runs and is verified against the published engine 0.39.0 (the default build).

What

1. Packed Q5_1 / Q5_0 converter path (#170)

GemmaMemSegConverter.convertOne gains Q5_1 and Q5_0 cases that keep the weights packed (Q5_1BlockTensorData / Q5_0BlockTensorData after the row-major → block-major relayout) instead of the #169 dequant-to-FP32 fallback. FunctionGemma-270M "Q5_K_M" ships 81 of 236 tensors as Q5_1 (attn_q / attn_k / ffn_gate / ffn_up) — these now run the in-kernel dequant matmul like Q4_K/Q5_K/Q6_K/Q8_0 already did.

Gated on kernel availability, not engine version: a new GGMLQuantizationType.hasPackedMatmulKernel() (llm-core) asks the engine's KernelRegistry whether a registered provider actually carries an FP32 × packed-Q5_x matmul kernel (installing ServiceLoader providers first on JVM, mirroring the engine's own lazy wiring). If not, the converter takes the #169 FP32-dequant fallback — still correct, never garbage. The same gate covers the board path (packGemmaKQuant) and llama (packLlamaKQuant) for all three "legacy" 32-elem formats (Q4_0/Q5_0/Q5_1); the four long-packed formats (Q4_K/Q5_K/Q6_K/Q8_0) behave exactly as before.

2. Shared GGUF-block packer (#184 (2))

The GGUF-block → engine-*BlockTensorData packing that gemma, llama and apertus each carried privately (relayout + wrap) is hoisted to sk.ainet.lang.nn.quant.BlockQuantPacking (transformer-core, commonMain, lang-core-only deps), keyed by the engine's TensorEncoding and covering all seven packed-kernel formats. sk.ainet.apps.llm.weights.toBlockEncoding() (llm-core) is the ggml-keyed front door. Model modules keep only weight selection + naming. The old per-model relayoutKSeriesRowMajorToBlockMajor entry points remain as @Deprecated internal shims (deprecate-don't-delete); all production call sites now use the shared packer.

3. Pre-transpose marker (#184 (3), "Solution C")

• sk.ainet.lang.nn.quant.PreTransposedWeight — marker interface: tensor data whose logical shape is already the transposed [in, out].
• BlockQuantPacking.packPreTransposed(...) — packs a weight with the logical shape already swapped (same block-major bytes the engine's lazy packed ops.transpose would produce — zero copy) and attaches the marker via internal delegating views that still satisfy every engine dispatch check (is Q*TensorData, PackedBlockStorage).
• linearProject (transformer-core) reads the marker and skips ops.transpose, dispatching ops.matmul(x, W) straight into the packed kernel path. Unmarked weights keep the classic transpose path (engine lazy packed transpose remains the fallback).

Converters do not yet emit pre-transposed weights by default — that flip ("enable-packed-default") is deliberately left for after the engine 0.40.0 train lands (see below), with the marker infrastructure fully tested now.

Verification (default build = published engine 0.39.0)

• :transformer-core:jvmTest — LinearProjectionPreTransposedTest: synthetic Q5_1 weight, marked pre-transposed path is bit-identical to the classic packed path and matches the analytic FP32 reference; marker also proven on plain FP32 data.
• :transformer-core:linuxX64Test — BlockQuantPackingTest (all seven encodings: geometry vs physicalBytes, relayout block-transpose, pack round-trip, pre-transposed shape-swap + marker) green on Kotlin/Native too.
• :llm-inference:gemma:jvmTest — GemmaQ5xPackedParityTest:
  • synthetic byte-level parity for both Q5_1 and Q5_0: converter's packed pipeline vs converter's FP32-dequant pipeline on a multi-block [3, 64] weight (the checkpoint has no Q5_0 tensor — Q5_0 parity is synthetic-only, by necessity);
  • real checkpoint (functiongemma-physical-ai-v10-Q5_K_M.gguf): all 81 Q5_1 tensors are Q5_1BlockTensorData after conversion — none FP32-inflated.
• GemmaQ5KPackedParityTest (existing, -PincludeIntegration): full-model greedy decode of NATIVE_OPTIMIZED — which now packs Q5_1 too — token-for-token equal to the DEQUANTIZE_TO_FP32 baseline, on both the MemSeg converter path and the wired load(NATIVE_OPTIMIZED) board path: all three decode [262146, 236769, 3255, 718, 498, 1373, 262152, 106] → <tool_0>(state="on")<end><end_of_turn> for the "Turn the light on." prompt; packed throughput on this host 2.20 tok/s incl. prefill (vs the ~0.67 tok/s Packed SIMD matmul kernel for Q5_1 / Q5_0 (keep NATIVE_OPTIMIZED fast, not dequant-to-FP32) #170 reported for the Q5_1 dequant-fallback build).
• jvmTest suites of llm-core / llama / apertus / gemma + linuxX64Test of llm-core / gemma / llama: green.
• apiCheck green; llm-core.api / transformer-core.api gain additions only (toBlockEncoding, hasPackedMatmulKernel, BlockQuantPacking, PreTransposedWeight) — no removals, no signature changes.

What lights up with engine 0.40.0 / SKaiNET#951

Under 0.39.0 the Q5_x packed path dispatches to the scalar/Panama kernels (priority ≤ 50). #951 adds the priority-100 native tier (FFM on JVM, Kotlin/Native cinterop, Android JNI) for Q5_0/Q5_1. Because the converter gates on kernel availability and the dispatch resolves by provider priority, bumping the engine pin to 0.40.0 requires zero changes in this repo: the same weights automatically ride the NEON-bodied native kernels, closing the native/Android performance gap for Q5_1-heavy checkpoints (e.g. FunctionGemma's remaining ~0.67→packed tok/s gap from #170).

Remaining steps to close #170 / #184

1. Engine 0.40.0 release with #951 merged (native Q5_0/Q5_1 kernel tiers).
2. Bump skainet = "0.40.0" in gradle/libs.versions.toml (one line).
3. "Enable-packed-default": flip gemma/llama converters to BlockQuantPacking.packPreTransposed + re-run the real-checkpoint parity suite, completing Hoist quant packing + RowDequantSource out of sk.ainet.models.gemma into shared layers #184 (3) end-to-end (the marker + linearProject support land here, already tested).
4. Hoist quant packing + RowDequantSource out of sk.ainet.models.gemma into shared layers #184 (1) (RowDequantSource → engine ops.gather) was completed by feat(gemma): engine-backed row-dequant token_embd — re-home RowDequantSource, keep JVM eager embed packed #289; maintainer may close Hoist quant packing + RowDequantSource out of sk.ainet.models.gemma into shared layers #184 once (2)/(3) merge and the default flip lands.

🤖 Generated with Claude Code

Co-Authored-By: Claude Fable 5 noreply@anthropic.com