feat(backend): NEON body for the Q4_0 matmul kernel (#920) - #939
Merged
Merged
Conversation
skainet_q4_0_matmul was the only priority quant format without a SIMD path — scalar C only, while q8_0/q4k/q5k/q6k all had NEON bodies. Q4_0 is named alongside Q8_0/Q4_K as a priority mobile format in the #920 plan, so it needs a NEON story before ARM ships as a perf target. The body unpacks the split-layout nibbles (low nibbles decode elements 0..15, high nibbles 16..31) with vand/vshr, re-centres by 8 in the signed int8 domain, widens to f32 and FMAs against the activation lanes — the same widen+vfmaq_f32 structure as the q8_0 body, because activations are FP32. Plain NEON only, no dotprod/i8mm requirement, so the path runs on every AArch64 core including armv8.0 (Cortex-A53). The loop is restructured to the block-outer/row-inner order q8_0 and q4k already use: the weight is block-major, so consecutive rows within a block are 18 bytes apart and weight reads become sequential. Per output row the blocks still accumulate in ascending order — results are numerically identical to the previous row-outer form. Also adds the missing Kotlin/Native Q4_0 parity test: the jvmTest side had one, but the aarch64 (qemu/board) lane had no Q4_0 coverage at all, so a NEON regression there would have been invisible. Verified: - jvmTest + linuxX64Test green (host archives, scalar/auto-vec path) - linuxArm64Test under qemu-aarch64 with the K/N-bundled aarch64 gcc 8.3 (-march=armv8.2-a+fp16+dotprod): 27/27 tests green, including the 4 new Q4_0 parity cases - objdump of the cross-built archive shows fmla vector FMAs inside skainet_q4_0_matmul — the NEON body executed, not the scalar fallback Refs #920
aharakal
approved these changes
Aug 10, 2026
This was referenced Aug 10, 2026
Merged
michalharakal
added a commit
to MacOS/SKaiNET
that referenced
this pull request
Aug 11, 2026
Bump VERSION_NAME 0.38.0 -> 0.39.0 and update all version-carrying docs: - CHANGELOG: consolidate [Unreleased] under [0.39.0] with a headline summary (includes the SKaiNET-developers#947 AAR-publishing CI entry, now merged to develop). - README: BOM snippet -> 0.39.0, What's New in 0.39.0, Contributors (0.39.0). - docs/antora.yml: skainet_version attribute 0.38.0 -> 0.39.0 (used in the docs' dependency snippets). - kernel-support-matrix.adoc: regenerated via generateKernelMatrix — the Android column now shows native-jni for Q8_0/Q4_0/Q4_K/Q5_K/Q6_K (the new JNI backend), replacing panama-vector; version stamp -> 0.39.0. 0.39.0 headline: on-device AI on Android becomes real — the skainet-backend-jni-cpu JNI NEON backend (~24 tok/s SmolLM2-135M Q8_0 on a Pixel 8a vs ~3.8 scalar), plus Android streaming GGUF loads (SKaiNET-developers#922), linkable K/N kernel klibs (SKaiNET-developers#942), the Q4_0 NEON kernel (SKaiNET-developers#939), GGUF loader fail-fast (SKaiNET-developers#919), and a tensor-storage correctness pass (SKaiNET-developers#927-SKaiNET-developers#931). Local prep only — not pushed/tagged. Cut off develop after SKaiNET-developers#947 merged, so the branch already carries the AAR-publishing workflow + pinned NDK.
michalharakal
added a commit
to MacOS/SKaiNET
that referenced
this pull request
Aug 11, 2026
…ative, JNI (SKaiNET-developers#708) 0.39.0 shipped packed GGUF loading for Q5_0/Q5_1 plus the scalar and Panama kernels, but the priority-100 native tier carried no Q5_x kernels: the JVM cascaded to Panama, and Kotlin/Native and Android ran the formats on the priority-0 scalar floor. Mirrors the Q4_0 pattern (SKaiNET-developers#920/SKaiNET-developers#939): - q5_0_matmul.c / q5_1_matmul.c: block-outer/row-inner C kernels with a plain-NEON body (no dotprod/i8mm — runs on every AArch64 core). The qh high-bit plane expands via a per-lane vtstq_u8 bit test onto the split nibbles; dequant folds algebraically (Q5_0: d*(dot-16*Σx), Q5_1: d*dot+m*Σx) so the per-block input sum hoists out of the output-row loop and the codes widen unsigned through skainet_neon_u8x16_to_f32x4x4. - FFM wrappers NativeQ5_0/Q5_1MatmulKernel + NativeKernelProvider wiring. - Kotlin/Native NativeKnQ5_0/Q5_1MatmulKernel + NativeKnKernelProvider wiring (cinterop bindings regenerate from skainet_kernels.h). - JNI bridge: q50Matmul/q51Matmul shims + JniKernels externals + JniKernelProvider accessors, sources added to both CMake lists. - Parity tests vs the scalar references on all three lanes (jvmTest, nativeTest, androidTest) + KernelSupportMatrixTest tier update. Verified: jvmTest 8/8+8/8 (FFM vs scalar), linuxX64Test 4/4+4/4 (cinterop vs scalar), JNI arm64-v8a cross-build exports both symbols with fmla/cmtst in the disassembly (NEON body compiled in). The qemu-aarch64 NEON parity lane (-PcrossArm64=true) runs in CI as for Q4_0. Unblocks the packed Q5_1 converter path for functiongemma-270m "Q5_K_M" checkpoints under NATIVE_OPTIMIZED — SKaiNET-transformers#170. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First implementation slice of #920 (native kernels for mobile):
skainet_q4_0_matmulwas the only priority quant format without a SIMD path — scalar C only, while q8_0/q4k/q5k/q6k all had NEON bodies.The kernel
Split-layout nibble unpack (
vand/vshr— low nibbles decode elements 0..15, high nibbles 16..31), re-centre by 8 in the signed int8 domain, widen to f32 and FMA against the activation lanes — the same widen+vfmaq_f32structure as the q8_0 body (activations are FP32; an int8 dotprod would need int8 activations). Plain NEON, no dotprod/i8mm requirement — the path runs on every AArch64 core including armv8.0 (Cortex-A53), which matters for the Android runtime-dispatch story in the #920 plan.The loop is also restructured to the block-outer/row-inner order q8_0/q4k already use (sequential 18-byte-strided weight reads for the in-order-core case). Per output row, blocks still accumulate in ascending order — numerically identical to the previous row-outer form.
Test-coverage gap closed
The jvmTest side had a Q4_0 FFM parity test, but the Kotlin/Native lane (qemu/board) had no Q4_0 coverage at all — a NEON regression there would have been invisible. New
NativeKnQ4_0MatmulKernelParityTestmirrors the Q8_0 one (4 shapes up to 4096×64, full-range random nibbles exercising both lanes and the re-centring).Verified
jvmTest+linuxX64Testgreen (host archives — scalar/auto-vectorized path, loop-restructure regression)linuxArm64Test -PcrossArm64=trueunder qemu-aarch64 with the K/N-bundled aarch64 gcc 8.3 (-march=armv8.2-a+fp16+dotprod): 27/27 green, including the 4 new Q4_0 parity casesobjdumpof the cross-builtlibskainet_kernels.ashowsfmlavector FMAs insideskainet_q4_0_matmul— the NEON body executed under qemu, not the scalar fallback (same verification standard as the AARCH64-VERIFIED banner)skainet_simd.hbanner updated to reflect the new coverageWe can run the same parity binary on physical aarch64 hardware (as done for the original NEON verification) on request.
Refs #920