feat(backend): native Q5_0/Q5_1 packed matmul kernels — FFM, Kotlin/Native, JNI (#708) - #951
Merged
Merged
Conversation
…ative, JNI (#708) 0.39.0 shipped packed GGUF loading for Q5_0/Q5_1 plus the scalar and Panama kernels, but the priority-100 native tier carried no Q5_x kernels: the JVM cascaded to Panama, and Kotlin/Native and Android ran the formats on the priority-0 scalar floor. Mirrors the Q4_0 pattern (#920/#939): - q5_0_matmul.c / q5_1_matmul.c: block-outer/row-inner C kernels with a plain-NEON body (no dotprod/i8mm — runs on every AArch64 core). The qh high-bit plane expands via a per-lane vtstq_u8 bit test onto the split nibbles; dequant folds algebraically (Q5_0: d*(dot-16*Σx), Q5_1: d*dot+m*Σx) so the per-block input sum hoists out of the output-row loop and the codes widen unsigned through skainet_neon_u8x16_to_f32x4x4. - FFM wrappers NativeQ5_0/Q5_1MatmulKernel + NativeKernelProvider wiring. - Kotlin/Native NativeKnQ5_0/Q5_1MatmulKernel + NativeKnKernelProvider wiring (cinterop bindings regenerate from skainet_kernels.h). - JNI bridge: q50Matmul/q51Matmul shims + JniKernels externals + JniKernelProvider accessors, sources added to both CMake lists. - Parity tests vs the scalar references on all three lanes (jvmTest, nativeTest, androidTest) + KernelSupportMatrixTest tier update. Verified: jvmTest 8/8+8/8 (FFM vs scalar), linuxX64Test 4/4+4/4 (cinterop vs scalar), JNI arm64-v8a cross-build exports both symbols with fmla/cmtst in the disassembly (NEON body compiled in). The qemu-aarch64 NEON parity lane (-PcrossArm64=true) runs in CI as for Q4_0. Unblocks the packed Q5_1 converter path for functiongemma-270m "Q5_K_M" checkpoints under NATIVE_OPTIMIZED — SKaiNET-transformers#170. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aharakal
previously approved these changes
Aug 11, 2026
aharakal
approved these changes
Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #708. Engine half of SKaiNET-developers/SKaiNET-transformers#170.
What
0.39.0 ships packed GGUF loading for Q5_0/Q5_1 plus the scalar + Panama kernels, but the priority-100 native tier had no Q5_x kernels: the JVM cascaded to Panama (50), while Kotlin/Native and Android ran these formats on the priority-0 scalar floor. This PR completes the native tier for both formats, mirroring the Q4_0 pattern (#920/#939):
q5_0_matmul.c/q5_1_matmul.c— block-outer/row-inner C kernels with a plain-NEON body (no dotprod/i8mm requirement, so it runs on every AArch64 core). Theqhhigh-bit plane is expanded with a per-lanevtstq_u8bit test onto the split nibbles; the dequant folds algebraically (Q5_0:d*(dot − 16·Σx), Q5_1:d*dot + m·Σx) so the per-block input sum hoists out of the output-row loop and the unsigned codes widen throughskainet_neon_u8x16_to_f32x4x4.NativeQ5_0MatmulKernel/NativeQ5_1MatmulKernel+NativeKernelProviderwiring.NativeKnQ5_0MatmulKernel/NativeKnQ5_1MatmulKernel+NativeKnKernelProviderwiring (cinterop bindings regenerate fromskainet_kernels.h).q50Matmul/q51Matmulshims inskainet_jni.c,JniKernelsexternals,JniKernelProvideraccessors; sources added to both CMake lists so the AAR's two.sotiers carry them.KernelSupportMatrixTesttier declaration update.Verification
:skainet-backend-native-cpu:jvmTest—NativeQ5_0MatmulKernelParityTest8/8,NativeQ5_1MatmulKernelParityTest8/8 (FFM vs scalar reference; whole suite green).:skainet-backend-native-cpu:linuxX64Test—NativeKnQ5_0/Q5_1MatmulKernelParityTest4/4 each (cinterop vs scalar; whole suite green).:skainet-backend-jni-cpu:assembleRelease(NDK 28.2) — arm64-v8a.sos exportJava_…_q50Matmul/q51Matmul+skainet_q5_0/…_q5_1_matmul;llvm-objdumpshowsfmla/cmtstinskainet_q5_1_matmul, i.e. the NEON body compiled in (not the scalar fallback).:skainet-backend-jni-cpu:test(host) 4/4;:skainet-backend-cpu:jvmTest+apiCheckgreen (no public-API change in the BCV-gated module).linuxArm64Test -PcrossArm64=truelane (same one that validated the Q4_0 NEON body) covers it in CI, and the on-deviceJniKernelParityTestgainsq50_parity/q51_parity.Follow-up
GemmaMemSegConverter) lands separately (transformers#170); it already works against 0.39.0 via the Panama/scalar cascade — this PR closes the native/Android performance gap once released.🤖 Generated with Claude Code
Co-Authored-By: Claude Fable 5 noreply@anthropic.com