Why it matters
KllamaDemo is the first thing a new SKaiNET LLM user copies, and right now it
steers people onto the slowest and most memory-hungry path on exactly the devices
that can least afford it. We believed the comments, built our first mobile version
on DEQUANTIZE_TO_FP32, and got 1 tok/s plus an OOM — then tried
NATIVE_OPTIMIZED anyway and got 21 tok/s and a ~4x smaller footprint (145 MB vs
540 MB resident for SmolLM2-135M Q8_0), including on Kotlin/Native, where the
fallback was never triggered.
The stale claims
composeApp/src/commonMain/kotlin/sk/ainet/apps/kllama/chat/playground/QwenRuntimeBuilder.kt:19-22:
On wasmJs / iOS / Android we fall back to [buildQwenRuntimeFallback] …
Same correctness, much slower throughput — the SIMD/MemSeg kernels are JVM-only.
and :31-36 ("until upstream lands a wasmJs/Native SIMD backend"), and
QwenRuntimeBuilder.android.kt:7-9 ("the JVM fast path
(DecoderGgufMemSegConverter) isn't available. Fall back…").
The premise changed in SKaiNET-transformers 0.30.0 (Gemma packed path moved to
commonMain) and 0.32.0 (Llama: LlamaNetworkLoader.fromGguf(NATIVE_OPTIMIZED)
keeps weights packed via commonMain LlamaQuantLayout/LlamaPackedWeights).
NATIVE_OPTIMIZED no longer requires MemSeg/FFM — the packed path runs on
Kotlin/Native, and we've verified it on iOS-simulator builds. The demo pins
transformers 0.34.1, so the comments are stale even against its own
dependency versions.
The iOS/Android/web actuals then bake the assumption in: they all call
buildQwenRuntimeFallback, which hard-codes QuantPolicy.DEQUANTIZE_TO_FP32.
Also, README.md:93-96 ("Q4_1 quantization is not supported … tracked in
SKaiNET#654"): that issue is closed — the JVM MemSeg path handles Q4_1 via dequant
since transformers 0.30.0. The remaining Q4_1 gap is in the engine's streaming
loader, for which we've filed SKaiNET-developers/SKaiNET#919; the README note
deserves an update either way (the closed-issue link now under-sells what works).
Ask
- Default to
QuantPolicy.NATIVE_OPTIMIZED on all platforms in the demo; keep
DEQUANTIZE_TO_FP32 at most as an explicitly-labeled compatibility escape
hatch, not the mobile default.
- Update the kdoc in
QwenRuntimeBuilder.kt and the comment in
QwenRuntimeBuilder.android.kt to reflect the post-0.32.0 reality.
- Refresh the README "Known issues" Q4_1 note.
Happy to send the PR — we have before/after numbers from real devices to put in
the commit message.
Why it matters
KllamaDemo is the first thing a new SKaiNET LLM user copies, and right now it
steers people onto the slowest and most memory-hungry path on exactly the devices
that can least afford it. We believed the comments, built our first mobile version
on
DEQUANTIZE_TO_FP32, and got 1 tok/s plus an OOM — then triedNATIVE_OPTIMIZEDanyway and got 21 tok/s and a ~4x smaller footprint (145 MB vs540 MB resident for SmolLM2-135M Q8_0), including on Kotlin/Native, where the
fallback was never triggered.
The stale claims
composeApp/src/commonMain/kotlin/sk/ainet/apps/kllama/chat/playground/QwenRuntimeBuilder.kt:19-22:and
:31-36("until upstream lands a wasmJs/Native SIMD backend"), andQwenRuntimeBuilder.android.kt:7-9("the JVM fast path(DecoderGgufMemSegConverter) isn't available. Fall back…").
The premise changed in SKaiNET-transformers 0.30.0 (Gemma packed path moved to
commonMain) and 0.32.0 (Llama:
LlamaNetworkLoader.fromGguf(NATIVE_OPTIMIZED)keeps weights packed via commonMain
LlamaQuantLayout/LlamaPackedWeights).NATIVE_OPTIMIZEDno longer requires MemSeg/FFM — the packed path runs onKotlin/Native, and we've verified it on iOS-simulator builds. The demo pins
transformers 0.34.1, so the comments are stale even against its own
dependency versions.
The iOS/Android/web actuals then bake the assumption in: they all call
buildQwenRuntimeFallback, which hard-codesQuantPolicy.DEQUANTIZE_TO_FP32.Also, README.md:93-96 ("Q4_1 quantization is not supported … tracked in
SKaiNET#654"): that issue is closed — the JVM MemSeg path handles Q4_1 via dequant
since transformers 0.30.0. The remaining Q4_1 gap is in the engine's streaming
loader, for which we've filed SKaiNET-developers/SKaiNET#919; the README note
deserves an update either way (the closed-issue link now under-sells what works).
Ask
QuantPolicy.NATIVE_OPTIMIZEDon all platforms in the demo; keepDEQUANTIZE_TO_FP32at most as an explicitly-labeled compatibility escapehatch, not the mobile default.
QwenRuntimeBuilder.ktand the comment inQwenRuntimeBuilder.android.ktto reflect the post-0.32.0 reality.Happy to send the PR — we have before/after numbers from real devices to put in
the commit message.