Skip to content

KllamaDemo steers every non-JVM platform to DEQUANTIZE_TO_FP32 — the "SIMD/MemSeg kernels are JVM-only" comments are stale since transformers 0.32.0 #27

Description

@michalharakal

Why it matters

KllamaDemo is the first thing a new SKaiNET LLM user copies, and right now it
steers people onto the slowest and most memory-hungry path on exactly the devices
that can least afford it. We believed the comments, built our first mobile version
on DEQUANTIZE_TO_FP32, and got 1 tok/s plus an OOM — then tried
NATIVE_OPTIMIZED anyway and got 21 tok/s and a ~4x smaller footprint (145 MB vs
540 MB resident for SmolLM2-135M Q8_0), including on Kotlin/Native, where the
fallback was never triggered.

The stale claims

composeApp/src/commonMain/kotlin/sk/ainet/apps/kllama/chat/playground/QwenRuntimeBuilder.kt:19-22:

On wasmJs / iOS / Android we fall back to [buildQwenRuntimeFallback] …
Same correctness, much slower throughput — the SIMD/MemSeg kernels are JVM-only.

and :31-36 ("until upstream lands a wasmJs/Native SIMD backend"), and
QwenRuntimeBuilder.android.kt:7-9 ("the JVM fast path
(DecoderGgufMemSegConverter) isn't available. Fall back…").

The premise changed in SKaiNET-transformers 0.30.0 (Gemma packed path moved to
commonMain) and 0.32.0 (Llama: LlamaNetworkLoader.fromGguf(NATIVE_OPTIMIZED)
keeps weights packed via commonMain LlamaQuantLayout/LlamaPackedWeights).
NATIVE_OPTIMIZED no longer requires MemSeg/FFM — the packed path runs on
Kotlin/Native, and we've verified it on iOS-simulator builds. The demo pins
transformers 0.34.1, so the comments are stale even against its own
dependency versions.

The iOS/Android/web actuals then bake the assumption in: they all call
buildQwenRuntimeFallback, which hard-codes QuantPolicy.DEQUANTIZE_TO_FP32.

Also, README.md:93-96 ("Q4_1 quantization is not supported … tracked in
SKaiNET#654"): that issue is closed — the JVM MemSeg path handles Q4_1 via dequant
since transformers 0.30.0. The remaining Q4_1 gap is in the engine's streaming
loader, for which we've filed SKaiNET-developers/SKaiNET#919; the README note
deserves an update either way (the closed-issue link now under-sells what works).

Ask

  1. Default to QuantPolicy.NATIVE_OPTIMIZED on all platforms in the demo; keep
    DEQUANTIZE_TO_FP32 at most as an explicitly-labeled compatibility escape
    hatch, not the mobile default.
  2. Update the kdoc in QwenRuntimeBuilder.kt and the comment in
    QwenRuntimeBuilder.android.kt to reflect the post-0.32.0 reality.
  3. Refresh the README "Known issues" Q4_1 note.

Happy to send the PR — we have before/after numbers from real devices to put in
the commit message.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions