Android eager decode is ~83% non-matmul overhead: base DefaultCpuOps runs boxed per-element ops — port the JVM primitive fast paths (#920 follow-up) #949

Follow-up to #920, measured on transformers 0.39.0 with the JNI NEON backend active (SKaiNET-transformers#272 has the e2e context; test in SKaiNET-transformers#288).

The numbers

SmolLM2-135M Q8_0, NATIVE_OPTIMIZED, OptimizedLLMRuntime(DIRECT), Pixel 8a (Tensor G3), 44 tokens greedy:

• e2e decode: 2.64 tok/s (377 ms/token). x86_64 emulator: 2.48 tok/s — near-identical despite far slower matmuls.
• KernelProfile over the decode loop: 10339/10339 matmuls through the packed-quant path (JNI NEON, 0 dense-FP32, 0 generic), 2.81 s of 16.6 s decode = 17%. The kernels deliver — ~64 ms/token including per-call JNI overhead at ~235 calls/token, consistent with the ~24 tok/s projection from fix(jni): eager library init; emulator-validated parity suite; gate q4k/q6k on #944 (#920) #945's benchmark.
• The other 83% is runtime overhead, and simpleperf (19,707 samples @ 2 kHz, cpu-cycles, mid-decode window) says exactly where it goes:

Per-DSO: 48% JIT app cache (Kotlin), 32.7% libart (interpreter/JNI/alloc/GC machinery), 15.4% the NEON .so.

Root cause

PlatformCpuOpsFactory.android returns the base DefaultCpuOps — the JVM gets DefaultCpuOpsJvm. On the base class, every non-matmul op runs through elementwise(a, b, op) (DefaultCpuOps.kt:164), which per element:

1. allocates two fresh IntArrays (mapIndex for broadcast index mapping),
2. calls a.data.get(*ai) — vararg spread + boxed V return (Float.valueOf in the profile),
3. runs a when (dtype) KClass comparison per element (getJavaObjectType / Reflection.getOrCreateKotlinClass / Intrinsics.areEqual),
4. boxes the lambda result back.

addScalar/mulScalar additionally materialize a full tensor of the scalar (dataFactory.full) before the boxed elementwise (DenseTensorDataFactory.init at 3.6%). These generic megamorphic methods also keep ART bouncing between interpreter and JIT (the 8% ART bucket).

Meanwhile most of DefaultCpuOpsJvm's overrides are not JVM-specific: silu (line 739) is a plain primitive FloatArray loop; add/subtract/multiply/divide are vectorFloatBinary(vectorLambda) { scalarLambda } where the scalar fallback is a primitive loop; transpose/sum/mean/relu similar. Only the Panama Vector / FFM parts are jvmMain-bound.

Proposed fix

1. Hoist the primitive FloatArray fast paths (binary elementwise scalar loops, silu/relu, sum/mean, transpose, softmax/norm helpers, concat) out of DefaultCpuOpsJvm into shared code (commonMain base or a shared source set), and let Android use them — keep Panama/FFM specializations in jvmMain on top.
2. Give addScalar-family primitive paths too (no materialized scalar tensor).
3. Minor, free: DirectCpuExecutionContext.ops is get() = opsFactory(tensorDataFactory) — a new ops instance per access, which also re-resolves the per-instance by lazy kernels. Cache it.

Expected effect: with matmul at ~64 ms/token, cutting the 313 ms/token overhead by the 5–10× that primitive loops typically buy over boxed per-element access puts Android e2e at ~6–10 tok/s — comfortably past the field report's 3 tok/s usability gate, from a change that's mostly code motion.

Happy to take this: extract-and-share PR first, then re-measure on the same device/test for before/after.