You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
off-heap direct address (JNI GetDirectBufferAddress, FFM MemorySegment zero-copy)
The C functions are pointer-based and never cared — the cost is in the bridges: today each occupied cell of formats × order × source × (JNI|FFM) is its own interface/class/shim (Q4KMatmulKernel, Q4KMemSegMatmulKernel, JniBufferPackedMatmulKernel, …), most cells are simply empty, and an empty cell fails soft: PackedViewMatmulKernel/JniBufferPackedMatmulKernel silently fall back to the decoding reference — correct, hours-scale, invisible.
Proposed
Order stays two symbols in C (the loop structures genuinely differ; keep hot loops branch-free) — but a pair becomes the definition of done for a new packed format: feed-order + _rm, parity-tested against each other on permuted bytes (RowMajorMatmulParityTest is the template). The inner block-dot helpers are shared, so the second variant is ~60 lines.
Byte source collapses in the bridge: one view kernel per (encoding, order) key that unwraps whatever Storage arrives — Heap → array shim, OffHeap/Mapped → direct shim — instead of parallel kernel classes registered under the same key. Order is already in the dispatch key (BLOCKED_ROW_MAJOR/BLOCKED_INPUT_MAJOR, Packed storage that knows its own block order, so KERNEL_FEED can reach a kernel #1120); source is a runtime property of the operand and belongs inside run(), not in the key.
Fallback becomes visible: when a packed view kernel punts to the decoding reference (unsupported storage, strided activation), emit a trace event — SKEEP-003 §5.1 already demands adapters be visible in the trace; a 1000× silent slowdown deserves the same.
The kernel support matrix (generateKernelMatrix) grows the two columns, so coverage gaps are documentation, not archaeology — and StorageCapabilities.mappedServableEncodings can eventually be derived from registrations instead of hand-maintained (the plan/load/dispatch truth-coupling from feat(#1189): packed-tensor mapped staging — Q4_K/Q6_K served straight from the mmap on Android #1190's planner fix).
Why now
Android is the next primary target, and the mapped tier is its load-bearing feature: every future format pays these two dimensions. Naming them in the contract makes each new kernel a checklist instead of a rediscovery.
Follow-up to #1189/#1190 (siblings: #1191 FFM tier, #1192 remaining formats). This one is about the kernel contract, not any single kernel.
The two hidden dimensions
#1190 exposed that every packed matmul kernel actually lives in a matrix that the SPI never named:
(o, b)sits in the weight bytes:(b · outputDim + o) · bpb— the prepacked fast path (skainet_q4k_matmul)(o · blocksPerRow + b) · bpb— what an mmap'd GGUF is (skainet_q4k_matmul_rm)ByteArray(JNIGetPrimitiveArrayCritical, FFM staging copy)GetDirectBufferAddress, FFMMemorySegmentzero-copy)The C functions are pointer-based and never cared — the cost is in the bridges: today each occupied cell of
formats × order × source × (JNI|FFM)is its own interface/class/shim (Q4KMatmulKernel,Q4KMemSegMatmulKernel,JniBufferPackedMatmulKernel, …), most cells are simply empty, and an empty cell fails soft:PackedViewMatmulKernel/JniBufferPackedMatmulKernelsilently fall back to the decoding reference — correct, hours-scale, invisible.Proposed
_rm, parity-tested against each other on permuted bytes (RowMajorMatmulParityTestis the template). The inner block-dot helpers are shared, so the second variant is ~60 lines.(encoding, order)key that unwraps whateverStoragearrives —Heap→ array shim,OffHeap/Mapped→ direct shim — instead of parallel kernel classes registered under the same key. Order is already in the dispatch key (BLOCKED_ROW_MAJOR/BLOCKED_INPUT_MAJOR, Packed storage that knows its own block order, so KERNEL_FEED can reach a kernel #1120); source is a runtime property of the operand and belongs insiderun(), not in the key.generateKernelMatrix) grows the two columns, so coverage gaps are documentation, not archaeology — andStorageCapabilities.mappedServableEncodingscan eventually be derived from registrations instead of hand-maintained (the plan/load/dispatch truth-coupling from feat(#1189): packed-tensor mapped staging — Q4_K/Q6_K served straight from the mmap on Android #1190's planner fix).Why now
Android is the next primary target, and the mapped tier is its load-bearing feature: every future format pays these two dimensions. Naming them in the contract makes each new kernel a checklist instead of a rediscovery.