Skip to content

Kernel SPI: make weight addressing order and byte source declared contract dimensions #1193

Description

@michalharakal

Follow-up to #1189/#1190 (siblings: #1191 FFM tier, #1192 remaining formats). This one is about the kernel contract, not any single kernel.

The two hidden dimensions

#1190 exposed that every packed matmul kernel actually lives in a matrix that the SPI never named:

  1. Addressing order — where block (o, b) sits in the weight bytes:
    • feed order (b · outputDim + o) · bpb — the prepacked fast path (skainet_q4k_matmul)
    • row-major file order (o · blocksPerRow + b) · bpb — what an mmap'd GGUF is (skainet_q4k_matmul_rm)
  2. Byte source — where the weight bytes live when the bridge hands them to C:
    • heap ByteArray (JNI GetPrimitiveArrayCritical, FFM staging copy)
    • off-heap direct address (JNI GetDirectBufferAddress, FFM MemorySegment zero-copy)

The C functions are pointer-based and never cared — the cost is in the bridges: today each occupied cell of formats × order × source × (JNI|FFM) is its own interface/class/shim (Q4KMatmulKernel, Q4KMemSegMatmulKernel, JniBufferPackedMatmulKernel, …), most cells are simply empty, and an empty cell fails soft: PackedViewMatmulKernel/JniBufferPackedMatmulKernel silently fall back to the decoding reference — correct, hours-scale, invisible.

Proposed

  1. Order stays two symbols in C (the loop structures genuinely differ; keep hot loops branch-free) — but a pair becomes the definition of done for a new packed format: feed-order + _rm, parity-tested against each other on permuted bytes (RowMajorMatmulParityTest is the template). The inner block-dot helpers are shared, so the second variant is ~60 lines.
  2. Byte source collapses in the bridge: one view kernel per (encoding, order) key that unwraps whatever Storage arrives — Heap → array shim, OffHeap/Mapped → direct shim — instead of parallel kernel classes registered under the same key. Order is already in the dispatch key (BLOCKED_ROW_MAJOR/BLOCKED_INPUT_MAJOR, Packed storage that knows its own block order, so KERNEL_FEED can reach a kernel #1120); source is a runtime property of the operand and belongs inside run(), not in the key.
  3. Fallback becomes visible: when a packed view kernel punts to the decoding reference (unsupported storage, strided activation), emit a trace event — SKEEP-003 §5.1 already demands adapters be visible in the trace; a 1000× silent slowdown deserves the same.
  4. The kernel support matrix (generateKernelMatrix) grows the two columns, so coverage gaps are documentation, not archaeology — and StorageCapabilities.mappedServableEncodings can eventually be derived from registrations instead of hand-maintained (the plan/load/dispatch truth-coupling from feat(#1189): packed-tensor mapped staging — Q4_K/Q6_K served straight from the mmap on Android #1190's planner fix).

Why now

Android is the next primary target, and the mapped tier is its load-bearing feature: every future format pays these two dimensions. Naming them in the contract makes each new kernel a checklist instead of a rediscovery.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions