Follow-up from the #1130 measurement (Pixel 8a, ART growth limit 256 MB): under WeightResidency.MAPPED only dense F32 tensors stay file-backed — quantized payloads materialize as heap ByteArrays, so a 1.06 GB Q4_K_M dies at load with
java.lang.OutOfMemoryError: Failed to allocate a 131272720 byte allocation
with 52020928 free bytes ... growth limit 268435456
while the 100 MB fits-case proved the mapped machinery itself holds major faults at zero through steady-state decode. The cap is the Java heap; the fix is for packed weights to never touch it.
Design sketch
PackedBlockStorage over mapped storage — a packed TensorData whose packedData is not a ByteArray but a window into MappedFileStorage/direct ByteBuffer (the loader already knows each tensor's absolute file offset; Q4_K/Q6_K/BITNET_B1_58 payloads are contiguous per tensor and read-only).- Kernel entry points that read off-heap bytes — the JNI shims currently take
ByteArray via GetPrimitiveArrayCritical; they need a sibling taking a direct-ByteBuffer address (GetDirectBufferAddress) or a raw pointer, calling the same C kernels (which are pointer-based already). The FFM tier gets this for free (MemorySegment of the mapping).
- Loader wiring — the quantized branches of
StreamingGgufParametersLoader honour WeightResidency.MAPPED by emitting the mapped-backed packed data instead of copying to heap; keep-on-heap remains the default off-Android.
- Prepack interaction — feed-order prepack (
BLOCKED_INPUT_MAJOR) is a write, so it either stays a heap copy (bounded: only the layers actually used, measured at 70 MB for a 135M model), is done once into an app-files cache file and mapped back, or the ByteArray-kernels accept canonical order via the row-major keys.
What it unlocks
Heap then holds only KV + forward slab (~25 MB at ctx 512): model size becomes bounded by device RAM and page-cache behaviour instead of the ~256 MB app cap — 1.5B–7B Q4_K on an 8 GB phone becomes a fault-rate question (which #1130's harness, PR #1188, can measure directly: it already records per-step major faults).
Measured baseline to beat, from #1130: SmolLM2-135M fits-case — 2 load faults, 0 steady-state faults, 63 ms/step.
Follow-up from the #1130 measurement (Pixel 8a, ART growth limit 256 MB): under
WeightResidency.MAPPEDonly dense F32 tensors stay file-backed — quantized payloads materialize as heapByteArrays, so a 1.06 GB Q4_K_M dies at load withwhile the 100 MB fits-case proved the mapped machinery itself holds major faults at zero through steady-state decode. The cap is the Java heap; the fix is for packed weights to never touch it.
Design sketch
PackedBlockStorageover mapped storage — a packedTensorDatawhosepackedDatais not aByteArraybut a window intoMappedFileStorage/directByteBuffer(the loader already knows each tensor's absolute file offset; Q4_K/Q6_K/BITNET_B1_58payloads are contiguous per tensor and read-only).ByteArrayviaGetPrimitiveArrayCritical; they need a sibling taking a direct-ByteBufferaddress (GetDirectBufferAddress) or a raw pointer, calling the same C kernels (which are pointer-based already). The FFM tier gets this for free (MemorySegmentof the mapping).StreamingGgufParametersLoaderhonourWeightResidency.MAPPEDby emitting the mapped-backed packed data instead of copying to heap; keep-on-heap remains the default off-Android.BLOCKED_INPUT_MAJOR) is a write, so it either stays a heap copy (bounded: only the layers actually used, measured at 70 MB for a 135M model), is done once into an app-files cache file and mapped back, or the ByteArray-kernels accept canonical order via the row-major keys.What it unlocks
Heap then holds only KV + forward slab (~25 MB at ctx 512): model size becomes bounded by device RAM and page-cache behaviour instead of the ~256 MB app cap — 1.5B–7B Q4_K on an 8 GB phone becomes a fault-rate question (which #1130's harness, PR #1188, can measure directly: it already records per-step major faults).
Measured baseline to beat, from #1130: SmolLM2-135M fits-case — 2 load faults, 0 steady-state faults, 63 ms/step.