Skip to content

Model-footprint analysis: 'will it fit on the device?' in seconds, for GGUF / safetensors / ONNX #5

Description

@michalharakal

Idea

A model-footprint analysis CLI: given a model file, answer "will this fit on the target device?" in seconds — before anyone invests a day in converting it.

The concrete pain: there is no quick way to check whether a model fits on an embedded device with limited memory and compute (usable RAM ≈ 2.1 GB) before committing to a full conversion run. The tool should:

  • accept GGUF, safetensors and ONNX files and analyze them from headers/metadata only (no payload load — a multi-GB file is answered in seconds);
  • report the actual resident size of the weights — as stored, and (where the format carries encodings) as they would be held in memory after resolution, since a Q4 weight that has to dequantize costs ~8× its file size;
  • print a verdict against the device budget (fits / does not fit, with margin and actionable suggestions such as KV-cache quantization or a smaller context).

What already exists in SKaiNET core

Most of the machinery landed with the SKEEP-003 memory-architecture arc (SKaiNET-developers/SKaiNET#932):

  • Header-only planning for GGUFStreamingGGUFReader.planInput() builds a PlanInput from the tensor table + architecture metadata without reading payloads; MemoryPlans.plan() prices weights, KV cache and the forward slab against a Budget and renders a fit table with suggestions.
  • Per-device policyPlannerProfile (e.g. MOBILE_2GB: reserve, mapped weights, auto KV quantization, strict mode) and DeviceFit/DeviceMemory for checking both RAM pools.
  • Resolved cost, not file costWeightFormResolver (what the file holds × profile × kernel capabilities) and AllocationResolver.explain() (P7: AllocationResolver — one resolver decides domain and scope SKaiNET#1143): the plan prices what will actually be held, and can say per tensor where it lands and why, pre-load.
  • A working CLI skeletonskainet-apps/skainet-plan already does all of the above for GGUF and prints the plan.

Tracked as a coding issue in core: SKaiNET-developers/SKaiNET#1169 (safetensors + ONNX front ends, the ~2.1 GB budget flag and verdict ergonomics).

Research questions

  1. ONNX weight enumeration without materialization — initializers live inside the protobuf graph; what is the cheapest reliable way to enumerate names/dims/dtypes (and external-data references) without loading tensor bytes, and how do quantized ONNX conventions (QDQ, MatMulInteger) map onto a resident-bytes estimate?
  2. Runtime overhead model per format/runtime — file weights are a lower bound; what per-runtime headroom model (allocator overhead, graph buffers, KV cache for decoder models without GGUF-style metadata) makes the 2.1 GB verdict trustworthy rather than optimistic? Can we calibrate it against measured RSS on the device (plan-vs-actual, as SKEEP-003 M1 did for GGUF decode)?
  3. Verdict quality — false "fits" is the expensive failure. What safety margin and phrasing keep the tool honest when geometry metadata is missing (safetensors/ONNX), where only weights + headroom can be priced?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions