Library-first Bun benchmark for instrumented, llama.cpp-compatible Chat Completions engines. The benchmark downloads pinned BFCL V4 records, compiles one immutable semantic workload plan, and replays the same messages, tools, schedules, and serving policy against every target.
Fetch and inspect the corpus:
bun benchmark corpus fetch
bun benchmark corpus statusExperiments are TypeScript modules that bind immutable model artifacts to engines. Inspect and prepare one before running it:
bun benchmark experiments list
bun benchmark experiments show packages/inference-benchmark/experiments/qwen36-q8-mlx-lm-vs-llama-cpp.experiment.ts
bun benchmark prepare packages/inference-benchmark/experiments/qwen36-q8-mlx-lm-vs-llama-cpp.experiment.tsModel downloads use the standard Hugging Face cache ($HF_HOME/hub, otherwise
~/.cache/huggingface/hub) and are reused by Magnitude. Reusable TypeScript model definitions pin
the logical upstream model and every engine-loadable artifact. GGUF artifacts pin the repository,
commit, filename, byte size, and SHA-256; multi-file MLX artifacts use a committed complete-file
lock. HF_TOKEN or HUGGING_FACE_HUB_TOKEN is used for gated repositories.
Create a complete MLX lock when registering a new artifact:
bun benchmark models lock-mlx <repository> <revision> <bits> <group-size> <output.lock.json>
bun benchmark models lock-mlx-unquantized <repository> <revision> <bfloat16|float16> <output.lock.json>Inspect the deterministic workload without running inference:
bun benchmark plan packages/inference-benchmark/experiments/qwen36-q8-mlx-lm-vs-llama-cpp.experiment.tsRun the prepared experiment and inspect its immutable evidence:
bun benchmark run packages/inference-benchmark/experiments/qwen36-q8-mlx-lm-vs-llama-cpp.experiment.ts
bun benchmark runs list
bun benchmark runs show <run-id>
bun benchmark runs watch <run-id>Start the local dashboard over those same experiments and run files:
bun benchmark dashboardThe dashboard is served at http://127.0.0.1:5187; its API is at
http://127.0.0.1:4897. It has no separate configuration or database.
Managed oMLX is the unified MLX serving path for new standardized results. Its UV environment pins
oMLX and all material MLX dependencies, imports oMLX through an owned adapter, and exposes exactly
one locked snapshot through a private run-scoped base directory. Preparation may install and
qualify that environment; measurement runs use --frozen --no-sync and cannot install or download.
Every variant and block gets a fresh process. Memory is sampled from the complete managed process
tree.
Speculative decoding is an explicit discriminated setting. Embedded MTP and DSpark weights belong to the target snapshot; DFlash declares a separately locked MLX draft snapshot. Readiness fails if oMLX selects a different backend, and speculative runs fail unless every request returns exact request-local drafted and accepted counts and the run demonstrates nonzero drafting.
requestPolicy.contextTokensPerSequence is logical context per sequence and
requestPolicy.parallelSequences is serving capacity. Both values are part of the plan digest. The
llama.cpp adapter multiplies them for its shared --ctx-size; ICN's reported allocation must match
them before measurement starts.
Every target must return standard terminal usage, including cached prompt tokens, and the
llama.cpp-compatible timings object. The benchmark rejects missing or inconsistent evidence; it
never estimates token counts, prefill time, or decode time from text length or client latency.
Workload checkpoints are semantic depths and concurrency levels, not predicted token counts.
Reports use each target's terminal counts for actual work and label a comparison as a product
comparison if corresponding requests render different prompt-token counts.
| Workload | Responsiveness | Prefill | Decode | Memory | Distribution |
|---|---|---|---|---|---|
| Single request | Baseline TTFT | Isolated input rate | Isolated output rate | Baseline and peak | Variance and failures |
| Sequential session | TTFT by depth | Rate by context depth | Rate by depth | History growth | Turn-level tails |
| Independent concurrency | TTFT under load | Prefill under load | Decode under load | Batch peak | Throughput and fairness |
| Forked concurrency | Branch TTFT | Prefix reuse and uncached rate | Branch decode rate | Prefix and branch footprint | Throughput and fairness |
| Concurrency pressure | TTFT by concurrency | Prefill degradation | Decode degradation | Footprint by concurrency | Saturation and tails |
| Memory pressure | TTFT with resident histories | Prefill under memory load | Decode under memory load | Peak and retained scaling | OOM and failure onset |
| Context scaling | TTFT by prompt size | Full-prompt rate by size | Decode by size | Peak by size | Long-context degradation |
For a focused long-context comparison, a context-sweep suite stacks canonical completed BFCL
interactions until the shared serialized messages and tools are closest to each configured character
target. Token targets use a declared approximation such as 3.5 characters per token; both engines
receive exactly the same content, and reports use each engine's terminal prompt-token count as the
actual context length. Context-scaling reports keep BFCL correctness separate from measurement:
semantically invalid completions remain visibly invalid, but complete native timing evidence is
still summarized. Protocol and transport failures are never measured.
Experiments may set requestPolicy.requestTimeoutMs for trials expected to exceed the five-minute
default, such as 200K-context prefills.
Trial planning uses the reusable model definition's logical model ID and declared context limit, so an all-MLX experiment needs no GGUF. Artifact locks and qualifications remain separate prepared evidence. The experiment's context policy is an upper bound.
Reusable model definitions separate a logical model from its concrete artifacts. Experiments import those artifacts and bind each one to an engine:
export default defineExperiment({
id: "qwen36-q4-engine-comparison",
title: "Qwen3.6 35B-A3B 4-bit engine comparison",
suite: agentCore({ profile: "standard" }),
requestPolicy: {
contextTokensPerSequence: 32768,
parallelSequences: 4,
maxOutputTokens: 256,
temperature: 0,
topP: 1,
seed: 42,
enableThinking: false,
},
variants: [
{
id: "llama-cpp-q4",
artifact: qwen36.artifacts.llamaQ4,
engine: llamaCpp({ /* engine-only settings */ }),
},
{
id: "mlx-lm-q4",
artifact: qwen36.artifacts.mlx4,
engine: mlxLm({ /* engine-only settings */ }),
},
],
execution: { blocks: 2, variantOrder: "balanced" },
})A minimal managed baseline binds an MLX artifact to the frozen adapter:
{
id: "omlx-baseline",
artifact: model.artifacts.mlx4,
engine: omlx({
pythonProject: "../engines/omlx",
cache: { kind: "disabled" },
memoryGuard: { kind: "off" },
speculativeDecoding: { kind: "none" },
}),
}An embedded-MTP variant changes only the speculative policy. Put it alongside the baseline in a
defineSpeculativeDecodingComparison({...}) definition, with balanced blocks divisible by the
number of variants. Normal defineExperiment({...}) definitions hold speculative policy constant.
{
id: "omlx-mtp",
artifact: model.artifacts.mlx4,
engine: omlx({
pythonProject: "../engines/omlx",
cache: { kind: "disabled" },
memoryGuard: { kind: "off" },
speculativeDecoding: { kind: "mtp", maxDraftTokens: 3 },
}),
}The public library exports prepareCorpus, compileTrialPlan, evaluate, compare, target
builders, analysis functions, report renderers, experiment builders, preparation, and run lifecycle
operations from src/index.ts.