MODEL-NEMOTRON-H: the WEIGHT LOADER — the real checkpoint LOADS in its packed format (the model is NOT yet reachable from the ABI, see #810) - #752
Merged
Conversation
…, in the formats it ships in (#517) Closes the blocker §6b named as "still owed after W4": nothing materialized the 18487 enumerated tensors, so every checkpoint load left NemotronHHostWeights unmaterialized and the forward refused by name. Spec §6d is the authority. RED first, off the real 20.1 GiB checkpoint at the CONTENT-pinned revision 29f2d174 (#569): `Status: FAILURE!`, exit 1, THREW "host weights are not materialized", peak RSS after "load" 29 MiB. Note the instrument trap in the same output -- `assertions: 3 | 3 passed | 0 failed` beside a red gate, because the case THREW. EVERY WEIGHT IS HELD IN THE FORMAT THE CHECKPOINT SHIPS IT IN, and that is arithmetic rather than taste: the 5888 routed-expert projections alone are 29.4e9 parameters, 16.5 GiB packed against 58.7 GB at bf16. A dequantize-at-load loader does not fit on any box this project owns, and on a unified-memory box that is a reboot rather than a failed load. Measured host mirror 18013 MiB against 18013 MiB read out of the shards. The host reference forward composes vt::MatmulBT and has no NVFP4 and no FP8 entry point, so it widens a quantized operand TRANSIENTLY at the GEMM call site through the shared model_loader/nvfp4_dequant.h seam. That arm is DECLARED -- named in the header, named at the call site, counted by the load report -- and the quantized GEMMs stay kMoeGroupedGemmNvfp4Marlin and the fp8-linear registration, which W6 selects. NemotronHOwned::View now refuses a non-dense weight by name, so a packed buffer cannot be reinterpreted as the model dtype. Gated STRUCTURALLY, not only by tokens, because a checkpoint read as uniform NVFP4 stays numerically plausible and still matches tokens while moving the wrong bytes: enumerated / in the index 18487 / 18487 materialized + deferred 18217 + 270 = 18487 (MTP tower -> W5, by name) NVFP4 W4A16 g16 5935 projections / 17805 tensors FP8 W8A8 static 46 projections / 138 tensors fp8 KV scales 12 unquantized bf16 / f32 216 / 46 the five rows sum to 18217 == materialized widenings 69, and no more host bytes 18,888,922,112 (17.59 GiB) peak RSS 17.70 GiB after load, 18.38 GiB after a forward {W4A16_NVFP4: 5935, FP8: 46} is exactly the histogram W1 measured over the 5981 quantized_layers entries, now confirmed against the TENSORS. A finding worth carrying: A_log, D and dt_bias ship BF16 on disk and the forward requires them f32; those 69 widenings are upstream's own polarity and are counted so they cannot spread. EVIDENCE, not the W6 token gate: one forward per committed oracle prompt, argmax of the last position against the golden's FIRST generated token -- 3/3 (6993, 1032, 1349) on x86_64 AND on Jetson Thor. W6 still owns the token gate. Five mutations, each applied alone and restored with its SHA-256 re-verified: uniform-NVFP4 and the dropped KV scales refuse by name; the un-widened SSM scalars red the widening count and then the dtype; and the flipped nibble order and the ignored weight_scale_2 each give 0/3 goldens while 45 of 46 assertions STILL PASS -- a loader gate built only out of counts would have called both clean. The load lives in nemotron_h_weights.cpp, beside the enumeration it must agree with tensor-for-tensor, which is where every other architecture puts its load and which avoids a CMakeLists.txt edit that would have owed a docs/USAGE.md change with nothing honest to write in it. Reported, not repaired: test_op_parity is RED on this base for a reason owned by MODEL-MUSIC-MUSIC3 (#672) -- RunGoldenPass does `m["op"]` on every manifest.json under the goldens root and minimax_music3_oracle's has no `op` key. It landed at 34dc578, an ancestor of this base. Squashed off origin/main so check-doc-checkpoint's PER-COMMIT rule is satisfied trivially, the same call §5e made for W3; `git diff` against the merged branch tree is EMPTY. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…it may not form, and NOTHING new ran in CI (#517) FOLLOWING_AGENTS_PROTOCOL Repairs the two MEDIUM findings, three LOWs and two NITs a fresh review returned against PR #752 at `9bffa2b60`. The review CONFIRMED the substance -- 18217 of 18487 tensors materialized, 270 `mtp.*` deferred by name, host bytes 18,888,922,112 re-derived bit-exact from the 52 safetensors headers, 3/3 first-token agreement on x86_64 and Thor -- and every number it confirmed is unchanged here. F1. `CopyDense` read the safetensors mmap through `reinterpret_cast<const uint16_t*>(t.data)` / `<const float*>(t.data)`. `StTensor::data` is `8 + <JSON header length> + <sum of preceding sizes>` and NOT ONE of those terms is required to be even, so forming either pointer is undefined whether or not the access faults -- and the same file's scalar read already used `memcpy`, so it was internally inconsistent. This is #627's class, whose THIRD recurrence closed on main the same day at `fc903b8dd` (#674) after main had been RED on `sanitize-cpu (address,undefined)` since #641; that repair is mirrored exactly. Reads go through `vt::LoadUnaligned` (the seam #301 left behind, eight sibling loaders already on it) and the two bulk `memcpy`s drop their typed pointer, which they never needed. No sanitizer configuration touched, no scope widened. It also makes docs/FEATURES.md:83 true again -- that row already claims the `*_weights.cpp` loaders never form a typed pointer into the mapping. Anchors re-derived at HEAD and asserted unique (count == 1), because recorded line numbers go stale within a PR. A standalone `-fsanitize=address,undefined -fno-sanitize-recover=all` probe over an ODD address: the shipped cast exits 1 with `load of misaligned address 0x521000000101 ... requires 2 byte alignment`; `vt::LoadUnaligned` exits 0 and matches the memcpy oracle exactly. The reviewer measured this LATENT on this checkpoint (0 of 216 BF16, 0 of 6085 F32 misaligned) and that is recorded rather than used as a reason to leave it. F2. The seams were unreachable without the 20.1 GiB checkpoint: deleting `View`'s `VT_CHECK` left the live gate at 46/46 and both offline suites green, and with no `CHECKPOINT_ROOT` the loader test recorded `Passed 0.00 sec` with ZERO assertions -- indistinguishable from a real pass. The guard is not decorative: without it `View` hands out a 128-element bf16 tensor over a 64-byte NVFP4 buffer, 192 bytes OUT OF BOUNDS. (a) `test_nemotron_h_quantized_forms.cpp`, a new OFFLINE gate needing no checkpoint and registered unconditionally, asserts `View` refuses by name and that `DenseBf16` reproduces a dequant derived INDEPENDENTLY from the upstream formula -- E2M1 table, fp8-e4m3 decoder and bf16 RNE round all written from their format definitions, anchored by hand (0x38->1.0, nibble 0x7->6.0, bf16(1.5)=0x3FC0). Every fixture is dyadic so every expected value is EXACT in bf16 and the comparison is on the BIT PATTERN, which keeps doctest::Approx's 1.19e-5 floor out of it entirely. The nibble order, the per-16 group scale and the MULTIPLIED weight_scale_2 are each asserted, and a fifth case reaches the file-private DenseFor/DenseCopy through NemotronHMlpMixer against a dense arm built from the INDEPENDENT reference, so a dequant defect cannot cancel itself out. RED-before, compile exit status printed beside every result and the binary sha with it (all four differ, so no stale binary): baseline 5 cases / 130 assertions SUCCESS; nibble order -> kHighFirst FAILURE, 96 of 96 elements differ, probe pair exactly swapped; weight_scale_2 ignored FAILURE, 36 assertions; View guard neutralized FAILURE; FP8 input_scale applied FAILURE. The first two are spec 6d's M4/M5 -- the pair a counts-only gate calls clean -- which previously needed the 20.1 GiB checkpoint and three oracle prompts and are now caught in 0.01 s with no checkpoint. (b) the checkpoint-gated skip is LOUD, NAMED and COUNTED, on the `PendingRunnerOps()` idiom: a registry of gated cases, a recorded verdict each, and a closing accounting case. With CHECKPOINT_ROOT unset it now reports 2 cases / 7 assertions and logs "checkpoint-gated cases: 0 ran, 1 skipped, of 1". Armed: not recording the skip REDs 4 of 6 assertions, and restoring the pre-repair silent return REDs 2 of 2. L1: `LoadMamba` now branches on `quantized` like LoadExpert/LoadMlp and the enumeration do, so a released bf16 NemotronH no longer refuses with a DTYPE message for what is a declared scheme. `mamba_proj_bias` remains owed and is named rather than left to be found. L2: the two "consumes no golden" scope statements now say what the code does -- one forward per prompt, argmax of the last position vs the golden's FIRST token; the 32-token decode stays W6's. L3: `186ae4cea4d7...` turned out to be a REAL sha, of the pre-rename TU at `c4029deed`; `08a65696d` then folded it into `nemotron_h_weights.cpp` as a documented pure move. The table now names both identities and says plainly that M1-M3 were not re-run after the move. N1: 16.5 GiB is 16.5 GB (15.4 GiB), fixed in both headers and the spec with the arithmetic written out. N2: the report counts quantized WEIGHTS, not dequant EVENTS. Carried into the spec rather than fixed: the FP8 arm is weight-only on the host path (input_scale carried, unapplied) so it is not a bit-mirror of vLLM's W8A8 static GEMM; and the golden arm's reach is 3 prompts x 1 token, at most 78 of 128 experts per layer, never the dense mlp, never the MTP tower. Gate: Release -Werror clean build exit 0, 0 warnings, 0 ENOSPC (44G free, 90%); new suite 5/5 130/130; loader with CHECKPOINT_ROOT unset 2/2 7/7; -DVLLM_CPP_SANITIZE='address,undefined' build of the three Nemotron-H targets clean, all three SUCCESS, no sanitizer finding. The live checkpoint gate is deliberately NOT re-run here -- the operator runs it on GB10. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
FOLLOWING_AGENTS_PROTOCOL `origin/main` at 3edb3c0, 36 commits ahead of this branch's fork point. Merged with an explicit message file rather than `--no-edit`, which writes a trailer-less message that `check-commit-trailers` does not exempt for merges. TWO KEYED RECORDS moved on main and NEITHER was taken from the automatic three-way merge (AGENTS.md Records). Both were re-resolved by key: main's version checked out WHOLESALE, this row's scoped edit reapplied on top with the anchor asserted unique, then the result verified against main. docs/FEATURES.md -- differs from main by EXACTLY ONE line, the `NemotronHForCausalLM` row. Every other row byte-for-byte main's, including the rows main added since the fork. tests/CMakeLists.txt -- differs from main by EXACTLY ONE additive hunk, 594a595,637: 43 lines added, 0 removed. main's 56 new lines since the fork are untouched. `.agents/roadmap_v1.md` also moved on main; this row does not edit it and the merged file is byte-identical to main's. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
… did not build the behaviour either side had FOLLOWING_AGENTS_PROTOCOL Found by RE-RUNNING the full gate after merging `origin/main`, not by reading the diff -- `git merge` reported no conflict at all, because the two changes touch different files. #784 (`b1cd4d8f6`, part of #730) rewrote `test_nemotron_h_scaffold`'s refusal subcase to call the REAL `reg.factory->load_weights(reg, config, source)` rather than downcast a fabricated `struct StubModel : vllm::LoadedModel`. UBSan was right about the stub, and the rewrite is correct ON MAIN, where `LoadNemotronHForCausalLM` reads only `source.kind` and never touches `source.safetensors`. §6d then gave NemotronH a loader that refuses an empty source by name. Merged, #784's `ModelSource source; source.kind = kSafetensors;` hits exactly that refusal and the subcase THREW before it ever reached the forward: test_nemotron_h_scaffold.cpp:666: ERROR: test case THREW exception: Model architecture NemotronHForCausalLM: the safetensors source carries no shards Neither parent is red. [[merge-tree-clean-is-not-builds]]. Repaired here because here is where the two sides meet, by SPLITTING the subcase rather than deleting either side's guarantee: - an empty safetensors source REFUSES AT LOAD, by name (`NemotronHForCausalLM`, `carries no shards`) -- the guarantee moved EARLIER and is asserted where it now lives, which is a stronger claim than the one it replaces; - the forward still refuses on unmaterialized weights, reached through the exported `vllm::NemotronHForward` on a default-constructed `NemotronHHostWeights`. That state is no longer reachable through the factory at all now that the loader exists, so it is asserted on a REAL `NemotronHHostWeights`. #784's substance is kept in full: no `StubModel`, no downcast onto an object that never was a `NemotronHLoadedModel`, no UB. ARMED, not decorative; compile exit and binary sha printed beside each result and the two binaries differ. Replacing the `carries no shards` refusal with `return model;` -> BUILD_EXIT=0, sha `4e6cdbd93478`, `Status: FAILURE!`, 11 of 12 cases, 2 of 38289 assertions failed. Neutering `VT_CHECK(host.materialized, ...)` to `VT_CHECK(true, ...)` -> BUILD_EXIT=0, sha `84179116c90c`, `Status: FAILURE!`, 11 of 12, 2 of 38289. Green after: 12/12 cases, 38289/38289, `Status: SUCCESS!` -- the split adds four assertions to W4's 38285 and the CASE count stays 12. Also in this commit, carried from the review repair: the new offline suite's independent fp8-e4m3 reference had the SUBNORMAL exponent wrong (2^-7 for 2^-6). No fixture reached it, so nothing was mis-judged, but a reference that is only right where it is exercised is not a reference. Fixed and anchored by hand at 0x01/0x07/0x08, and armed -- restoring 2^-7 REDs 2 of 134 assertions at BUILD_EXIT=0, sha `182816a73a4c`. `docs/FEATURES.md` deliberately does NOT claim the unquantized bf16 mamba arm L1 added: it is reachable and consistent with its own enumeration, but no released bf16 NemotronH checkpoint is within reach and no synthetic one exists, so it is argued rather than measured. An ungated branch is visible debt, not a supported surface, and the spec says so. Full gate at the merged head: clean Release `-Werror` build from an empty tree BUILD_EXIT=0, 0 warnings, 0 errors, 0 ENOSPC; `ctest -j4` 468 of 469 (the one failure being exactly this #818, repaired after that run); the four Nemotron-H suites 12/12+13/13+2/2+5/5 all SUCCESS afterwards. `test_op_parity` PASSES at this merge base -- main's own red (#755/#672) is gone. Closes #818. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…e from the ABI (#810), and #815 does not overlap F1 (#517) FOLLOWING_AGENTS_PROTOCOL Record only. No executable line moves. TWO facts a reader of §6d would otherwise get wrong. 1. #810. NemotronH does NOT run end to end through `include/vllm.h` -- not on `main`, and not with this branch merged. Both refuse at `src/vllm/v1/worker/gpu/runner.cpp:525`, `runner: Qwen3.5 MambaSpec shapes disagree with model config`, a line this branch does not touch. Established independently on the real 21 GiB checkpoint on GB10: the weights load fine (17.7 GiB RSS) and then ENGINE CONSTRUCTION refuses. So "the real checkpoint runs" means the HOST REFERENCE FORWARD runs on real weights, and the spec now says that in those words so nobody reads it as "the server runs it". That single `VT_CHECK` is the ONLY load-time blocker -- neutering it alone made `vllm_engine_load` succeed and reach the forward. The spec records why that is NOT a fix: `ForwardNemotronHForCausalLM` ignores `attn_kv`, `gdn_state`, `gdn_meta` and `num_reqs`, so a server past that check emits silently wrong tokens from decode step 2 -- precisely what a 1-token golden arm cannot see. The refusal is the only thing currently making the gap visible. #810 owns it, W6 owns the paged/device runner, and it is explicitly out of scope here. 2. PR #815 / `row/FIX-UNALIGNED-LOADERS-772` closes the SAME CLASS as F1 and does NOT overlap it. Checked, not assumed: its complete file list carries no `nemotron_h_*` file, and `git log -S'nemotron_h_weights'` over `origin/main..origin/row/FIX-UNALIGNED-LOADERS-772` is empty. The reason is structural -- #772's sweep was taken over `main`, and this site does not exist on `main`; it arrives with PR #752 itself. Established with a POSITIVE CONTROL rather than from a silent grep: the same regex that is silent on the repaired `nemotron_h_weights.cpp` FIRES on `voxtral.cpp:51`, `voxtral.cpp:344` and `qwen3_vl.cpp:78`, which is exactly #815's set. The two are complementary; whichever lands second re-resolves `docs/FEATURES.md` and `tests/CMakeLists.txt` by key, the only two files both touch. Anchors re-derived at HEAD with uniqueness asserted (count == 1) for all ten sites this row edits, and the drift is why: F1's two anchors have moved 455 -> 468 and 473 -> 490 since the reviewed head `9bffa2b60` ([[recorded-line-anchors-go-stale-within-the-same-pr]]). Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…#517) FOLLOWING_AGENTS_PROTOCOL Merged an IMMUTABLE SHA, `c01e4be93ddcea6c7bf922cbcdc351808c9097ad`, not the moving `origin/main` ref: one `.git` is shared by many sessions here and the ref moves under you mid-merge ([[origin-main-moves-under-you-in-a-shared-checkout]]). Five commits ahead of the previous merge base. Explicit `-F <message-file>` rather than `--no-edit`, which writes a trailer-less message that `check-commit-trailers` does not exempt for merges. TWO KEYED RECORDS moved on main. `git merge` reported NO CONFLICT on either, and that automatic three-way result was DISCARDED rather than accepted (AGENTS.md Records). Both were re-resolved by key: main's version checked out WHOLESALE, this row's scoped edit reapplied on top with the anchor asserted UNIQUE (count == 1), then the result verified against main: docs/FEATURES.md -- differs from main by EXACTLY ONE line, the `NemotronHForCausalLM` row (`diff` reports `142c142` and nothing else). Every other row byte-for-byte main's, including rows main added since the fork. tests/CMakeLists.txt -- differs from main by EXACTLY ONE additive hunk, `594a595,637`: 43 lines added, ZERO removed. Verified the two targets it registers appear exactly once each, and that main's own `test_offload_config` addition at :875 -- which taking this row's file wholesale would have silently DELETED -- survives. That last line is the reason the rule exists: the automatic merge happened to get it right this time, but a wholesale take would have evicted a target added by an unrelated PR, and nothing would have failed loudly. `.agents/roadmap_v1.md` also moved on main; this row does not edit it and the merged file is byte-identical to main's, asserted with `git diff --quiet`. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…470, clean build, all preflight gates green (#517) FOLLOWING_AGENTS_PROTOCOL Record only. No executable line moves. Re-ran the row's gate MYSELF at `e3f2992af` over `origin/main` @ `c01e4be93`, rather than carrying forward the numbers from the contended run earlier in this session: Release -Werror, CLEAN full build from an empty tree BUILD_EXIT=0, 0 `warning:`, 0 `error:`, 0 `No space left`, 1393/1393 full `ctest -j4` 470 of 470 PASSED, CTEST_EXIT=0, 124 s skipped: test_modelopt_mixed_precision_checkpoint, test_voxtral_e2e scripts/agent-preflight.sh All gates green disk 103G free / 76-77% before and after Two reds reported earlier in this session are now shown to be ENVIRONMENTAL, and both are recorded rather than quietly dropped. `test_cpu_x86_llamacpp_floor` RED at load average 91-130 with `NO_QUIET_WINDOW after 30s (busy=102% ...)`, and it reproduced on the SHARED CHECKOUT at `main`, which is what made it base-inherited rather than this row's ([[cpu-x86-floor-test-reds-under-box-load]]); it passes on the idle box. `test_cpu_threadpool` likewise. `test_op_parity`, RED on `main` when this task was scoped (#755/#672), PASSES at this merge base. The earlier contended run is KEPT in the spec rather than replaced: 468 of 469 with `test_nemotron_h_scaffold` red is how #818 was found, and it was found by RE-RUNNING the gate, not by reading the merge diff -- which reported no conflict at all. A negative result is a result. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…517) FOLLOWING_AGENTS_PROTOCOL Second merge of this task, at an IMMUTABLE SHA again: `origin/main` moved under me between the previous merge and the push, which is exactly what a shared checkout does ([[origin-main-moves-under-you-in-a-shared-checkout]]). Six commits, and two of them matter here. **#815 LANDED** as `ca01719e6` -- the branch this task asked me to check for overlap with F1. The verdict is now confirmed against `main` itself rather than against a branch: `git show --name-only ca01719` lists NO `nemotron_h_*` file, and `main`'s own `nemotron_h_weights.cpp` contains zero occurrences of `CopyDense` and neither of F1's two casts. The site F1 repairs does not exist on `main` at all; it arrives with THIS PR, which is why #772's sweep could not have seen it. The two changes are complementary and together close the class: after this merge, the regex that finds the defect fires nowhere in `src/`. **A new commit-style checker landed** (`ddff09093`, POLICY-SINGLE-PR-AND-STYLE) with `.agents/style/commits.md`, `.agents/style/prose.md` and `scripts/check-commit-style.py`, and `scripts/agent-preflight.sh` changed with it. That is a gate this branch had never been measured against, so the merge is followed by a full re-gate rather than by a push. BOTH KEYED RECORDS moved again, and #815 edited one of them. The automatic three-way merge was CLEAN and was DISCARDED for both (AGENTS.md Records): main's version taken WHOLESALE, this row's scoped edit reapplied with the anchor asserted unique, then verified against main. docs/FEATURES.md -- differs from main by EXACTLY ONE line, the `NemotronHForCausalLM` row. #815's own edit to :83 ("no loader forms a typed pointer into the mapping. Last three fixed by #772") survives untouched, which is the point of resolving by key rather than by file. tests/CMakeLists.txt -- differs from main by EXACTLY ONE additive hunk, 606a607,649: 43 added, ZERO removed. #815's new `test_loader_unaligned_offsets` target survives. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…e new style checker (#517) FOLLOWING_AGENTS_PROTOCOL `origin/main` moved again between the previous gate and the push, so the whole gate ran again rather than the earlier numbers being carried forward. The move mattered for a second reason: `ddff09093` landed `scripts/check-commit-style.py`, a checker this branch had never been measured against, and it changed `scripts/agent-preflight.sh` with it. At the merged head over `origin/main` @ `ca01719e6`: Release -Werror, CLEAN full build from an empty tree BUILD_EXIT=0, 0 `warning:`, 0 `error:`, 0 `No space left`, 1395/1395 full `ctest -j4` 471 of 471 PASSED, CTEST_EXIT=0, 111 s skipped: test_modelopt_mixed_precision_checkpoint, test_voxtral_e2e the four Nemotron-H suites scaffold 12/12, forward 13/13, loader 2/2, quantized_forms 5/5 scripts/check-commit-style.py --range ca01719..HEAD OK: commit writing style scripts/check-commit-trailers.py --range origin/main..HEAD OK: commit trailer contract disk 100G free / 77% One instrument failure is recorded because it nearly became a false report. A `tail` of the ctest log taken one minute before the run started read a PRE-EXISTING file naming a foreign worktree (`.claude/worktrees/agent-a956d471bf9a3b7d0/build-rel`) and ending in `Errors while running CTest`. The scratchpad is shared, so a stale log looks exactly like a fresh red ([[scratchpad-is-shared-across-sessions]]). It was discarded after checking that the live log names this worktree in its own header, and the result above comes only from the run that did. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
Collaborator
Author
|
Review repair pushed at #772 / #815 overlap — no overlap, and it is now checkable against Two things found by re-running the gate rather than by reading a diff.
Scope, stated plainly in the PR body: this branch makes the weights load. It does not make the model reachable from the public ABI — that is #810, at Not merging; the operator merges. |
localai-bot
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
This branch makes the WEIGHTS LOAD. It does NOT make the model reachable from the public ABI.
NemotronH does not run end to end through
include/vllm.h— not onmain, and notwith this branch merged. Both refuse at
src/vllm/v1/worker/gpu/runner.cpp:525,runner: Qwen3.5 MambaSpec shapes disagree with model config, a line this branch doesnot touch. Established independently on the real 21 GiB checkpoint on GB10: the weights
load fine (17.7 GiB RSS) and then engine construction refuses. Tracked as #810.
So "the real checkpoint runs" below means the host reference forward runs on real
weights. It must not be read as "the server runs it".
That single
VT_CHECKis the only load-time blocker — neutering it alone madevllm_engine_loadsucceed and reach the forward. That is not a fix.ForwardNemotronHForCausalLMignoresattn_kv,gdn_state,gdn_metaandnum_reqs,so a server past that check emits silently wrong tokens from decode step 2 — exactly
what a 1-token golden arm cannot see. The refusal is currently the only thing making the
gap visible. #810 owns it; W6 owns the paged/device runner. Out of scope here.
Closes the blocker
.agents/specs/nemotron-h-model.md§6b named as "still owedafter W4": nothing materialized the 18487 enumerated tensors, so every
checkpoint load left
NemotronHHostWeightsunmaterialized and the forwardrefused by name. Issue #517; spec §6d is the authority on this brick.
RED first
Off the real 20.1 GiB checkpoint, resolved by CONTENT (#569) to
/mnt/nas_share/checkpoints/nemotron-3.5-lightning-30b-nvfp4@29f2d174:The design decision, and why it is arithmetic
Every weight is held in the format the checkpoint SHIPS it in. Not tidiness:
the 5888 routed-expert projections alone are 29.4e9 parameters, 16.5 GB
packed against 58.7 GB at bf16 (both DECIMAL GB; 16.5 GB is 15.4 GiB, and the
review caught this written as GiB). A dequantize-at-load loader does not fit on any
box this project owns, and on a unified-memory box that is a reboot rather than a
failed load.
Measured host mirror 18013 MiB against 18013 MiB read out of the shards:
the mirror is the checkpoint, not a widened copy of it.
The consequence is recorded rather than hidden. The HOST reference forward
composes
vt::MatmulBTand has no NVFP4 and no FP8 entry point, so it widens aquantized operand transiently at the GEMM call site, through the shared
model_loader/nvfp4_dequant.hseam every other ModelOpt consumer here uses.That arm is named in the header, named at the call site, and the quantized GEMMs
stay
kMoeGroupedGemmNvfp4Marlinand the fp8-linear registration, which W6selects.
NemotronHOwned::Viewnow REFUSES a non-dense weight by name, so apacked buffer cannot be reinterpreted as the model dtype by a caller that did not
think about it.
The structural gate, as hard numbers
Gated structurally and not only by tokens, because a checkpoint read as uniform
NVFP4 stays numerically plausible and still matches tokens while moving the wrong
bytes.
model.safetensors.index.jsonmaterialized{W4A16_NVFP4: 5935, FP8: 46}is exactly the histogram W1 measured over the5981
quantized_layersentries, now confirmed against the TENSORS.A finding worth the reviewer's attention:
A_log,Danddt_biasshipBF16 on disk and the forward requires them f32. Those 69 widenings are
upstream's own polarity (
-torch.exp(self.A_log.float())) and are counted sothey cannot spread.
Evidence, NOT the W6 token gate
One forward per committed oracle prompt, argmax of the last position against the
golden's FIRST generated token: 3/3 (6993, 1032, 1349). W6 still owns the
token gate. This is here because it is the only check that can fail for a reason
the counts cannot see.
Mutation proof
in_projread as NVFP4 not FP8ships dtype F8_E4M3, not the U8 its scheme declaresenumerated but no host slot claimed itA_log/D/dt_biasinherit the model dtypewidened_tensors == 69red, thenwrong dtype for this armweight_scale_2read and then ignoredM4 and M5 are the pair that keeps the golden arm honest: a loader gate built only
out of counts would have called both clean.
Not in scope, refused by name
The MTP head (W5, its 270 tensors deferred and named), the GGUF arm (W7, refused
at load), the e2e token gate (W6), and any speed claim.
Review repair (fresh review of
9bffa2b60)A fresh reviewer confirmed every number above and returned 2 MEDIUM, 3 LOW, 2 NIT.
All seven repaired; none declined. Spec §6e is the authority.
CopyDenseread the mmap throughreinterpret_cast<const uint16_t*>(t.data)/<const float*>(t.data).StTensor::datais8 + <header len> + <sum of preceding sizes>— none required to be even, so the pointer may not be formedvt::LoadUnaligned; the two bulkmemcpys drop the typed pointer they never needed. Mirrorsfc903b8dd(#674).nemotron_h_weights.cpp:468,477,490,499Viewguard was correct but UNARMED: deleting it left the whole suite green, and with noCHECKPOINT_ROOTthe loader test readPassed 0.00 secwith zero assertionstest_nemotron_h_quantized_forms.cpp, registered unconditionally; (b) a named, counted skip with a closing accounting caseLoadMambadid not branch onquantized, so a bf16 checkpoint refused with a dtype message for what is a declared schemeLoadExpert/LoadMlpand the enumeration do. The arm is reachable but ungated anddocs/FEATURES.mddeliberately does not claim itnemotron_h_loader.cpp, which does not exist186ae4cea4d7…is a REAL sha — of the pre-rename TU atc4029deed;08a65696dfolded it intonemotron_h_weights.cppas a documented pure move. Both identities now named, and it says plainly M1–M3 were not re-run after the moveGate at the pushed head
c6255afa1, overorigin/main@ca01719e6-Werror, clean full build from an empty treeBUILD_EXIT=0, 0warning:, 0error:, 0No space left, 1395/1395ctest -j4CTEST_EXIT=0, 111 s. Skipped:test_modelopt_mixed_precision_checkpoint,test_voxtral_e2escripts/agent-preflight.shcheck-commit-style.py/check-commit-trailers.pyOK: commit writing style/OK: commit trailer contracttest_op_parity, red onmainwhen this work was scoped (#755/#672), passes atthis base.
test_cpu_x86_llamacpp_floorwent red earlier in the session at loadaverage 91-130 and reproduced on the shared checkout at
main, which is whatmade it environmental rather than this branch's; it passes on the idle box.
F2 RED-before → green-after
The new suite catches M4/M5 — the pair that leaves every structural count correct
— in 0.01 s with no checkpoint, where before they needed the 20.1 GiB
checkpoint and three oracle prompts. Compile exit and binary sha printed beside
every result; all binaries differ, so no stale binary printed a green status.
7b7df3464fb2Status: SUCCESS!kHighFirst)26bc2e92ed72Status: FAILURE!—96 of 96elements differ; probe pair exactly swapped (16576vs16128); the GEMM-reach case moves tooweight_scale_2ignored (1.0F)702537076950Status: FAILURE!—84 of 96differ, 36 assertions failedView'sVT_CHECKneutralized0303eeaeff7dStatus: FAILURE!input_scaleapplied (must be carried)630495efa1a9Status: FAILURE!The reference is derived independently from the upstream ModelOpt formula —
E2M1 table, fp8-e4m3 decoder and bf16 RNE round all written from their format
definitions, anchored by hand (
0x38→1.0, nibble0x7→6.0, bf16(1.5)=0x3FC0,and the subnormal ladder). Every fixture is dyadic, so comparisons are on the
bf16 bit pattern and
doctest::Approx's ~1.19e-5 floor never enters.F2(b) the skip is no longer indistinguishable from a pass
With
CHECKPOINT_ROOTunset: 2 cases / 7 assertions,Status: SUCCESS!, andthe log states
checkpoint-gated cases: 0 ran, 1 skipped, of 1. Armed — restoringthe pre-repair silent early return gives
Status: FAILURE!, 0 of 2 assertionspassed (sha
13e7a4d46101, BUILD_EXIT 0).#818 — a merge that was CLEAN and did not build
Merging
origin/mainmadetest_nemotron_h_scaffoldRED though both parentsare green. #784 rewrote its refusal subcase to call the real
load_weights(correct on
main, where the loader ignoressource.safetensors); §6d gives thatcall a loader that refuses an empty source. Repaired here by splitting the subcase
— refuse-at-load and refuse-on-unmaterialized-weights — keeping #784's substance
(no
StubModel, no downcast, no UB). Both halves mutation-armed.No overlap with #815 /
row/FIX-UNALIGNED-LOADERS-772Checked, not assumed. That branch closes the same class at
voxtral.cpp:51,344,qwen3_vl.cpp:78,qwen3_5_mtp.cpp:71andminimax_h3_vae_loader.cpp; its filelist contains no
nemotron_h_*file, andgit log -S'nemotron_h_weights'overorigin/main..origin/row/FIX-UNALIGNED-LOADERS-772is empty. Structural, not luck:#772's sweep was taken over
main, and this site does not exist onmain— itarrives with this PR. Established with a positive control rather than a
silent grep: the same regex that is silent on the repaired
nemotron_h_weights.cppfires on voxtral ×2 and
qwen3_vl.cpp:78, exactly #815's set. Complementary;whichever lands second re-resolves
docs/FEATURES.mdandtests/CMakeLists.txtby key.
Not in scope, refused by name
The MTP head (W5, its 270 tensors deferred and named), the GGUF arm (W7, refused
at load), the e2e token gate (W6), the ABI reachability blocker (#810), and any
speed claim.