gate(ROCM): M4 for Qwen3.5-0.8B GDN -- the oracle-backed near-tie gate as the RED anchor of the forward-divergence fix - #559
Conversation
6f6ee84 to
5efcebc
Compare
|
Reviewed as part of a sweep over the open external PRs. The measurement behind this is real — I recomputed the verdict independently from the committed goldens and got exactly your numbers: 13/16 fail, 3 pass, max gap 14125 mnats at (prompt 8, tok 8), 6 prompts differing at token 0. Golden provenance also checks out: Three things before this can land. 1. It would turn
The guard is behaving correctly: the existing 2. The gate does not fail safe by device.
Related, and cheap: the checkpoint-absent path at 3. The RED anchor does not have teeth against the defect it names. The pass/fail decision is The shape that works is to land the gate with GREEN goldens alongside #570, keeping the RED capture as recorded evidence in the spec and the parity ledger, where it already lives in full. One more worth fixing while you are here: the anchor-drift message ends "— re-run qwen3-neartie-gap.py to refresh the gap golden". After #570, that Numbers to correct in the prose: the six token-0 divergences are 0.375/0.750/0.750/0.938/1.000/1.062 nats, so the max is 1.062 not 1.188, and only 5 of the 6 are over the 0.5-nat band. And On CI: No AMD hardware here, so nothing was executed on ROCm; findings 1–3 are static, and the verdict arithmetic above came from your committed files. |
5efcebc to
a77a3b3
Compare
|
All three findings accepted and reworked (commit a77a3b3, rebased onto current main):
Gate on gfx1100: 16/16 PASS (15/16 strict, 1/16 band, max gap 0.125 nats); 0.6B gate re-run green. |
a77a3b3 to
30059ac
Compare
…lden lane in the Qwen3-0.6B near-tie gate Issue mudler#41 M4 milestone. The oracle is upstream vLLM AT THE PIN (555967922, runtime 0.23.1rc1.dev1511+g555967922 -- verified == the parity-pin block) built from source as a ROCm wheel for gfx1100 in rocm/vllm-dev:base and committed as the image vllm-rocm-oracle:555967922-gfx1100 (recipe: .agents/specs/rocm-m4-oracle.md). Gate result on this box (4x RX 7900 XTX, ROCm 7.14, enforce_eager, batch=1, flock ~/gpu.lock): oracle K=10 per-prompt greedy deterministic in every cell; our ROCm engine 16/16 prompts PASS -- 11/16 strict token-exact vs base greedy, 5/16 near-tie band, max teacher-forced gap 0.125 nats, 0 forward-divergent, 125/125 assertions, backend proof all 8 Qwen3-dense ops selections>0 and declines==0 on device 5. The p0 France/Italy flip is a literal 0.0000-nat tie in the oracle's own logits (the Metal M3b signature, reproduced on ROCm). Change shape: the parity gate's Metal/Tenstorrent device-golden lane gains the kROCM arm (identical anchor+band logic, ROCm's own oracle-backed golden pair); dgx base goldens byte-untouched; the *_rocm goldens are additive. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
…he 0.8B GDN forward divergence, root-caused and fixed Issue mudler#41. The ROCm dispatcher for the fused full-attention preamble keyed its template on the SOURCE dtype (qgate.dtype) instead of the OUTPUT dtype like the CUDA lane (LaunchAttnPreambleOut switches on q_out.dtype). The 0.8B bf16 model runs bf16 QKV-projection output + f32 q/k/gate outs (the f32-attention path; no FA-2 on ROCm), which the src-keyed dispatch mis-launched as all-bf16 -- writing bf16 bits through the f32 out pointers. Evidence chain (gfx1100, flock): CPU==pinned-oracle token-exact / ROCm diverges at tok0; VT_DUMP_ACT per-layer residual dumps: layers 0-2 clean (<=9.6e-3), layer 3 (first full-attn) 2.1e-1; sub-stage dumps: block_out 4.4e-1 with post_input_norm clean; VT_DUMP_ATTN op dumps: fa0_q rms-rel 1.196 with qkv/gate clean; VT_ROCM_ATTN_CPU_REF=1 unchanged. Fix: dispatch on q_out.dtype; bf16-out requires bf16-src (VT_CHECK). Teeth: the cross-device AttnQkNormRopeGate case gains the bf16-src -> f32-out combo at the real 0.8B dims (Hq=8, Hkv=2, Dh=256, rot=64); mutation-proven (fails with the fix reverted, passes with it). Gates on this branch (gfx1100, flock ~/gpu.lock): test_backend_cross_device 19/19 (346 assertions, incl. the new combo); test_qwen3_paged_engine 16/16 unchanged; e2e 0.8B 'The capital of France is' -> ' Paris.\nThe capital of France is' == pinned oracle == our CPU. The M4 0.8B gate with the re-derived GREEN goldens lands in the stacked sibling on row/ROCM-M4-GDN-GATE; full HIP ctest zero-delta vs base measured there. Also: three pre-existing hipcc/clang-only build breaks that block ANY full HIP build on this toolchain: same-line double CAPTURE in test_qwen3_5_gdn_spec_routing.cpp and test_ops_quant_repack.cpp, and -Wunused-const-variable on kGdnLayers (uses compile out on HIP). Debug instrumentation env-gated and off by default (VT_DUMP_ACT / VT_DUMP_ACT_SUB / VT_DUMP_ATTN; deepseek_v4 VT_DUMP_ACT precedent; env-doc-allowlist updated). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
…e, landed green-shaped with the dispatch fix stacked below Issue mudler#41 M4, second model. NEW tests/parity/test_qwen35_paged_engine.cpp -- the GDN sibling of the Qwen3-dense SACRED gate: the standard 16-prompt battery through the full paged engine, anchor + <=0.5-nat near-tie band identical to the dense gate, backend proof over the GDN op set. Goldens captured from the pinned vLLM-ROCm oracle on gfx1100 (K=10 deterministic in every cell) with a manifest.json (oracle identity + pinned HF revision 2fc06364 resolved via NEW parity::Qwen35_08BSnapshot; check-snapshot-pins clean). VERDICT: 16/16 PASS (15/16 strict token-exact vs the oracle, 1/16 near-tie band, max gap 0.125 nats, 0 forward-divergent) with the stacked AttnQkNormRopeGate output-dtype fix below. The PRE-FIX capture (13/16 forward-divergent, max gap 14.125 nats) is retained as evidence in the spec and parity ledger, not as committed goldens no code can pass. Post-review shape (localai-bot sweep): the manifest op joins PendingRunnerOps() (the qwen36_gguf_greedy precedent -- a runner-less manifest op turned main's CPU CI red); the gate exits 77 (CTest Skipped) on any non-ROCm device and when the pinned checkpoint is absent, instead of comparing a foreign engine against ROCm-derived goldens or printing SUCCESS with zero assertions; and the anchor-drift REQUIRE now reads REGRESSION SUSPECTED -- re-deriving goldens is the last step of a justified re-capture, never the response to a failure. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
30059ac to
c8ec69e
Compare
… validates an outside contributor's trailers (#773) (#782) Closes #773. `check-pr-size.py` and `check-commit-trailers.py` both required the base revision to be an ANCESTOR of head. CI passes `github.event.pull_request.base.sha`, the TIP of the base branch, which stops being an ancestor the moment main advances after the branch was cut -- continuously, on this repo. Measured on three open PRs before changing anything. Base is not an ancestor in any of them, and a merge base exists in all three: #506 ancestor=NO merge-base=e1087a881 #523 ancestor=NO merge-base=fdd452637 #559 ancestor=NO merge-base=fafa16f0f The consequence was not a noisy red check. Both checkers aborted BEFORE examining anything, so CI had never validated commit trailers on an external contribution: the gate enforcing FOLLOWING_AGENTS_PROTOCOL and Assisted-by: exited before reading a single commit. Across the external PRs reviewed this week, hand-checking by a reviewer was the only verification those trailers received. pr-size aborted identically, so path classification and the checker-evidence contract went unenforced on forks too. THE FIX. Diff from the merge base, which is what a pull request IS: `git diff A...B` is defined as `git diff $(git merge-base A B) B` and is what GitHub shows. Two-dot diffing against a moved main is not merely stricter, it is WRONG -- main's own commits render as reversions inside the contributor's diff, so paths they never touched get classified and charged to them. The new pr-size test asserts both halves: the PR's file present, main's absent. `executable_evidence` gets the same treatment, since the BASE version of a checker for the red-before half is the one at the merge base. WHAT DELIBERATELY DOES NOT MOVE. The old rule conflated two situations: ordinary divergence (merge base exists) now examines merge_base..head; unrelated histories (no merge base) STILL RAISES. Absence of information must never look like absence of work -- the script's own require_origin_main() docstring already states that principle for the other input. test_missing_and_nonancestor_objects_fail_closed uses an ORPHAN branch, so it still raises; only its regex changed, because the message now names what is actually wrong. Its assertRaises(ValueError) is untouched. The trailers case was SPLIT, not deleted: its divergent-branch half built two branches off a common root -- which share a merge base and are the ordinary shape of every PR -- so that half now asserts it validates, with the genuinely-unrelated case asserted separately. Nothing that used to fail closed stopped failing closed. Range changed, contract unchanged: test_a_bad_trailer_in_the_merge_base_range_is_still_reported puts a trailerless commit inside the new range and requires it still be reported. RED before on the unmodified checkers, GREEN after (74 passed, 148 subtests). Stop conditions checked individually rather than inferred. Full tests/scripts: 9 failed / 1359 passed, all nine pre-existing and reproduced on main. CI: agent-record and pr-size both SUCCESS on this PR -- the two checkers it repairs passing on a live PR. Remaining red is baseline only: windows-msvc-* are the PR-only arm (#584), and sanitize-cpu is red on main itself for #775 (test_nemotron_h_scaffold, nemotron_h_registry.cpp:112 downcasting a doctest StubModel to NemotronHLoadedModel). This PR touches no C++.
…d of aborting the pass (#776) (#853) Closes #776. `test_op_parity` THREW instead of failing an assertion: tests/parity/test_op_parity.cpp:1989: ERROR: test case THREW exception: [json.exception.type_error.302] type must be string, but is null The malformed input was tests/parity/goldens/minimax_music3_oracle/manifest.json -- an oracle capture with no "op" key at all, the only non-op manifest among the 66 committed. Attributed decisively rather than by inference: rebuilding the pre-fix source and moving that one directory aside turned the same binary green, 70 assertions -> 123. The throw was hiding a third of the gate. #776 IS A DUPLICATE, AND THAT IS NOT THE END OF IT. #755 is the same defect, closed by 043e568 (#761) eight minutes after #776 was filed, and the manifest is already repaired on main. Reporting only that would have been accurate and useless, because #761 closed the walker's INPUT SET and not its EXCEPTION SURFACE. Reproduced on the FIXED tree by nulling one tensor dtype in rmsnorm_f32_8x128: :2177: ERROR: test case THREW exception: [...type_error.302...] [doctest] assertions: 37 | 37 passed 37 where 153 should run -- 45 committed goldens never reached, on a tree where this was supposedly closed. The gate was one bad field away from silently not gating. WHY THAT MATTERS: test_op_parity is the guard that refuses a golden whose manifest names an op with no registered runner. It is what caught PR #559's missing runner arm. A throw at manifest-read time aborts the case, so every golden after the malformed one goes unchecked -- not merely a red job, a gate that had stopped gating. THE FIX, both halves. The two remaining throw sites (json::parse on the manifest, and any runner field read) funnel through GuardGoldenStage, which converts a std::exception into a FAIL_CHECK NAMING goldens/<case>/manifest.json, quotes the original exception, and continues to the next golden. The dispatch chain moved into RunGoldenCase so its `continue`s became `return false`, which is what permits a per-case catch. std::exception is the deliberate width: doctest::detail::TestFailureException is NOT derived from it (third_party/doctest/doctest.h:2563). That is load-bearing -- the "no runner for op" refusal is a FAIL, and a wider catch would have cleared the red while REMOVING the gate. MUTATION EVIDENCE, tree restored byte-for-byte each time (823 files, md5sum -c): dtype: null in a real golden -> named by path; 45 op cases STILL RAN, 152 assertions still checked (was: THREW, 37) invalid JSON manifest -> named by path; 46 cases ran, no THREW {"op": null} / missing "op" -> refused by name, pass continues {"op": "no_such_runner_op_559"} -> FATAL ERROR: no runner for op ... -- the #559 guard STILL BITES The last row is the one that mattered; without it the others would only prove the red was silenced. GREEN: focused 13 cases / 153 assertions, with 46 op cases + 1 non-op skip -- counts IDENTICAL to before, so the extraction changed nothing about what is gated. sanitize-cpu (address,undefined) rebuilt with the job's own flags and run under its env: SUCCESS. pytest tests/scripts: 1400 passed, 9 failed, all three distinct failures reproduced on a clean origin/main worktree. Survey: ~167 unguarded m.at(...) reads live inside runners in this file and the guard covers them generically. Other json::parse sites in tests/ each read one named fixture owned by their own case, so a throw there fails one already-named test rather than a walker over an open input set. No follow-up owed. CI: merged with checks queued -- the runner pool has been saturated for hours and no job started on this head. The sanitize lane, which is the one this change exists for, was reproduced locally with that job's own build flags and environment. Note main moved mid-gate: #840 relocated the issue table out of roadmap_v1.md into the append-only .agents/issue-index.md. roadmap_v1.md was taken from origin/main wholesale and the #776 row reapplied as an append to the new index.
M4 for Qwen3.5-0.8B (GDN) — the oracle-backed gate, landed green-shaped (issue #41)
Stacked: #549 (0.6B lane + oracle) ← the dispatch fix (#570) ← this gate. The fix lands below the gate, so every commit in the stack is individually green on the gate host and the committed goldens are ones the tree can actually reproduce.
Gate
test_qwen35_paged_engine.cpp— the GDN sibling of the Qwen3-dense SACRED gate: standard 16-prompt battery through the full paged engine, hard anchor + ≤0.5-nat near-tie band, backend proof over the 15-op GDN set. Goldens captured from the pinned vLLM-ROCm oracle on this box (K=10 per-prompt, deterministic in every cell) withmanifest.jsonrecording the oracle identity; the checkpoint resolves viaparity::HfSnapshotat the pinned HF revision2fc06364(check-snapshot-pins clean).Verdict: 16/16 PASS (15/16 strict token-exact vs the oracle, 1/16 near-tie band, max gap 0.125 nats, 0 forward-divergent).
Review-sweep rework (all three findings accepted)
qwen35_0_8b_greedyis inPendingRunnerOps()(theqwen36_gguf_greedyprecedent) — the manifest itself stays; its oracle identity is the point.Prose corrections from review
First-divergence maximum is 1.062 nats (not 1.188), 5 of the 6 token-0 divergences are over the band (not all), and the headline gap is at the first over-band position rather than a post-divergence maximum.