perf(rocm): head_dim=128 decode arm -- the ROCm half of #382, default OFF - #767

Implements .agents/specs/rocm-decode-attn-d128.md, which landed ahead of this change as #564.

The ROCm half of #382. The CUDA half merged as #425 (66399617); this mirrors it, adopting that arm's flag, default and stated reason rather than inventing new ones.

What changes

d == 128 — the Qwen3-dense / Llama / Mistral head size — reaches the fast decode kernels instead of falling through to the generic PagedAttnOnline. LoadRowEplBf16/StoreRowEplBf16 gain an EPL=4 (uint2) case beside the existing EPL=8/16; the dispatch gates and the two launch switches gain a d == 128 arm. No new kernel and no new algorithm — the kernel bodies were already generic over EPL.

Default OFF, opt in with VT_ATTN_DECODE_D128=1 — the same env var, default and reason as the merged CUDA arm. The arm is correctness-complete but not byte-exact against the kernel it replaces: warp-strided online softmax reduces the KV sequence in a different order than PagedAttnOnline's per-tile loop, so a greedy anchor can move at an exact bf16 tie. Shipping OFF keeps every existing golden byte-identical. The flip owes the near-tie razor, a distributional gate and regen under the ratified-tie rule, and per the spec must be argued per backend — see the reversal below. That is what keeps #382 open.

Reviewer note

Spec §4 item 3 writes the gate as (d == 128 && (decode_d128 || decode_wmma)). This commit implements it without the decode_wmma disjunct, which is what the same item's "Forward reference" paragraph instructs: VT_ATTN_DECODE_WMMA does not exist in the tree, and the flag lands with the rocWMMA arm on its own branch. The difference is intentional; it is visible in the diff before the note explaining it is.

Evidence

gfx1200 (RX 9060 XT, RDNA4, 32 CU), ROCm 7.2.3, $GPU_LOCK held. All figures are a same-binary flag A/B — no rebuild between arms — at 1024-token synthetic prompt, 128 generated, greedy, seed 0, 2 reps per cell agreeing within ~1%.

Qwen3-4B has no GQA fusion at any head_dim, so its 2.09x isolates the EPL widening from the fusion.

Qwen3.5-0.8B is the negative control and it earned its keep. Its head_dim is 256, so the d == 128 gate provably cannot reach it. Its first OFF rep came in a 33% outlier at 31.14 ms, which a blind 2-rep average would have reported as a ~1.2x "win" for a model the flag cannot affect. Re-run three times: 23.86 / 23.75 / 23.68 against ON's 23.52 / 23.57.

End-to-end output throughput rises less than TPOT on the same runs (0.6B 2.48x, 1.7B 2.05x, 4B 2.02x) because they carry a 1024-token prefill the flag does not touch. TPOT isolates decode; throughput dilutes it.

Concurrency — the advantage grows, it does not compress

Qwen3-1.7B, --num-prompts = 2x concurrency:

This refuted the prediction made before the run, which reasoned that a tiny grid at concurrency 1 flatters the fast kernel. The dominant effect is the reverse: from c8 to c16 the fallback scales only 1.33x against the arm's 1.54x, and scaling efficiency at c16 relative to perfect-linear-from-c1 is 38% OFF against 58% ON. PagedAttnOnline is the batch-scaling bottleneck, not merely slow per call, so the win is largest in the regime a server actually runs in.

The c1 row reproduces an independently-run four-model sweep to within ~1% (52.85/21.93 there vs 53.40/22.26 here).

Correctness

• ctest -R 'rocm|cross_device' 5/5, chained directly to the exact-SHA push.
• New case: "paged attention at Qwen3 geometry (bf16, GQA 2, head_dim 128) matches the CPU oracle", looped over RegisteredDevices(), NMSE <= 5e-4 vs the CPU oracle plus OpProviderStats::declines == 0. Genuinely new coverage — the existing generic cross-device test runs at d=8, f32 and never reached any bf16 EPL-templated kernel, so none of them had bf16 coverage in this suite. (The merged CUDA arm shipped with no test at all.)
• Because the arm ships OFF and its flag is read into a static const bool — once per process — the default registration can only ever gate the fallback. tests/CMakeLists.txt adds a second invocation with the flag set, same shape as the existing test_dense_gateup_fused_marlin_off_* pair. Verified non-vacuous against the test_voxtral_e2e reports SUCCESS with zero assertions when its weights env var is unset (and ~40 more gates share the shape) #463 trap: 1 case, 6 assertions, not zero.
• Full ctest 448/455. The 7 failures are proven pre-existing, not asserted: a clean main 2784dd7b worktree built from source with none of this code fails the identical set (only test_op_parity's index shifts 403 -> 404, from the added registration). They are a missing shellcheck, an mmap-RSS assertion, a JSON type error, and the SharedExpertGate ROCm registration gap owed to unmerged feat(rocm): MoE combine/gate ops — SharedExpertGate, MoeCombine, MoeCombineGate (issue #41) #509.
• agent-preflight fails 9, a strict subset of that same baseline's 10 (differing only by role-undeclared). check-commit-trailers and check-doc-checkpoint both pass against this base.

Carried finding

#382 measured this same EPL=4 arm 1.6x slower on sm_110 / Jetson AGX Thor, where gfx1200 measures it 2-3.6x faster. Recorded, not reconciled — different kernels, different fallbacks, different memory systems. It is why the default-ON flip must be argued per backend rather than once, and it is preserved in the spec rather than averaged away.

Not claimed

This does not close #488. That issue reports an 8.1x per-call gap against vLLM and explicitly asserts no cause; this removes one cause without re-measuring the residual against the oracle (blocked on a Nix-glibc vs container-glibc ABI mismatch). Also out of scope and named in the spec: the dtype gap (ROCm's decode-opt is bf16-only at every head_dim, so 4 of 5 dtype combinations still fall to PagedAttnOnline at d=128 — pre-existing, inherited, not introduced), qg=4/qg=8 fusion, d=128 prefill, and the rocWMMA arm.

Issue: #382
Spec: #564