record(dspark): the oracle varies 20% between invocations — parity is UNMEASURED (#442) - #855
Merged
Merged
Conversation
…is UNMEASURED (#442) The environment was rebuilt from nothing after the reimage: our engine (verified full fast-path, sm_121a), both checkpoints at the pinned revision, and the TRUE pinned oracle 0.23.1rc1.dev1511+g555967922 with torch 2.13.0+cu130 and flashinfer 0.6.15.post1 -- built from source because vllm==0.26.0 HARD-PINS torch==2.11.0 and flashinfer 0.6.14, which would have made it a different denominator rather than a close one. On that stack our engine measures more stably than at any point before the reimage: four independent arms at 141.9 / 140.7 / 141.8 / 142.3 tok/s, about 1% total spread, medians within 0.4%. THE ORACLE DOES NOT. Two invocations, same boot, same stack, same prompt, minutes apart: median 145.4 (bimodal 145/154) then 174.3 (tight). That is 20%, and it decides the answer -- the ratio is 0.98 against the first and 0.82 against the second. The obvious explanation is refuted. torch.compile and flashinfer caches live at /root/.cache INSIDE the container and are discarded by `docker run --rm`; the host's ~/.cache/vllm stayed at 24K across all invocations. Both runs paid full JIT. The spread is unexplained, not benign. The confirmation probe could not settle it either: invocations 3 and 4 both died RC=1 with flashinfer emitting two FAILED compiles for gen_gemm_sm120 kernels. WHY IT MATTERS BEYOND THIS RUN: every ratio this row has recorded -- 0.9757, 0.9646, 0.9569, 0.9889 -- used exactly ONE oracle invocation per paired run. The drift gate this row applies to OUR arms has no counterpart on the denominator. Those ratios are not thereby wrong, but their error bars are unknown and wider than the +/-0.01 the record has been quoting. Owed before any further parity claim: N>=5 oracle invocations per pair with a dispersion gate on the denominator; an explanation for the flashinfer gen_gemm_sm120 JIT failures, since a partial fallback changes which kernels the oracle runs; and persisted container caches so JIT is not re-paid per invocation, then a re-test of whether the spread survives. Parity is UNMEASURED on the rebuilt stack. Not 0.98, not 0.82 -- unmeasured, because choosing between them is choosing the answer. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…atio is 0.835 (#442) Two corrections to the entry this branch already carries, and the second one was caught before it reached a headline. THE CACHE HYPOTHESIS WAS RIGHT. That entry declared it REFUTED on the grounds that the host cache never grew. Wrong check: `~/.cache/vllm` on the host stayed at 24K because the cache that matters is `/root/.cache` INSIDE the container, which `docker run --rm` discards. Mounting it settles the question -- with the cache persisted the oracle stops moving, four invocations at 174.8 / 175.3 / 175.9 / 175.1. The 145.4 reading was cold JIT; 174.3 was the warm truth. THE SECOND ERROR WOULD HAVE BEEN WORSE. Our arm generates 89 tokens (finish_reason=stop); the oracle runner I wrote after the reimage generated 128. Longer generations amortise the same prefill over more decode steps, so 142/175 = 0.81 compared two different workloads. The pre-reimage fibacc.py used 89, matching ours; the mismatch was introduced when the reimage destroyed it and I rewrote it from a template. MATCHED AND WARM, both arms interleaved in one run: oracle 171.43 tok/s over 10 reps, ours 143.22. RATIO 0.835. That is far below the 0.9757 / 0.9646 / 0.9569 / 0.9889 on record, and this data cannot say which of two causes is responsible: 1. Every earlier paired run invoked the oracle ONCE. A cold-JIT denominator would have been handicapped ~17%, flattering our engine. 2. THE BOX IS NOT THE SAME MACHINE. dgx.casa now resolves to kairos-17dd (machine-id 17dd5b4e3f38452d); the recorded ratios were taken on promaxgb10-4ad8, which no longer exists. Same GB10 class, different host, and this platform has a documented 12.8% boot-to-boot clock swing. Our arm reads ~142 on both machines, which argues against a pure hardware explanation without excluding one. SAFE: on kairos-17dd, matched and warm, we are at 0.835 of the pinned oracle. NOT parity, and materially worse than this row has been reporting. NOT SAFE: concluding the older numbers were wrong. They came from a machine that no longer exists, and cross-machine ratios cannot be differenced any more than cross-boot absolutes can -- the same rule this file applies elsewhere. OWED: re-run the pre-reimage protocol (single cold oracle invocation) on THIS box. ~0.97 convicts cold JIT and every recorded ratio needs revising; ~0.83 means the machine changed and the old numbers stand for the old box. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
Records-only branch; clean merge, no overlap with this row's edits. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
Records-only branch; clean merge, no overlap with this row's edits. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records-only. The environment was rebuilt from nothing after the reimage, and the rebuild surfaced something that affects every ratio this row has recorded.
The rebuild — our engine (verified full fast-path,
sm_121a, 61dsparksymbols), both checkpoints at the pinned revision491c2f1e…, and the true pinned oracle0.23.1rc1.dev1511+g555967922with torch 2.13.0+cu130 and flashinfer 0.6.15.post1, built from source. That last part was necessary, not fussy:vllm==0.26.0hard-pinstorch==2.11.0andflashinfer-python==0.6.14, while the pin's ownpyproject.tomlrequirestorch==2.13.0. The release wheel is a different denominator, not an approximation.Our engine is stable. Four independent arms: 141.9 / 140.7 / 141.8 / 142.3 tok/s — ~1% total spread, medians within 0.4%, tighter than anything measured before the reimage.
The oracle is not. Two invocations, same boot, same stack, same prompt, minutes apart:
That is 20%, and it decides the result: 0.98 against invocation 1, 0.82 against invocation 2.
The obvious explanation is refuted. A compile-cache warming story fails:
torch.compileand flashinfer caches live at/root/.cacheinside the container and are discarded bydocker run --rm, and the host's~/.cache/vllmstayed at 24 K across all four invocations. Both runs paid full JIT. The confirmation probe couldn't settle it either — invocations 3 and 4 diedRC=1with flashinfer emitting twoFAILEDcompiles forgen_gemm_sm120kernels.Why this matters beyond one run: every ratio recorded on this row — 0.9757, 0.9646, 0.9569, 0.9889 — used exactly one oracle invocation per paired run. The drift gate applied to our arms has no counterpart on the denominator. Those ratios aren't thereby wrong, but their error bars are unknown and wider than the ±0.01 the record has been quoting.
Owed before any further parity claim: N≥5 oracle invocations per pair with a dispersion gate on the denominator; an explanation for the flashinfer JIT failures, since a partial fallback changes which kernels the oracle runs; and persisted container caches, then a re-test of whether the spread survives.
Parity is UNMEASURED on the rebuilt stack. Not 0.98, not 0.82 — unmeasured, because choosing between them is choosing the answer.
Gate:
scripts/agent-preflight.sh --staged— All gates green.FOLLOWING_AGENTS_PROTOCOL