[GDPVal] Add three reference anchors to the multistage overlay - #3319
Open
agronskiy wants to merge 1 commit into
Open
[GDPVal] Add three reference anchors to the multistage overlay#3319agronskiy wants to merge 1 commit into
agronskiy wants to merge 1 commit into
Conversation
Mirrors the eval-factory refs.yaml change. The nine-ref ladder stopped at 1223 while the AA GDPval-AA v2 board runs to 1764, so a frontier candidate had no anchor above it and its elo came out of a saturated win rate rather than out of the fit. deepseek_v4_flash_0731 1468 new ceiling inkling_small 1191 qwen35_122b 925 closes the 905-992 gap Elos are the AA rows read 2026-09-11. Each resolves under GDPVAL_REFS_ROOT like the existing nine, so a local run picks them up by pointing that env var at a directory holding the same per-reference subdirectory names. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com>
yaoyu-33
added
area:environment
github-actions
Bot
added
the
sla:triage-overdue
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The reference ladder stops at DeepSeek V4 Pro / 1223, and the AA GDPval-AA v2 board now runs to 1764. A candidate anywhere near the frontier therefore has no anchor above it, and since the anchors are held fixed by the fit, its elo falls out of a saturated win rate rather than out of the comparison — a model that beats every reference on nearly every task has no evidence separating 1300 from 1600.
The September Super-3.5 campaign shows it happening: Inkling-Small fitted at 1296–1338 against this ladder, where its own AA row reads 1191.
Three anchors added, elos from the AA board read 2026-09-11:
deepseek_v4_flash_0731inkling_smallqwen35_122bdeepseek_v4_flash_0731is AA's DeepSeek V4 Flash 0731 row, not the April DeepSeek V4 Flash (1116) — the deliverables are from the 0731 release. The original nine are untouched and keep their 2026-09-09 reading.Paired with the eval-factory MR that adds the same three to
configs/benchmarks/gym/gdpval/refs.yamland the corresponding mounts. Deliverables have to be staged under the refs root before either lands in a run.🤖 Generated with Claude Code