Fix AA-Briefcase-Lite judge routing and subtitle transport - #3334
Draft
AdamRajfer wants to merge 6 commits into
Draft
Fix AA-Briefcase-Lite judge routing and subtitle transport#3334AdamRajfer wants to merge 6 commits into
AdamRajfer wants to merge 6 commits into
Conversation
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Restore the original preparation implementation while validating the existing benchmark through an actual local Gym run. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Use a fixed-model adapter per judge and align panel reasoning settings. Pass SRT contents through the shared bounded text reader and override Claude's output budget to avoid reproduced truncation. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
AdamRajfer
force-pushed
the
arajfer/briefcase-lite-preparation
branch
from
September 12, 2026 04:30
60c8fc7 to
c2627a2
Compare
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the existing Lite judging path and makes the native Gym benchmark run both scoring modes:
Validation: 104 focused tests passed, one optional-dependency skip; scoped pre-commit passed. Regression tests reproduce the routing/subtitle/format defects before their fixes. Real cached-artifact judging completed one four-task set with all 55 binary checks and 16 pairwise trials valid, including seven Claude pairwise trials. The documented run command also passes native CLI/config validation.
Still draft: the local 16,384-token Claude override is not a complete budget fix. A second four-task set finished with only one valid task: empty Claude answers are recorded in report analytical-quality and workbook-presentation pairwise checks, and presentation binary A-13. Persisted text does not establish their stop reason. Historical binary truncation also remains unresolved; one complete rerun does not prove it eliminated. No further cap, effort, timeout or brevity change is included.
These are public Lite checks plus a local pairwise approximation, not official AA-Briefcase Elo. No new policy generation was needed for this validation. Dataset preparation, sandbox, rubrics and judge sampling are unchanged. TEX/PDF recompilation is deferred. Private artifacts, credentials and deployment configuration are excluded.
References: AA methodology, subtitle rubric.