Skip to content

Fix AA-Briefcase-Lite judge routing and subtitle transport - #3334

Draft
AdamRajfer wants to merge 6 commits into
aa-briefcase-lite-develfrom
arajfer/briefcase-lite-preparation
Draft

Fix AA-Briefcase-Lite judge routing and subtitle transport#3334
AdamRajfer wants to merge 6 commits into
aa-briefcase-lite-develfrom
arajfer/briefcase-lite-preparation

Conversation

@AdamRajfer

@AdamRajfer AdamRajfer commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Fixes the existing Lite judging path and makes the native Gym benchmark run both scoring modes:

  • Route each panel member through its own existing model adapter instead of silently sending every judge selection to Gemini. Match the published reasoning settings.
  • Pass SRT subtitle contents through the shared bounded text reader.
  • Request the required JSON on the first binary call using the existing format instruction; retain the fallback.
  • Select all four tasks with binary and local pairwise judging by default. Document native preparation, execution and cached-artifact rejudging. Add configuration and pairwise adapter-routing tests; no new orchestration.

Validation: 104 focused tests passed, one optional-dependency skip; scoped pre-commit passed. Regression tests reproduce the routing/subtitle/format defects before their fixes. Real cached-artifact judging completed one four-task set with all 55 binary checks and 16 pairwise trials valid, including seven Claude pairwise trials. The documented run command also passes native CLI/config validation.

Still draft: the local 16,384-token Claude override is not a complete budget fix. A second four-task set finished with only one valid task: empty Claude answers are recorded in report analytical-quality and workbook-presentation pairwise checks, and presentation binary A-13. Persisted text does not establish their stop reason. Historical binary truncation also remains unresolved; one complete rerun does not prove it eliminated. No further cap, effort, timeout or brevity change is included.

These are public Lite checks plus a local pairwise approximation, not official AA-Briefcase Elo. No new policy generation was needed for this validation. Dataset preparation, sandbox, rubrics and judge sampling are unchanged. TEX/PDF recompilation is deferred. Private artifacts, credentials and deployment configuration are excluded.

References: AA methodology, subtitle rubric.

@AdamRajfer AdamRajfer self-assigned this Sep 11, 2026
@copy-pr-bot

copy-pr-bot Bot commented Sep 11, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@AdamRajfer AdamRajfer changed the title Fix AA-Briefcase-Lite dataset preparation Fix AA-Briefcase-Lite judge routing and subtitle transport Sep 12, 2026
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Restore the original preparation implementation while validating the existing benchmark through an actual local Gym run.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Use a fixed-model adapter per judge and align panel reasoning settings. Pass SRT contents through the shared bounded text reader and override Claude's output budget to avoid reproduced truncation.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer
AdamRajfer force-pushed the arajfer/briefcase-lite-preparation branch from 60c8fc7 to c2627a2 Compare September 12, 2026 04:30
@copy-pr-bot

copy-pr-bot Bot commented Sep 12, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant