Distillation GPQA-D: 62.63 with the public 4k setup vs 68.7/70.0 #26

Thank you for releasing RecursiveMAS. With repository commit 38f7da45, the public Distillation checkpoints, and the release-recommended 4000-token GPQA configuration, I obtain 62.63 +/- 2.79 over generation seeds 42-46 (620/990 pooled; best run 65.66).

This does not reproduce 70.0 in Distillation Table 6 of the paper, or the 68.7 single-run reference in the README.

Repository-derived single-model controls also show a small gap between released MAS and the 4B Learner, while both remain below the 9B Expert. Public-code Link retraining is strongly checkpoint-sensitive, peaking at step 5000 and degrading by the final step.

Released-checkpoint results

The paper reports GPQA-D scores of 72.7 / 61.4 / 70.0 for Expert / Learner / RecursiveMAS, averaged over five runs. It specifies frozen base LLMs, batch size 4, maximum sequence length 4096, temperature=0.6, top_p=0.95, and 4000 GPQA generation tokens. The README labels 68.7 as a single-run reference and notes that public checkpoints do not replace paper task-specific setups.

I used the public Expert, Learner , and Outer Link snapshots from huggingface. Evaluation used all 198 gpqa_diamond examples, fixed dataset/option seed 42, generation seeds 42-46, batch size 16, 4000 output tokens, latent length 64, three rounds, temperature=0.6, top_p=0.95, top_k=-1, release answer recovery, thinking disabled, and the official Distillation Transformers module.

No locally trained weights were used for the 62.63 result. Saved outputs contained no empty parsed answers after release retry/fallback handling.

9B / MAS / 4B diagnostic comparison

I also ran more control experiments. The direct-model controls below reuse repository interfaces for GPQA loading/option construction, generation, answer retry, parsing, and accuracy, with the same generation configuration. They use a common reduced direct-answer prompt and replaced the latent MAS pipeline by single model generation. These are diagnostic controls, not official paper baselines or an unmodified run.py mode.

Under this diagnostic shell, MAS added only 2/792 correct answers over 4B and remained 31/792 behind 9B. Prompt choice can affect direct-model performance, so this is a narrow component comparison rather than a replacement for paper baselines.

Public-code Link retraining

Then I retrained Links from native Qwen3.5-9B/4B backbones using RecursiveMAS/Distillation-Math (1904 rows, revision ef31013eb54ae6b6ab18e517d6ce720f62463462) and the same commit. Backbones were frozen. Training explicitly used batch size 4, maximum sequence length 4096, bfloat16, seed 42, learning rate 5e-4, cosine scheduling, three Outer rounds, final-round-only supervision, latent cap 80, and 20000 optimizer steps.
These links are used for MAS evaluation.

After step 5000, GPQA-D declined by 4.14 points while logged loss continued to fall. The 63.33 value is diagnostic, not an unbiased held-out result, because GPQA-D was inspected when selecting it. The evidence is checkpoint sensitivity, not a claim of outperforming the release.

Request

1. Exact commands, seeds, backend, package versions, and checkpoint revisions for paper 70.0 and README 68.7 are not provided.
2. README training examples omit batch size and sequence length; public defaults are batch size 2 and Inner maximum length 2048, unlike paper values 4 and 4096.
3. The validation set, early-stopping rule if exists, and selected Inner/Outer training steps are undocumented.

Could you please provide the task-specific GPQA-D Inner and Outer Link checkpoints used for Table 6, together with the exact inference command and checkpoint-selection configuration? Since, these things are quiet important but ambiguous in both github repo and paper. A pinned environment and per-run predictions would also be very helpful.

Environment

My environment was Ubuntu 24.04, RTX 4090 cards, driver 610.43.02, Python 3.12.3, PyTorch 2.11.0+cu130, and Transformers 5.13.0. This differs from the repository pins and paper H100/A100 environment, so I am not claiming environment identity, and if there may be anything wrong with the environment?

I am focusing on Agent Communication, and I am genuinely interested in RecursiveMAS and would like to understand and build on the reported results. Releasing the task-specific Links would enable a straightforward reproduction and separate checkpoint-reconstruction issues from inference-configuration issues.