@zanderjiang , @JiangJiaWei1103, and I are working on eliminating trainer–inference mismatch in SkyRL with IsoExec.
As discussed in the IsoExec blog post, rollout and training can produce different token probabilities even with the same model weights. IsoExec aims to eliminate these differences.
SkyRL’s current mismatch metrics compare only sampled-token logprobs: the rollout response keeps the sampled token’s logprob, and the trainer compares these against its recomputed action-token logprobs. Zero difference on those tokens does not establish that the complete output distributions match.
We would like an opt-in mode that compares the full-vocabulary logprob distributions from rollout and the trainer before any weight updates, using the same model weights and token histories.
This would provide stronger validation of trainer–inference consistency for the evaluated inputs and help debug numerical differences introduced by model, kernel, or parallelism changes.
@zanderjiang , @JiangJiaWei1103, and I are working on eliminating trainer–inference mismatch in SkyRL with IsoExec.
As discussed in the IsoExec blog post, rollout and training can produce different token probabilities even with the same model weights. IsoExec aims to eliminate these differences.
SkyRL’s current mismatch metrics compare only sampled-token logprobs: the rollout response keeps the sampled token’s logprob, and the trainer compares these against its recomputed action-token logprobs. Zero difference on those tokens does not establish that the complete output distributions match.
We would like an opt-in mode that compares the full-vocabulary logprob distributions from rollout and the trainer before any weight updates, using the same model weights and token histories.
This would provide stronger validation of trainer–inference consistency for the evaluated inputs and help debug numerical differences introduced by model, kernel, or parallelism changes.