Paper suggestion: Bayes@N for repeated-sampling evaluation #278

Paper suggestion for Resources / Evaluation Metrics:

• Title: Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation
• Authors: Mohsen Hariri, Amirhossein Samandar, Michael Hinczewski, Vipin Chaudhary
• Venue: ICLR 2026; first preprint October 2025
• Paper: https://arxiv.org/abs/2510.04265
• Implementation: https://github.com/mohsenhariri/scorio

Bayes@N estimates performance and sampling uncertainty from repeated binary or graded outcomes. The connection to code evaluation is the treatment of repeated pass/fail outcomes and the paper's analysis of Pass@k. The experiments use mathematical reasoning benchmarks, so I am proposing it as a methodological reference for code-sampling metrics, not as a coding benchmark result.

Would this fit Evaluation Metrics? A matching entry would be:

• "Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation" [2025-10] [ICLR 2026] [paper]

I checked the list and existing issues/PRs for the paper and Scorio and found no duplicate. I am an author of the paper and maintain Scorio.