All pull requests

Pull requests

• Store exact per-instance IFEval metrics

  #509·no0p opened Oct 25, 2026Contributor·

  ·

• Release the vLLM engine when an evaluation worker closes

  #508·no0p opened Oct 25, 2026Contributor·

  ·

• Add an opt-in MATH-500 delimiter correction

  #507·no0p opened Oct 25, 2026Contributor·

  ·

• Add an inline SGLang generation provider

  #506·no0p opened Oct 25, 2026Contributor·

  ·

• Add SciReas-Pro scientific reasoning tasks with a fixed 500-item sample

  #503·armancohan opened Oct 22, 2026·

  ·

• [WIP] add screenspot and screenspot_v2 GUI grounding tasks for OLMo 3.5 midtrained checkpoints

  #501·peter-sushko opened Oct 19, 2026·

  ·
  1

• Record per-task scoring time

  #499·gregorybchris opened Oct 17, 2026Collaborator·

  ·

• Attribute inference worker time to every batch and agent rollout

  #498·gregorybchris opened Oct 17, 2026Collaborator·

  ·

• Record the vLLM venv's package versions in the run manifest

  #496·gregorybchris opened Oct 17, 2026Collaborator·

  ·

• Refuse a code-execution task under a harness without sandboxes

  #491·gregorybchris opened Oct 17, 2026Collaborator·

  ·

• Add merge and stacked-PR rules to AGENTS.md

  #488·abhishekraok opened Oct 16, 2026Collaborator·

  ·
  1

• Avoid redundant quadratic scans in answer extraction

  #485·abhishekraok opened Oct 15, 2026Collaborator·

  ·
  6

• Recover quarantined sandbox slots with bounded retries

  #471·abhishekraok opened Oct 13, 2026Collaborator·

  ·
  23

• Isolate sandbox commands from the SWE-ReX server

  #470·abhishekraok opened Oct 13, 2026Collaborator·

  ·
  29

• Mark unavailable sandbox scoring incomplete

  #469·abhishekraok opened Oct 13, 2026Collaborator·

  ·
  13

• Keep GAP metrics consistent in comparisons and progression

  #466·abhishekraok opened Oct 13, 2026Collaborator·

  ·
  26

• Keep thinking models' reasoning across agent turns on vLLM

  #463·gregorybchris opened Oct 13, 2026Collaborator·

  ·

• RULER: add a :chat variant and ruler_all_chat suites

  #460·abhishekraok opened Oct 11, 2026Collaborator·

  ·
  5

• [Do not merge] OLMo 3.5 stage-1 evaluation: olmo_core_vlm for OLMoDDP/KDA checkpoints (our working branch)

  #458·jason718 opened Oct 9, 2026Contributor·

  ·

• Share the long-context task base, task registration and JSONL/Hub helpers between RULER and HELMET

  #455·jopetty opened Oct 6, 2026Contributor·

  ·
  1

• Resolve named primary metrics to their Metric instance

  #453·jopetty opened Oct 6, 2026Contributor·

  ·
  1

• Add thinking_prefilled: forced-think outputs without </think> are unclosed (on #389)

  #451·abhishekraok opened Oct 5, 2026Collaborator·

  ·
  6

• Add the PixelRAG reader benchmark (Wikipedia QA over retrieved screenshots)

  #447·jason718 opened Oct 3, 2026Contributor·8 of 13 tasks·

  ·

• Add ChartMuseum, μ-Bench, SPUR, SciMDR-Eval and GEOBench-VLM multimodal benchmarks

  #446·jason718 opened Oct 3, 2026Contributor·6 of 9 tasks·

  ·

• Add vLLM image support and prompt/image ablation knobs (port of #347 to main)

  #439·donovanclay opened Oct 1, 2026·9 of 13 tasks·

  ·