Abstract:

Reward models (RMs) have become an indispensable fixture of the language model (LM) post-training playbook, enabling policy alignment and test-time scaling. Research on the application of RMs in code generation, however, has been comparatively sparse, with existing work largely focusing on execution feedback. This choice constrains post-training to optimizing functional correctness over self-contained executable code. In this work, we examine the training and evaluation of multilingual, multi-criteria code RMs. To this end, we first compile Themis-CodeRewardBench, a benchmark to evaluate code RMs across five preference dimensions (i.e., criteria) and eight programming languages, on which we profile 50+ code, math, and general-purpose RMs. Observing the limited proficiency of current RMs beyond scoring for functional correctness, we develop Themis-CodePreference, the largest open-source collection of code preferences to date (more than 350k preference pairs), and use it to train Themis-RM, a suite of multilingual code reward models for flexible multi-criteria scoring, ranging in size from 600M to 32B parameters. Our experiments and ablations demonstrate positive scaling trends, strong cross-lingual transfer when training on diverse preferences, and the importance of multi-criteria training for reliable code reward modeling.

Themis reward models are trained using the Bradley-Terry preference framework with a multi-stage data pipeline that mines, filters, scores, and assembles high-quality code preference pairs from open-source repositories. The models are evaluated on Code RewardBench (CRB), a benchmark of 8,866 preference pairs spanning 5 quality aspects and 8 programming languages.

Themis/
├── Dataset/            Pipeline for constructing the preference training dataset
│   ├── Commit_Mining_SQL/    BigQuery SQL for extracting single-file commits
│   ├── Repos/                Curated per-language repository allowlists (~348k repos)
│   ├── Utils/                Content retrieval, deduplication, and filtering tools
│   ├── Commit_Mining_Terms/  Aspect-specific term lists for commit classification
│   └── Prompts/              Jinja2 templates, LLM judge driver, system prompt mapper
│
├── Training/           Distributed reward model training on multi-node GPU clusters
│   ├── train_reward_model.py     Training script (FSDP2 + Liger fused kernels)
│   ├── launch_reward_training.sh Per-node launch script (container + accelerate)
│   ├── Themis.Dockerfile         Container build (PyTorch + NCCL)
│   └── fsdp2_config.yaml        Accelerate / FSDP2 configuration
│
└── Evaluation/         Benchmarking suite for 51 reward models on Code RewardBench
    ├── Evaluation_Scripts/   20 per-architecture evaluation scripts
    └── Evaluation_Runs/      Pre-computed results (results.json + scores.parquet)

The end-to-end pipeline has three phases: dataset construction, model training, and evaluation.

                          DATASET CONSTRUCTION
                          ────────────────────
  BigQuery (github_repos)
      │
      ▼
  ┌─────────────────────┐   ┌───────────────────┐   ┌──────────────────┐
  │ 1. Commit Mining    │──▶│ 2. Repo Filtering │──▶│ 3. Ext Filtering │
  │    (SQL)            │   │    (allowlists)   │   │    (lang → ext)  │
  └─────────────────────┘   └───────────────────┘   └──────────────────┘
                                                            │
      ┌─────────────────────────────────────────────────────┘
      ▼
  ┌──────────────────────┐   ┌──────────────────┐   ┌──────────────────┐
  │ 4. Content Retrieval │──▶│ 5. Deduplication │──▶│ 6. Aspect Filter │
  │    (git fetch)       │   │    (MinHash LSH) │   │    (ModernBERT)  │
  └──────────────────────┘   └──────────────────┘   └──────────────────┘
                                                            │
      ┌─────────────────────────────────────────────────────┘
      ▼
  ┌──────────────────────┐   ┌──────────────────┐   ┌──────────────────┐
  │ 7. LLM Scoring &     │-─▶│ 8. LLM-as-a-Judge│──▶│ 9. Training Data │
  │    Instruction Synth │   │    (A/B voting)  │   │    Assembly      │
  └──────────────────────┘   └──────────────────┘   └──────────────────┘
                                                            │
                          MODEL TRAINING                    │
                          ──────────────                    │
      ┌─────────────────────────────────────────────────────┘
      ▼
  ┌───────────────────────────────────────────────────────────────────┐
  │ Bradley-Terry preference training with FSDP2 on multi-node GPUs   │
  │ (BT loss + LM regularisation + magnitude penalty, Liger kernels)  │
  └───────────────────────────────────┬───────────────────────────────┘
                                      │
                          EVALUATION  │
                          ──────────  │
      ┌───────────────────────────────┘
      ▼
  ┌───────────────────────────────────────────────────────────────────┐
  │ Code RewardBench: 8,866 pairs × 5 aspects × 8 languages           │
  │ Evaluated across scalar, MoE, and generative RM architectures     │
  └───────────────────────────────────────────────────────────────────┘

Themis-RM models achieve best-in-class accuracy on Themis-CodeRewardBench, a code-specific reward model benchmark, while also matching or exceeding much larger models on established general-domain benchmarks (RewardBench V1, RewardBench V2, JudgeBench). Models are grouped by parameter class; bold marks the best in each group.

All datasets are available on HuggingFace:

The dataset pipeline mines single-file commits from GitHub's BigQuery public dataset using a modified version of the commit mining pipeline from OctoPack (CommitPack), filters them through repository reputation and language allowlists, retrieves file contents via shallow git fetches, deduplicates with MinHash LSH, classifies commits by quality aspect using term matching and ModernBERT classifiers, scores them with LLM judges, synthesises instructions, and assembles the final preference pairs with stochastic system prompts.

The SQL query restricts to permissively licensed repositories only (MIT, Apache-2.0, BSD-2/3-Clause, ISC, CC0-1.0, EPL-1.0, MPL-2.0, Unlicense, AGPL-3.0, LGPL-2.1, Artistic-2.0). The BigQuery GitHub snapshot used contains commits up to early 2022 — predating the widespread availability of LLM code generation tools — ensuring that all mined code changes represent genuine human-authored preferences. This raw pool is subsequently subset by time: training commit data is sourced no later than March 2019; benchmark commit data is scoped to June 2019 – January 2021, from disjoint repositories.

9 stages covering BigQuery extraction, repository and extension filtering, content retrieval, deduplication, aspect-based term filtering, LLM scoring and instruction synthesis, LLM-as-a-judge preference labelling, and training data assembly.

See Dataset/README.md for the full pipeline documentation, including per-stage scripts, arguments, invocation examples, and template reference.

Themis-RM models are trained in two stages using the Bradley-Terry preference framework on multi-node GPU clusters. The training script supports FSDP2 distributed training, Liger fused Triton kernels for efficient linear cross-entropy computation, and an optional LM regularisation loss.

1. Preference Model Pre-Training (PT): Trains on Themis-GeneralPreference (110k+ general-domain and code retrieval preferences) for 2 epochs to instill common human-inspired notions of preference evaluation such as relevance, helpfulness, and harmlessness.
2. Preference Modeling (PM): Trains on Themis-CodePreference (350k+ code preference pairs across 5 quality dimensions and 8 programming languages) for 1 epoch to specialize on multi-criteria code scoring.

See Training/README.md for the full training documentation, including container setup, cluster configuration, environment variables, FSDP2 config, all training arguments, launch commands, monitoring, checkpointing, and troubleshooting.

The evaluation suite benchmarks reward models on Code RewardBench (CRB) — 8,866 preference pairs across 5 quality aspects (Functional Correctness, Runtime Efficiency, Memory Efficiency, Security Hardness, Readability & Maintainability), 8 programming languages, and 19 source subsets. The suite supports 4 architecture categories across 20 scripts and includes pre-computed results for 51 models.

See Evaluation/README.md for the full evaluation documentation, including the benchmark dataset schema, evaluation protocol, per-script invocation examples, output format, and the complete results table.

Our experiments (detailed in the paper) investigate four research questions:

• RQ1 — Multi-criteria code scoring: Existing RMs are largely unusable for scoring code along non-functional axes (efficiency, security), often degenerating to random scoring. Themis-RM-0.6B outscores multiple >100x larger general-purpose RMs, while Themis-RM-32B sets a clear state-of-the-art across all criteria. Scalar reward modeling is well-suited to reference-free code evaluation — the most competitive existing RMs are all scalar RMs.

• RQ2 — Minimizing cross-criteria interference: Both the PT phase and auxiliary training losses (LM regularisation, magnitude penalty) improve performance across the board. Criteria-conditioned system prompts effectively disentangle multi-dimensional preferences, eliminating the need for criteria-specific modules, ensembling, or model merging. Training on functional correctness preferences transfers well to non-functional criteria, and positive transfer between all quality criteria means no single-criterion model matches the full multi-criteria Themis-RM.

• RQ3 — Cross-lingual transfer: Training on all eight programming languages yields the best performance, suggesting net-positive cross-lingual transfer. Python RMs transfer better to dynamically typed languages; Java RMs transfer better to statically typed ones. Training on diverse multi-criteria preferences leads to stable multilingual reward modeling with small performance differences across languages.

• RQ4 — Downstream robustness: Themis-RM achieves state-of-the-art adversarial robustness on judge-hacking perturbations and matches the best RMs specialized for correctness in listwise re-ranking of code contest solutions — a task proven to predict downstream post-training utility. Scaling trends in downstream robustness are even stronger than in pairwise accuracy.

Distributed Training Tutorial — A companion tutorial by us that walks through multi-node distributed training of scalar reward models on cloud GPU clusters. Covers cluster provisioning, high-speed networking, container management, and FSDP-based training. Useful as a standalone guide for anyone looking to reproduce the Themis training setup or adapt it to their own reward modelling workloads. Follows a simplified recipe that leverages the Axolotl framework for training reward models with the Bradley-Terry loss.

@article{themis2025,
  title={Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring},
  author={Paul, Indraneil and Gurevych, Iryna and Glava\v{s}, Goran},
  journal={arXiv preprint arXiv:2605.00754},
  year={2025}
}

Apache 2.0 — see LICENSE.