devops-bench

An executable benchmark for DevOps code. Tasks are built by damaging real infrastructure units from Helmcode/stack-v3-devops, and a candidate answer is judged by running helm, terraform and hadolint over it rather than by asking a model whether it looks right.

Every task therefore has a reference answer a human actually wrote, and a pass or fail that a tool decided.

Repair, completion and generation tasks for all seven classes of the corpus, generated and verified end to end. 202 tests.

674 tasks across 22 mutations: 275 repair, 150 completion and 249 generation. Every one of the 674 reference answers scores.

A second, opt-in network tier adds validators that download or execute. Its Terraform half is working and verified; its docker build half is specified and deliberately refuses to run until an isolating runner exists. See below.

Full tables, per class and per task type, in RESULTS.md. One pass per model over the same 674 tasks, temperature 0, no retries and no tool access:

Read each as a band rather than a point. Two passes over the same generation tasks, with byte-identical prompts and temperature 0, disagreed on 8.7% of them in both directions: continuous batching makes the same prompt a different computation depending on what else is in the batch. The noise on a 249-task slice is roughly ±4 points, which is wider than most differences anyone will want to claim. Compare per-task verdict flips, not headline rates, and never attribute a few points of movement to a code change without checking whether that change could even reach the affected class.

Two things are visible across all three. Repair and completion are near saturation while generation is not, so a future task set should weight generation harder if it is to keep discriminating. And ansible_role is the hardest class for every model, by a margin that grows as the model gets weaker: 87%, 85%, 68%. Writing a role that ansible-lint accepts turns out to be a better discriminator than anything involving Kubernetes.

the original must PASS the validator, and the mutant must FAIL it

Both halves are checked by running the tools, never assumed. Without the first you grade a model on damage it did not cause, because a quarter of real Terraform modules do not parse and 40% of charts do not render. Without the second the task is already solved and every model scores a free point.

Repair: something is wrong and must be corrected.

Completion: something is absent and must be written. The reference answer is what a human wrote, but it is not the only correct answer, and it does not have to be: the tools decide, so a different base image or a different job that validates scores just as well.

Completion needs its own guard. Validation alone cannot tell a completion from a deletion, because a chart missing its helper renders perfectly once the include call is removed, and a Dockerfile with no FROM validates once every other instruction is gone too. So a completion must also put back the file it was missing and satisfy a pattern proving the requirement was met rather than deleted.

Not every omission makes a task. Removing values.yaml from a chart was the obvious candidate and it is not one: Helm treats absent values as empty and renders happily, so the damage is invisible and nothing can be scored. Measured, not assumed.

Terraform has no completion tasks. Offline, terraform validate stops at the missing provider before it checks whether anything is declared, so an omitted declaration cannot be detected at all.

Generation: nothing is given but a specification, and the unit must be written from scratch.

This is the one task type where "did the validator pass" is nearly no signal at all. FROM alpine is a valid Dockerfile and answers nothing; a workflow with one empty job validates. So a generation task carries machine-checkable requirements alongside its prose, and both the prose and the requirements are extracted from the reference unit rather than written by hand:

Requirements are about what, never how. node:22-alpine and node:20-bookworm are the same base family; actions/checkout@v4 and the same action pinned to a SHA are the same action; ansible.builtin.package and package call the same module; a Terraform module in one file satisfies what one in three files does. Two tests hold that line: every unit must satisfy the requirements extracted from it, and a different implementation of the same job must satisfy them too. Without the second, the benchmark would be testing recall of one repository.

The specification names no file paths. Dictating them would be specifying how, and nothing would check it: the only naming that is actually enforced is whatever the validators need to find (Dockerfile, *.tf, *.yaml), so that is all the prose states.

Structural: the tool stops accepting the unit. Delete the helper a chart's include calls, unbalance a template block, remove a closing brace.

Practice: the tool still accepts the unit but a specific rule fires. Set USER to root, replace a pinned base image with latest. These are scored per rule code, because the exit code cannot see them.

The distinction is not cosmetic. A mutation whose damage no tool can see produces a task nobody can be graded on, and that is what policy.py exists to fix.

One list of operators is still deliberately not registered:

• PENDING_NETWORK_TIER: offline, terraform validate parses and then stops at "Missing required provider" before it ever checks references, so deleting a variable declaration the module still uses cannot be detected. That needs terraform init, so network access and isolation.

hadolint reports nothing whatsoever for a Dockerfile with no USER. helm has nothing to say about a workload with no resource limits. Both are real defects, both were invisible, and an entire family of tasks was unscoreable because of it: the damage could be done but never detected.

policy.py supplies the missing signal. Codes are prefixed POLICY_ so they are never mistaken for DL3002, findings are advisory and never make a unit count as invalid (a Dockerfile without USER builds and runs), and every rule is grounded in something measured across the corpus rather than invented:

Kubernetes rules run against rendered manifests, not templates: {{ toYaml .Values.resources }} in a template says nothing about whether any resources end up set, so charts are rendered first and the output is checked.

A rule that fires on sound input would be worse than no rule, so each is tested against what it must flag and what it must leave alone: USER root followed by USER nobody is correct practice, ADD of a URL or an archive is what ADD is for, and 0.0.0.0/0 in a README is not a misconfiguration.

Three of the five validators needed configuring before their verdict meant "is this unit valid", and each was measured rather than guessed:

• hadolint exits non-zero on warnings, and warnings fire on nearly every real Dockerfile, so --failure-threshold error is used and rule codes are collected separately.
• actionlint rejected 29 of 40 real workflows, and the complaints were dominated by action (63 times: actions/checkout@v2 is old, not wrong) and shellcheck (42 times: shell style inside run:). Only validity-class diagnostics gate. Its shellcheck and pyflakes integrations are switched off, because actionlint shells out to them when they happen to be installed, which would make the same workflow pass on one machine and fail on another.
• ansible-lint on its default production profile rejected 19 of 20 real roles, almost entirely on naming and fully-qualified-name style, which says nothing about whether the role works. --profile min is syntax errors and unloadable files, and rejected none of the same 20. --offline keeps it from reaching Galaxy for collections.
• helm lint is stricter than helm template and rejects sound charts for pre-existing reasons, so it is advisory.

Three tools, three different ways of saying "this is not to my taste" rather than "this is broken". It is worth assuming a validator needs calibrating until measured otherwise.

For Kubernetes manifests there is no offline tool at all: kubeconform fetches JSON schemas over the network, which would give up the property that makes this runnable on any node. manifests.py implements the subset of API validation that can be done from the document alone, which is the same subset helm lint applies: does it parse, does it declare what it is, and does the kind carry the fields the API server rejects it without.

helm template succeeds on a chart with no templates. hadolint is content with a two-line Dockerfile that builds nothing. So a candidate that deletes or empties the files it was given would score a point under the obvious implementation. Every candidate is checked against three things: it kept the files it was given, it did not gut them, and the tools accept it. tests/test_scoring.py spells out each cheat.

selftest() runs every validator against a known-good and a known-broken fixture before any task is generated, and generation is skipped for a class that fails.

This exists because of a real incident. The validators redirected HOME to a temporary directory as a hygiene measure, which disabled the tfenv shim that terraform turns out to be on the development machine. Terraform never ran, the validator reported "ok" for a sound module and for a module with an unclosed block alike, and 120 units produced zero tasks with no error anywhere. A validator that silently stops working makes every model look perfect, which is the worst failure a benchmark can have.

uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python pytest datasets

uv pip install --python .venv/bin/python ansible-lint   # for ansible_role

Tools, all offline: hadolint, helm, terraform, actionlint, ansible-lint, and docker for docker compose config, which is client-side and needs no daemon. Manifest validation needs no external tool at all. Tests skip cleanly for whichever are missing, and tools installed beside the interpreter are found without being on PATH.

# Generate and verify tasks. All three task types are produced by default,
# balanced against each other; --task-type narrows it.
.venv/bin/python generate_tasks.py \
    --config dockerfile --config helm_chart --config terraform_module \
    --config manifest_set --config workflow --config ansible_role --config compose \
    --sample 200 --limit 100 --timeout 180 --out tasks

# Every reference answer must score as a repair
.venv/bin/python verify_tasks.py

Running a model over the set, against any OpenAI-compatible endpoint:

NAN_API_KEY=sk-… .venv/bin/python evaluate.py \
    --model qwen3.6 --base https://api.example/v1 --concurrency 6

# Split the failures into ones the model owns and ones the harness owns
.venv/bin/python analyse.py results/qwen3.6-*.jsonl

# Regenerate the reference table from every complete run in results/
.venv/bin/python compare.py > RESULTS.md

That split is the part worth reading. An aggregate rate that silently counts unparsed replies as wrong answers is measuring the parser, so analyse.py reports the readable-answer rate alongside the raw one and names any requirement that no answer could have satisfied.

Or score answers you produced elsewhere:

import json
from devopsbench.scoring import score_all
from devopsbench.tasks import Task

tasks = [Task(**json.loads(line)) for line in open("tasks/dockerfile.jsonl")]
answers = {task.task_id: my_model_repairs(task) for task in tasks}
scores, summary = score_all(tasks, answers)
print(summary)

Each task carries broken_files to repair, instruction describing the damage, reference_files as the human answer, and repo_path plus commit_id for provenance.

Task sets are balanced by mutation rather than by first-applicable. Shuffling the operators and taking the first that applies looks fair and is not: corrupt_instruction applies to every Dockerfile while drop_user applies to the 11% that set one, so the common mutation took 50 slots of 80 and the set measured one repair fifty times.

Offline tier (default). Everything parses or lints. helm template, hadolint, terraform validate, actionlint, ansible-lint, docker compose config: none of them execute the unit and none reach the network, so a plain machine is a fine place for them. Timeouts are applied anyway, because helm template evaluates Go templates that can loop.

Network tier (--tier network). Validators that download or execute. The two halves are not equally dangerous and are treated differently:

terraform init downloads providers from a registry. It fetches third-party code but does not run the repository's own code, so it is allowed on the local runner. This is implemented and verified, and it is what makes semantic Terraform tasks possible: offline, terraform validate parses and then stops at "Missing required provider" before it checks anything, so a variable used but never declared is invisible. After init, the same command reports Reference to undeclared input variable. That unlocked terraform_missing_variable, which had been sitting unregistered because nothing could detect it.

Measured on 14 real modules in this tier: 7 usable, 7 rejected as already broken. That rejection rate is not a bug: with providers resolved, validate finds genuine semantic errors in real published modules that no offline check can see.

docker build executes every RUN line of a Dockerfile written by a stranger. It is implemented and it refuses to run on any runner that does not isolate, including by default:

>>> docker_build({"Dockerfile": "FROM alpine\nRUN echo hi\n"}, LocalRunner())
UnsafeRunner: `docker build` executes code from the unit under test and needs an
isolating runner; local does not isolate.

The check lives in runners.py rather than in a comment because the failure mode is a stranger's shell script running on a production GPU host, and no flag should be able to cause that. FirecrackerRunner is the only runner that declares isolates = True, and when unconfigured it raises rather than falling back to the host, because a runner that silently degrades is exactly how such an accident happens.

What the Firecracker runner still needs, none of it in this repository: an mTLS client credential for the microVM daemon, a non-production instance of it to talk to, and sign-off from whoever operates the hosts. Endpoint and credential paths are passed in by the operator, so no infrastructure detail is recorded here. Until all three exist, docker build is unreachable, which is the correct state for it to be in.

A task records the tier it was generated in, and scoring uses that tier by default. A defect only terraform init reveals is invisible offline, so scoring a network-tier task offline would pass the unrepaired answer as readily as the fixed one.

Apache-2.0. Task contents are derived from Helmcode/stack-v3-devops (ODC-By 1.0); the code inside remains under its original licences, and repo_path and commit_id travel with every task so attribution is possible.