feat: add WP-CLI workflow execution suite

lezama · Oct 22, 2026 · Oct 7, 2026 · Oct 12, 2026 · Oct 21, 2026 · Oct 21, 2026
commit b45bdf7abfb96a51574e205720b9adefbe150bdf
@@ -30,9 +30,9 @@ Use this skill when adding or reviewing execution tests for WP-Bench.
 - `test_function`: PHP signature of the entry point the verifier calls, e.g. `wpbp_queries_004( string $category_slug, array $tag_slugs ): WP_Query`. Set it whenever assertions invoke the function. Shown to the model and prepended to the runtime assertions as a weight-0 `function_exists` check (unscored). Use parameter names that convey meaning; pin the return type only when assertions check it.
 - `expected_behavior`: Reviewer documentation. Must differ from `prompt`.
 - `reference_solution`: Canonical passing code used for author verification. Required for every test, including plugin-artifact tests.
-- `artifact_kind`: What the model must produce. `php_snippet` (default) or `wp_plugin_files` (a JSON `files` map installed as a plugin before assertions run).
+- `artifact_kind`: What the model must produce. `php_snippet` (default), `wp_plugin_files` (a JSON `files` map installed as a plugin), or `wp_cli_shell` (a plain Bash command/script string).
 - `reference_files`: For `wp_plugin_files` tests, the reference plugin files (relative path → contents) used by `--check-reference-solution`. Required in addition to `reference_solution`.
-- `exploit_solutions`: Optional list of PHP snippets that must **fail** the assertions. Each defines the gateway function (when there is one) and mimics a plausible shortcut — a hard-coded fixture answer, an incomplete implementation, a wrong API. `--check-exploits` runs them after the generic battery. They are always PHP snippets, even for plugin-artifact tests (never a `{"files": ...}` map). Maintainer-side QA; not exported to the dataset.
+- `exploit_solutions`: Optional list of candidate strings that must **fail** the assertions. Each defines the gateway function (when there is one) and mimics a plausible shortcut — a hard-coded fixture answer, an incomplete implementation, a wrong API. `--check-exploits` runs them after the generic battery. They are Bash scripts for `wp_cli_shell`; otherwise they are PHP snippets, including for plugin-artifact tests (never a `{"files": ...}` map). Maintainer-side QA; not exported to the dataset.
 - `static_checks`: Coarse guardrails for required or forbidden code patterns.
 - `runtime_checks.setup`: Optional PHP fixture setup evaluated before the submitted code.
 - `runtime_checks.assertions`: WordPress-executed behavioral assertions evaluated after the submitted code.
@@ -147,7 +147,55 @@ Clean up state in `teardown` when it persists beyond the PHP process or can affe
 
 Avoid cleanup for in-process-only registries when each verifier run starts a fresh WP-CLI process. Extra cleanup can make failing cases noisy and less diagnostic.
 
-The harness resets the WordPress environment between execution tests by default (`run.execution_isolation: reset_per_test` — database reset plus fresh install), so cross-test leakage is prevented even when a teardown is missed. Teardown still matters within a single test: assertions run in the same process and site state as the submitted code, and authors iterating with `execution_isolation: none` rely on it.
+The default `run.execution_isolation: reset_per_test` gives every candidate a fresh private container, pristine WordPress files, and a trusted SQLite baseline. It discards the container and all descendants after verification, so database, filesystem, and process state cannot leak into other tests. Teardown still matters within a single test: assertions run in the same process and site state as the submitted code, and authors iterating with `execution_isolation: none` rely on it.
+
+## WP-CLI Workflow Tests
+
+Use `wp-cli-v1` for realistic operational tasks that combine discovery, selection,
+changes, reports, and preservation. Prioritize model capability over testing tiny
+WordPress units. This suite targets single-site WordPress 7.1 with SQLite;
+avoid unsupported MySQL operations, network installs, or multisite instead of
+adding compatibility shims.
+
+- The completion and `reference_solution` are plain Bash strings, optionally
+  fenced, never a JSON command array. Bash runs in the WordPress root with
+  `jq` and standard shell utilities. It does not implicitly set errexit/pipefail.
+- Trusted PHP setup and assertions run in the parent verifier; every `wp`
+  invocation is a separate WordPress process. Fixture callbacks must live in
+  temporary plugin/mu-plugin files to run in CLI subprocesses.
+- The verifier flushes its object cache and reloads roles before assertions.
+  Read output from `$GLOBALS['wpbp_shell_result']` (`stdout`, `stderr`,
+  `exit_code`). Candidate stdout cannot substitute for the verifier response.
+- `runtime_checks.exit_code` defaults to 0 and is an unscored execution gate.
+  `repeat: 2` runs the same script twice on the same fixtures, checks both exits,
+  and asserts the final state. Assertions see the last run's output; result
+  `command.runs` retains every run. The time budget is shared across repeats.
+- Vary fixture IDs for both posts and users, and report values such as counts,
+  versions, or UUID markers. Fixed output should not pass. Check unrelated
+  records and structured value types, and author overly broad/incomplete scripts
+  as exploit candidates. Run every reference and exploit through real SQLite.
+- Native operations are required; PHP evaluation/preloading, direct SQL, and
+  implementation-file edits are out of contract. Authored forbidden patterns
+  are coarse policy checks; container isolation is the containment boundary.
+- Source authority includes the installed WP-CLI commands and their official
+  documentation, as well as WordPress 7.1 source. Do not demand an exact command
+  spelling when another native workflow produces the correct state.
+
+Pitfalls found while authoring this suite:
+
+- `wp post delete` permanently deletes posts already in trash even without
+  `--force`. That is a correct alternative, not an exploit.
+- A `name` query is singular and can hide drafts from the default CLI user.
+  Discover drafts with a list query (or an appropriate user context). Assertions
+  can use `post_name__in` to count matching drafts and detect duplicates.
+- `WP_Post::to_array()` includes derived `post_category` and `tags_input` fields.
+  Compare relationships separately when the task changes categories/tags.
+- Publishing legitimately adds `_pingme`/`_encloseme` metadata. Updating drafts
+  with a zero GMT date can refresh their local date. Preserve the content the
+  prompt asks to preserve without rejecting these normal core side effects.
+- A future timestamp can turn an inserted `publish` fixture into `future`.
+  Give published contrasts a past date; use dates sufficiently far ahead for
+  scheduling tasks.
 
 ## Plugin Artifact Tests
 

@@ -51,7 +51,7 @@ jobs:
         run: pip install -e ./python[dev]
 
       - name: Dataset shape and metadata tests
-        run: pytest python/tests/test_execution_dataset.py python/tests/test_dataset_metadata.py -q
+        run: pytest python/tests/test_execution_dataset.py python/tests/test_dataset_metadata.py python/tests/test_wp_cli_shell.py -q
 
       - name: Parquet export builds
         run: |
@@ -93,7 +93,17 @@ jobs:
       - name: Container isolation integration tests
         env:
           WP_BENCH_TEST_IMAGE: wp-bench-grader:dev
-        run: pytest python/tests/test_container_runtime_integration.py -q
+        run: pytest python/tests/test_container_runtime_integration.py python/tests/test_shell_runtime_integration.py -q
+
+      - name: WP-CLI workflow reference solutions
+        run: >-
+          wp-bench run --config wp-bench.example.yaml --suite wp-cli-v1
+          --check-reference-solution --execution-concurrency 4
+
+      - name: WP-CLI workflow exploit audit
+        run: >-
+          wp-bench run --config wp-bench.example.yaml --suite wp-cli-v1
+          --check-exploits --execution-concurrency 4
 
       - name: SQLite database reference solutions
         run: >-

@@ -38,7 +38,7 @@ cd runtime
 docker build -t wp-bench-grader:dev .
 ```
 
-Runtime 2.1 runs WordPress 7.1 with SQLite. It uses the official
+Runtime 2.2 runs WordPress 7.1 with SQLite. It uses the official
 [SQLite Database Integration](https://github.com/WordPress/sqlite-database-integration/releases/tag/v3.0.2)
 3.0.2 drop-in, pinned by version and archive checksum. No database server is
 started. For a persistent diagnostic runtime, run
@@ -255,16 +255,62 @@ Test suites live in `datasets/suites/<suite-name>/`:
 
 - `execution/` — Code generation tasks with assertions (one JSON file per category)
 
-The default suite `wp-core-v1` covers WordPress core APIs, hooks, database operations, and security patterns.
+The default suite `wp-core-v1` contains 350 tasks covering WordPress core APIs,
+hooks, database operations, and security patterns.
+
+### WP-CLI workflows
+
+`wp-cli-v1` adds 24 operational tasks across content, taxonomy and metadata,
+users and capabilities, options and transients, reporting, and maintenance.
+Tasks combine selection, changes, and preservation: examples include retiring
+an author while reassigning content, consolidating categories, repairing a
+plugin that crashes bootstrap, producing scoped JSON/CSV reports, and provisioning
+content without duplicates when the script runs again.
+
+```bash
+wp-bench run --config wp-bench.example.yaml --suite wp-cli-v1 --execution-concurrency 4
+# Maintainer validation without model calls:
+wp-bench run --config wp-bench.example.yaml --suite wp-cli-v1 --check-reference-solution
+wp-bench run --config wp-bench.example.yaml --suite wp-cli-v1 --check-exploits
+```
+
+The model returns a **plain command or Bash script string**, optionally inside
+a Bash code fence. Bash executes it from the WordPress root, with ordinary
+quoting, pipelines, loops, command substitution, heredocs, and redirection.
+`jq` and standard shell utilities are available; `errexit` and `pipefail` are
+not enabled automatically. Each `wp` command boots WordPress separately.
+These tasks require native WP-CLI operations and exclude PHP evaluation,
+PHP preloading, direct SQL, and edits to WordPress implementation files.
+Network downloads, MySQL-only database operations, and multisite are outside
+this initial SQLite suite.
+
+Trusted PHP creates fixtures before Bash runs and checks the resulting
+WordPress state and report output afterward. IDs and some fixture values vary
+to reject fixed answers. Several tasks run the same script twice on the same
+site to check convergence and empty selections. The expected script exit status
+is a pass gate (default `0`); intermediate nonzero statuses can be handled with
+normal shell branches. Results retain stdout, stderr, exit status, elapsed time,
+and repeat runs under `grader.raw.command`. Each output stream is capped at
+1 MiB. The existing timeout and private container limits apply to the script
+and its descendants, and timeouts or output overflow score zero.
+
+The WP-CLI suite is scored separately from `wp-core-v1`; existing PHP task
+contracts and scoring remain unchanged. Compare scores only within the same
+suite and execution profile.
 
 ### Loading from Hugging Face
 
 ```yaml
 dataset:
   source: huggingface
   name: WordPress/wp-bench-v1
+  suite: wp-core-v1       # filter a multi-suite export; omit to load all suites
 ```
 
+`--suite wp-cli-v1` selects CLI rows without changing the Hugging Face repository
+name. The new suite is available locally immediately; Hub users need an export
+that includes it.
+
 ## Results & Reporting
 
 After running benchmarks, visualize results with the included Jupyter notebook:

@@ -7,11 +7,13 @@ This directory contains the benchmark test suites and tooling for publishing to
 ```
 datasets/
 ├── suites/                    # Source of truth (human-editable JSON)
-│   └── wp-core-v1/
-│       └── execution/         # Code generation tests (one file per category)
-│           ├── hooks.json
-│           ├── rest-api.json
-│           └── ...
+│   ├── wp-core-v1/
+│   │   └── execution/         # Code generation tests (one file per category)
+│   │       ├── hooks.json
+│   │       ├── rest-api.json
+│   │       └── ...
+│   └── wp-cli-v1/
+│       └── execution/         # 24 WP-CLI workflows, six categories
 ├── data/                      # Generated Parquet for HF (gitignored)
 │   └── test.parquet
 ├── export_dataset.py          # Converts suites → Parquet
@@ -67,7 +69,7 @@ dataset:
 | `static_checks` | object | Regex patterns to check in generated code |
 | `runtime_checks` | object | Assertions to run in WordPress environment |
 | `reference_solution` | string | Example correct solution |
-| `artifact_kind` | string | What the model must produce: `php_snippet` (default) or `wp_plugin_files` |
+| `artifact_kind` | string | What the model must produce: `php_snippet` (default), `wp_plugin_files`, or `wp_cli_shell` |
 | `reference_files` | object | For `wp_plugin_files` tests: reference plugin files (relative path → contents) |
 
 For `wp_plugin_files` tasks the model must return a JSON object with a
@@ -77,6 +79,34 @@ the files as a plugin, loads it, runs the task's assertions, and removes
 it. Artifacts are validated before install: no absolute paths, no `..`
 traversal, limited file count and total size.
 
+For `wp_cli_shell`, `reference_solution` and model completions are plain Bash
+command/script strings, optionally fenced. No JSON command array is used.
+`runtime_checks.setup`, `assertions`, and `teardown` remain trusted PHP.
+Assertions read the final stdout/stderr/exit status from
+`$GLOBALS['wpbp_shell_result']` and inspect WordPress state after its object
+cache and roles have been refreshed. Each CLI invocation is a separate process;
+PHP hooks installed only in setup do not carry into the CLI process. Put fixture
+callbacks in a temporary plugin or mu-plugin when a command must execute them.
+
+`runtime_checks.exit_code` defaults to `0`. A mismatch, timeout, or output overflow
+fails execution without awarding exit-status points. `runtime_checks.repeat`
+defaults to `1`; `2` runs the script twice on the same fixtures, checks every
+exit status, and asserts the final state. Only the last run's output is exposed
+to assertions; all runs are retained in the result. The script gets the remaining
+execution timeout after fixture setup, shared across repeats.
+
+`exploit_solutions` contains Bash strings for CLI tasks, PHP snippets for other
+tasks. The CLI audit also runs a no-op, a nonzero exit, and empty JSON output.
+Author incomplete or overly broad workflows that must fail, alongside a passing
+reference. Grading checks outcomes and preservation rather than requiring one
+exact command spelling. Forbidden static patterns are coarse policy checks,
+not a shell security boundary; Docker provides containment.
+
+The local loader selects `dataset.name`; the Hub loader uses `dataset.name` as
+the repository and optional `dataset.suite` to filter the combined export.
+`--suite` handles both sources. Exports include both suites, and omit authored
+exploit candidates. CLI scores should be reported separately from core scores.
+
 
 ### Per-Test Metadata