Repository navigation
Add WP-CLI workflow execution suite - #71
Conversation
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. Props Bot commandsYou can trigger Props Bot actions by commenting on this PR:
|
lezama
left a comment
lezama
left a comment
There was a problem hiding this comment.
The shell verifier is small and easy to follow, and the preservation checks are strict. Nothing blocking, but two things I'd sort out before merge.
- Saturation. In the PR's own run the four models land within one task of each other on the 8 advanced tasks, and a lot of the first 24 come down to knowing one flag (
--skip-update,--due-now --exclude,--all-tables-with-prefix). Can we run all 32 a few times on 3-4 models and drop or harden the ones every model passes? Otherwise this suite clusters at the top like the core one. - The no-PHP rule is only a regex, and two shortcuts get a full pass today (inline on the shell verifier).
What I ran on 8d998f4: ruff, mypy, pytest python with WP_BENCH_TEST_IMAGE (288 passed), and the two new CI steps (32/32 references pass, 0/32 exploitable).
| { | ||
| "type": "custom_assertion", | ||
| "description": "Existing content, metadata, and relationships change only within the requested scope", | ||
| "code": "$f=$GLOBALS['cli_fixture'];\n$slugs=array_flip($f['posts']??array());\nforeach($f['content_before'] as $id=>$before) {\n $slug=$slugs[$id]??'';\n $actual=get_post($id,ARRAY_A); $expected=$before['post'];\n // Derived taxonomy fields are checked independently below.\n unset($actual['post_category'],$actual['tags_input'],$expected['post_category'],$expected['tags_input']);\n if(in_array($slug,array('cli-ready-a','cli-ready-b'),true)) {\n if(in_array('delete',array('post_status','post_date','post_date_gmt','post_modified','post_modified_gmt'),true)) { if($actual!==null) { return false; } continue; }\n foreach(array('post_status','post_date','post_date_gmt','post_modified','post_modified_gmt') as $key) { unset($actual[$key],$expected[$key]); }\n }\n if($actual!==$expected) { return false; }\n $meta=get_post_meta($id); $wanted=$before['meta'];\n if(in_array($slug,array('cli-ready-a','cli-ready-b'),true)) {\n foreach(array_unique(array_merge(array_keys($meta),array_keys($wanted))) as $key) {\n foreach(array('_pingme','_encloseme') as $allowed) {\n if($key===$allowed || (str_starts_with($allowed,'prefix:') && str_starts_with($key,substr($allowed,7)))) { unset($meta[$key],$wanted[$key]); }\n }\n }\n }\n if($meta!==$wanted) { return false; }\n $cats=wp_get_post_categories($id); $tags=wp_get_post_tags($id,array('fields'=>'ids'));\n sort($cats); sort($tags); sort($before['category']); sort($before['tags']);\n if(!in_array($slug,array(),true) && $cats!==$before['category']) { return false; }\n if(!in_array($slug,array(),true) && $tags!==$before['tags']) { return false; }\n} return true;", |
There was a problem hiding this comment.
nit: the generated assertions keep template leftovers, like in_array('delete', array('post_status', ...)) which is always false, $login === '--none--', or foreach (array() as $key). They work, but they make the 32 tasks hard to review. A shared helper in the runtime (snapshot before, compare against an allowlist after) would remove most of the repeated JSON and be easier to trust.
There was a problem hiding this comment.
Agreed that the repeated preservation checks should be centralized. We are deferring the shared snapshot/allowlist helper and template cleanup to a separate PR so the reusable runtime infrastructure can be reviewed and tested for both PHP and CLI tasks. The existing preservation assertions remain in place here.
The per-test regex was the only guard, so `wp 'eval' ...` or a file dropped into wp-content/mu-plugins/ passed e-cli-options-001 in full. Candidate scripts now run with a WP-CLI config that disables eval, eval-file, shell, db query/cli/import, and package. `wp` is an exported Bash function (writable mounts are noexec) that pins that config and rejects --exec/--require, which WP-CLI runs before any config require. The run scores 0 if the script leaves a new or changed PHP file, .maintenance, or WP-CLI config under the WordPress root or temp dir, or if WordPress loaded a PHP file the script added and then removed. The config requires record-loaded-files.php to log what each `wp` call loaded, and the config itself is watched. Files changed in the same second as the snapshot are compared by hash, since ctime has one-second resolution. A script that calls the php binary or skips the `wp` function on purpose is still left to the static checks; the README says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shell got the same budget as the outer docker exec but started later, after WordPress boot and setup, so the outer timeout always won. A slow script then came back as _timeout_raw_result with no command block, and its stdout, stderr, and runs were lost. The shell now gets grader.timeout_seconds minus 5 seconds, at least 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With source: huggingface and no dataset.suite, every suite in the export was loaded while results were still labeled run.suite (wp-core-v1 by default). Once the Hub export has CLI rows, the default config would mix all 382 tests into a run labeled wp-core-v1. dataset.suite now defaults to run.suite for Hub datasets. An explicit dataset.suite or --suite still overrides it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
e-cli-maintenance-001 wrote mu-plugins/cli-cron.php, and e-cli-maintenance-002 and e-cli-reporting-003 wrote plugin directories, all with no teardown. Under execution_isolation: none they stayed for later tests, including cli-broken, which throws on every WordPress boot while active. Each test now removes its files, the active_plugins entries it added, and (for maintenance-001) the options its cron callbacks write, so a second pass over the suite starts from the same state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
On the saturation point in your review: a7252a8 removes five overlapping workflows that all four sampled models passed, leaving 27 tasks with retained IDs unchanged. We are deferring repeated multi-model runs and further saturation analysis to follow-up work. The PR description now clearly separates historical model scores from validation of the current runtime. |
lezama
left a comment
lezama
left a comment
There was a problem hiding this comment.
Thanks Jason, this looks good to me. The pruning makes sense and keeping dataset.suite and run.suite in sync closes the last gap I had. Happy to look at the preservation helper in the follow-up :)
What
Add
wp-cli-v1: 27 model capability tasks that execute plain command or Bash script strings through real WP-CLI in private WordPress 7.1 / SQLite containers. The suite covers content operations, taxonomy and metadata, users and capabilities, options and transients, reporting, maintenance, and migration. Thirteen workflows run twice to check convergence and empty selections.Tasks combine discovery, selection, changes, and preservation. Advanced workflows include prefix-scoped URL migration with serialized values, serialized option-key migration, classic menu repair, ordered widget relocation, autoload cleanup, enclosure deduplication, incident credential revocation, and CSV onboarding that preserves existing accounts. Each advanced task records its research basis and links to official documentation, upstream reports, installed WP-CLI implementations, and WordPress 7.1 source.
The verifier captures stdout, stderr, exit status, elapsed time, and repeat runs; applies shared timeout and output limits; refreshes WordPress state before trusted assertions; and runs teardown. Bash and jq are included in the runtime. Suite filtering works for both local datasets and multi-suite Hugging Face exports, with the selected suite also used as the result label.
Native WordPress operations are required. Runtime WP-CLI configuration disables common PHP/SQL execution shortcuts; the exported
wpfunction rejects PHP preloads; file snapshots and loaded-file tracking reject planted PHP/config files. Read-onlywp db prefixandwp db tablesremain available. These are benchmark policy checks, not a shell security boundary; Docker provides containment. Network downloads, MySQL-only operations, and multisite are outside this suite.Why
The existing PHP suite does not measure how models operate WordPress through WP-CLI. These tasks grade realistic workflows and preservation constraints using the SQLite runtime from #68 and per-candidate isolation/concurrency from #69. Equivalent native command workflows are accepted; the grader does not require the reference script's exact spelling.
Five overlapping tasks that all four sampled models passed were removed:
e-cli-content-001,e-cli-taxonomy-meta-003,e-cli-users-001,e-cli-options-002, ande-cli-reporting-003. Retained IDs are unchanged. Repeated model runs to assess saturation are separate follow-up work; four successful answers alone do not establish that a test is permanently saturated.Review follow-ups
Preserve and verify all four contributor commits:
eval, PHP preload flags, and temporary mu-plugin shortcuts, including files removed after WordPress loads them and fixture rewrites in the same second.dataset.suitenow sets the result-suite label, with--suiteoverriding both.The proposed shared preservation helper is deferred to a subsequent PR. It is reusable test infrastructure, and changing every preservation assertion is a separate refactor from adding CLI support.
Verification
ruff check python— passedmypy python— passed, 48 filesWP_BENCH_TEST_IMAGE=wp-bench-grader:cli-review pytest python -q— 307 passed, including all Docker integration checksphp -lpython datasets/export_dataset.py— 377 rows exported, preserving shell contracts and excluding authored exploit candidatesgit diff --check— cleanThe timeout regression checks retained stdout/stderr through the real harness timeout path. Policy integration checks cover quoted commands, config overrides, PHP preload flags, and planted or rewritten PHP files. Hugging Face regressions cover default selection, explicit filters, conflicting run labels, and CLI overrides.
Score impact
wp-cli-v1and bumps the runtime/plugin from 2.1.0 to 2.2.0.A CLI pass requires every behavioral assertion, the expected exit status on every repeat, completion within execution/output limits, and no authored forbidden error pattern. Existing PHP contracts and scoring version are unchanged. Compare CLI scores only for the same task set and execution profile; the current 27-task suite is not directly comparable to the earlier 24- or 32-task runs. Publishing the generated export to the Hub is separate from this PR.
Historical model results for the eight advanced additions (October 12, before the latest runtime-policy and timeout changes; one answer per model/test, no skills):
All seven failed outputs were reproduced against unchanged assertions; targeted maintainer corrections passed. Those corrected scripts are diagnostic only and excluded from model scores. These historical scores are not a new model run of the current 27-task suite.