Repository navigation
Isolate SQLite executions and enable concurrency - #69
Merged
Merged
Conversation
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
JasonTheAdams
added this pull request to stack #70
October 6, 2026 23:26
JasonTheAdams
requested review from
borkweb and
lezama
and removed request for
borkweb
October 6, 2026 23:28
Member
Author
borkweb
approved these changes
Oct 7, 2026
- reset() checked grader.kind, then the baseline, then the isolated state. setup() already refuses to capture a baseline for a cli grader, so one baseline check covers all three. - The exploit audit released the runtime after each test, but _run_verifier already releases it after every candidate and reset() releases on its own failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1TCiQDA5bMGBYQNabsQFh
## Summary A container that exits on its own while being removed no longer counts as a failed release, and a failed cleanup no longer replaces the error that ended the run. ## Why With `--rm`, Docker can answer "removal already in progress" when a container dies (for example an OOM kill) during `docker rm -f`. That raised out of `release()`, discarded the verification result and aborted the run even with `continue_on_error`. Separately, a cleanup failure during a `TestError` or Ctrl-C replaced the original error, so the run lost its `SystemExit(1)` or 130 exit code. ## Testing - [ ] `ruff python` and `mypy python` - [ ] `WP_BENCH_TEST_IMAGE=wp-bench-grader:dev pytest python -q` after building the image in `runtime/`
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U1TCiQDA5bMGBYQNabsQFh
lezama
force-pushed
the
feat/isolated-sqlite-execution
branch
from
October 7, 2026 00:32
22c461a to
7a40d6f
Compare
This was referenced Oct 7, 2026
lezama
added a commit
that referenced
this pull request
Oct 22, 2026
## What Add `wp-cli-v1`: **27 model capability tasks** that execute plain command or Bash script strings through real WP-CLI in private WordPress 7.1 / SQLite containers. The suite covers content operations, taxonomy and metadata, users and capabilities, options and transients, reporting, maintenance, and migration. Thirteen workflows run twice to check convergence and empty selections. Tasks combine discovery, selection, changes, and preservation. Advanced workflows include prefix-scoped URL migration with serialized values, serialized option-key migration, classic menu repair, ordered widget relocation, autoload cleanup, enclosure deduplication, incident credential revocation, and CSV onboarding that preserves existing accounts. Each advanced task records its research basis and links to official documentation, upstream reports, installed WP-CLI implementations, and WordPress 7.1 source. The verifier captures stdout, stderr, exit status, elapsed time, and repeat runs; applies shared timeout and output limits; refreshes WordPress state before trusted assertions; and runs teardown. Bash and jq are included in the runtime. Suite filtering works for both local datasets and multi-suite Hugging Face exports, with the selected suite also used as the result label. Native WordPress operations are required. Runtime WP-CLI configuration disables common PHP/SQL execution shortcuts; the exported `wp` function rejects PHP preloads; file snapshots and loaded-file tracking reject planted PHP/config files. Read-only `wp db prefix` and `wp db tables` remain available. These are benchmark policy checks, not a shell security boundary; Docker provides containment. Network downloads, MySQL-only operations, and multisite are outside this suite. ## Why The existing PHP suite does not measure how models operate WordPress through WP-CLI. These tasks grade realistic workflows and preservation constraints using the SQLite runtime from #68 and per-candidate isolation/concurrency from #69. Equivalent native command workflows are accepted; the grader does not require the reference script's exact spelling. Five overlapping tasks that all four sampled models passed were removed: `e-cli-content-001`, `e-cli-taxonomy-meta-003`, `e-cli-users-001`, `e-cli-options-002`, and `e-cli-reporting-003`. Retained IDs are unchanged. Repeated model runs to assess saturation are separate follow-up work; four successful answers alone do not establish that a test is permanently saturated. ## Review follow-ups Preserve and verify all four contributor commits: - Runtime enforcement catches quoted `eval`, PHP preload flags, and temporary mu-plugin shortcuts, including files removed after WordPress loads them and fixture rewrites in the same second. - Reserve five seconds of the grader timeout so slow scripts can report their captured output before the outer timeout. - Default Hugging Face filtering to the selected run suite. A further regression found during review is also fixed: an explicit `dataset.suite` now sets the result-suite label, with `--suite` overriding both. - Tear down fixture plugins and their activation entries. Remove the integration case for the deleted plugin-status reporting task, while retaining cleanup checks for the cron and bootstrap-recovery fixtures. The proposed shared preservation helper is deferred to a subsequent PR. It is reusable test infrastructure, and changing every preservation assertion is a separate refactor from adding CLI support. ## Verification - [x] `ruff check python` — passed - [x] `mypy python` — passed, 48 files - [x] `WP_BENCH_TEST_IMAGE=wp-bench-grader:cli-review pytest python -q` — **307 passed**, including all Docker integration checks - [x] Built the updated runtime image; all **9 runtime PHP files** passed `php -l` - [x] Real SQLite reference run with eight workers — **27/27 CLI references passed** - [x] CLI exploit audit with eight workers — **127 generic/authored scripts rejected; 0/27 tasks exploitable** by this battery - [x] `python datasets/export_dataset.py` — **377 rows** exported, preserving shell contracts and excluding authored exploit candidates - [x] `git diff --check` — clean The timeout regression checks retained stdout/stderr through the real harness timeout path. Policy integration checks cover quoted commands, config overrides, PHP preload flags, and planted or rewritten PHP files. Hugging Face regressions cover default selection, explicit filters, conflicting run labels, and CLI overrides. ## Score impact - [x] This PR changes scoring/dataset/runtime behavior — adds **`wp-cli-v1`** and bumps the runtime/plugin from **2.1.0 to 2.2.0**. A CLI pass requires every behavioral assertion, the expected exit status on every repeat, completion within execution/output limits, and no authored forbidden error pattern. Existing PHP contracts and scoring version are unchanged. Compare CLI scores only for the same task set and execution profile; the current 27-task suite is not directly comparable to the earlier 24- or 32-task runs. Publishing the generated export to the Hub is separate from this PR. Historical model results for the eight advanced additions (October 12, before the latest runtime-policy and timeout changes; one answer per model/test, no skills): | Model | Passed | Execution pass rate | | --- | ---: | ---: | | Claude Opus 5.5 | 7/8 | 87.5% | | Claude Sonnet 5.5 | 6/8 | 75% | | GPT-6.1 Sol | 6/8 | 75% | | GPT-6 Astra | 6/8 | 75% | All seven failed outputs were reproduced against unchanged assertions; targeted maintainer corrections passed. Those corrected scripts are diagnostic only and excluded from model scores. These historical scores are not a new model run of the current 27-task suite. --------- Co-authored-by: Miguel Lezama <lezama@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Stacked on #68, targeting
refactor/sqlite-runtime. Merge #68 first.Give every candidate a fresh Docker container and enable isolated concurrent execution. Database-only resets leave plugins, uploads, temporary files, logs, and processes shared; a candidate can enumerate another test's plugin files. This PR makes serial and concurrent runs use the same private runtime boundary.
www-data, drop all Linux capabilities, prevent privilege escalation, and make the image filesystem read-only. Each container has CPU, memory, filesystem, and process limits.as_completed. Coordinate cleanup with in-flight container creation.run.execution_concurrencyunderreset_per_testand add--execution-concurrency. Reference checks, exploit audits, and model/skill variants use the private runtime path.New grader settings default to
container_cpus: 1.0,container_memory_mb: 512,container_filesystem_mb: 256, andcontainer_pids_limit: 64. Swap is disabled and/tmpis limited to 64 MiB. Default concurrency remains 1. The existing shared runtime and external CLI grader remain diagnostic modes withexecution_isolation: none.Why
SQLite makes a private database practical, but database separation alone does not prevent one candidate from observing or modifying filesystem/process state belonging to another test. Fresh containers provide the same state isolation at every concurrency level and discard mutations between serial tests too.
Containers still share the host kernel and physical resources. CPU quotas are limits, not reservations; published runs must use a fixed documented profile and sufficient host capacity. This PR does not promise identical timeout behavior under arbitrary host load. Already-running model requests remain subject to their provider timeout/retry policy when aborting.
Verification
ruff check python— clean.mypy python— clean across 46 source files.pytest python -q— 255 passed; 6 opt-in Docker checks skipped in this command.WP_BENCH_TEST_IMAGE=wp-bench-grader:isolated-sqlite pytest python/tests/test_container_runtime_integration.py -q— all 6 passed. Covers filesystem/database/core mutations, concurrent plugin visibility, offline maintenance hooks, enforced limits, and background-process cleanup on timeout/exit.bash -n, andgit diff --check— passed.Local full-suite timings on Docker Desktop (16 CPUs, 8.22 GB available memory, Apple Silicon), including CLI startup, baseline preparation, container creation, restoration, verification, and cleanup:
The serial timing is estimated from the run's start/end file timestamps; parallel timings use a monotonic clock. These compare the isolated execution path across concurrency levels. Fresh containers add startup/copy/cleanup overhead relative to a database-only reset.
CI builds the image, runs the six live isolation checks, and executes SQLite reference checks and the exploit audit with concurrency 4.
Score impact
The scoring formula, suite fixtures, assertions, and reference solutions are unchanged. Serial/concurrent parity is verified for all references and the saved model sample. Filesystem permissions, offline networking, and resource limits can change outcomes compared with the previous database-only runtime, so published scores must identify the runtime image and profile rather than assume cross-runtime comparability.