Skip to content

Isolate SQLite executions and enable concurrency - #69

Merged
lezama merged 4 commits into
trunkfrom
feat/isolated-sqlite-execution
Oct 7, 2026
Merged

lezama merged 4 commits into
trunkfrom
feat/isolated-sqlite-execution

Conversation

@JasonTheAdams

Copy link
Copy Markdown
Member

What

Stacked on #68, targeting refactor/sqlite-runtime. Merge #68 first.

Give every candidate a fresh Docker container and enable isolated concurrent execution. Database-only resets leave plugins, uploads, temporary files, logs, and processes shared; a candidate can enumerate another test's plugin files. This PR makes serial and concurrent runs use the same private runtime boundary.

  • Prepare one trusted WordPress install from a resolved image ID. Keep its SQLite snapshot and configuration on the host, and remove the preparation container before grading.
  • Copy pristine WordPress files into a private bounded tmpfs and restore the database for each candidate. Existing Compose containers and their data are never used or reset by isolated runs.
  • Disable networking, run as www-data, drop all Linux capabilities, prevent privilege escalation, and make the image filesystem read-only. Each container has CPU, memory, filesystem, and process limits.
  • Remove the entire container after verification, timeout, crash, or abort, including background processes. Cancel queued futures on errors and Ctrl-C, including interrupts while waiting in as_completed. Coordinate cleanup with in-flight container creation.
  • Support run.execution_concurrency under reset_per_test and add --execution-concurrency. Reference checks, exploit audits, and model/skill variants use the private runtime path.
  • Disable WordPress's automatic update-check hooks in the offline verifier. Without this, the admin settings reference triggers a WordPress.org request and fails with networking disabled. Dataset assertions and reference solutions are unchanged.
  • Record the isolation boundary, requested concurrency, resolved image ID, and grader limits in result metadata. Document the execution profile required for published scores and cleanup after forced termination.

New grader settings default to container_cpus: 1.0, container_memory_mb: 512, container_filesystem_mb: 256, and container_pids_limit: 64. Swap is disabled and /tmp is limited to 64 MiB. Default concurrency remains 1. The existing shared runtime and external CLI grader remain diagnostic modes with execution_isolation: none.

Why

SQLite makes a private database practical, but database separation alone does not prevent one candidate from observing or modifying filesystem/process state belonging to another test. Fresh containers provide the same state isolation at every concurrency level and discard mutations between serial tests too.

Containers still share the host kernel and physical resources. CPU quotas are limits, not reservations; published runs must use a fixed documented profile and sufficient host capacity. This PR does not promise identical timeout behavior under arbitrary host load. Already-running model requests remain subject to their provider timeout/retry policy when aborting.

Verification

  • ruff check python — clean.
  • mypy python — clean across 46 source files.
  • pytest python -q — 255 passed; 6 opt-in Docker checks skipped in this command.
  • WP_BENCH_TEST_IMAGE=wp-bench-grader:isolated-sqlite pytest python/tests/test_container_runtime_integration.py -q — all 6 passed. Covers filesystem/database/core mutations, concurrent plugin visibility, offline maintenance hooks, enforced limits, and background-process cleanup on timeout/exit.
  • Runtime image build, all seven PHP syntax checks, entrypoint bash -n, and git diff --check — passed.
  • All 350 reference solutions pass at concurrency 1, 4, and 8, with identical per-test score dictionaries across runs.
  • Replayed 16 saved model completions, including a plugin artifact and both passing/failing answers: 9 pass and 7 fail identically at concurrency 1, 4, and 8. No model calls were made.
  • Database/security exploit audit at concurrency 4 — 78 incorrect candidates rejected, 0/9 auditable tests exploitable.
  • Live Ctrl-C after six streamed results — exit 130 in 0.475 seconds, partial results retained, no disposable containers left.
  • Every completed benchmark/audit left zero disposable containers; the pre-existing local Compose runtime remained running and untouched.

Local full-suite timings on Docker Desktop (16 CPUs, 8.22 GB available memory, Apple Silicon), including CLI startup, baseline preparation, container creation, restoration, verification, and cleanup:

Concurrent candidates References passing Wall time Speedup over isolated serial
1 350/350 approximately 259 s 1×
4 350/350 72.3 s approximately 3.6×
8 350/350 54.7 s approximately 4.7×

The serial timing is estimated from the run's start/end file timestamps; parallel timings use a monotonic clock. These compare the isolated execution path across concurrency levels. Fresh containers add startup/copy/cleanup overhead relative to a database-only reset.

CI builds the image, runs the six live isolation checks, and executes SQLite reference checks and the exploit audit with concurrency 4.

Score impact

  • This PR does not change how any test is scored.
  • This PR changes runtime behavior — bump runtime to 2.1.0 and record the container isolation boundary and execution profile.

The scoring formula, suite fixtures, assertions, and reference solutions are unchanged. Serial/concurrent parity is verified for all references and the saved model sample. Filesystem permissions, offline networking, and resource limits can change outcomes compared with the previous database-only runtime, so published scores must identify the runtime image and profile rather than assume cross-runtime comparability.

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: JasonTheAdams <jason_the_adams@git.wordpress.org>
Co-authored-by: lezama <migueluy@git.wordpress.org>
Co-authored-by: borkweb <borkweb@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@JasonTheAdams
JasonTheAdams added this pull request to stack #70 October 6, 2026 23:26
@JasonTheAdams
JasonTheAdams requested review from borkweb and lezama and removed request for borkweb October 6, 2026 23:28
@JasonTheAdams

Copy link
Copy Markdown
Member Author

I'm planing on replacing #55 with this since this is true isolation and we can therefore guarantee results (per @mcsf's warning). This will also lay the groundwork to turn our tests into "tasks" for use in Harbor.

Base automatically changed from refactor/sqlite-runtime to trunk October 7, 2026 00:31
JasonTheAdams and others added 4 commits October 6, 2026 21:32
- reset() checked grader.kind, then the baseline, then the isolated
  state. setup() already refuses to capture a baseline for a cli
  grader, so one baseline check covers all three.
- The exploit audit released the runtime after each test, but
  _run_verifier already releases it after every candidate and reset()
  releases on its own failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U1TCiQDA5bMGBYQNabsQFh
## Summary
A container that exits on its own while being removed no longer counts as a failed release, and a failed cleanup no longer replaces the error that ended the run.

## Why
With `--rm`, Docker can answer "removal already in progress" when a container dies (for example an OOM kill) during `docker rm -f`. That raised out of `release()`, discarded the verification result and aborted the run even with `continue_on_error`. Separately, a cleanup failure during a `TestError` or Ctrl-C replaced the original error, so the run lost its `SystemExit(1)` or 130 exit code.

## Testing
- [ ] `ruff python` and `mypy python`
- [ ] `WP_BENCH_TEST_IMAGE=wp-bench-grader:dev pytest python -q` after building the image in `runtime/`
@lezama
lezama force-pushed the feat/isolated-sqlite-execution branch from 22c461a to 7a40d6f Compare October 7, 2026 00:32
@lezama
lezama merged commit 37b012e into trunk Oct 7, 2026
5 checks passed
@lezama
lezama deleted the feat/isolated-sqlite-execution branch October 7, 2026 00:36
lezama added a commit that referenced this pull request Oct 22, 2026
## What

Add `wp-cli-v1`: **27 model capability tasks** that execute plain
command or Bash script strings through real WP-CLI in private WordPress
7.1 / SQLite containers. The suite covers content operations, taxonomy
and metadata, users and capabilities, options and transients, reporting,
maintenance, and migration. Thirteen workflows run twice to check
convergence and empty selections.

Tasks combine discovery, selection, changes, and preservation. Advanced
workflows include prefix-scoped URL migration with serialized values,
serialized option-key migration, classic menu repair, ordered widget
relocation, autoload cleanup, enclosure deduplication, incident
credential revocation, and CSV onboarding that preserves existing
accounts. Each advanced task records its research basis and links to
official documentation, upstream reports, installed WP-CLI
implementations, and WordPress 7.1 source.

The verifier captures stdout, stderr, exit status, elapsed time, and
repeat runs; applies shared timeout and output limits; refreshes
WordPress state before trusted assertions; and runs teardown. Bash and
jq are included in the runtime. Suite filtering works for both local
datasets and multi-suite Hugging Face exports, with the selected suite
also used as the result label.

Native WordPress operations are required. Runtime WP-CLI configuration
disables common PHP/SQL execution shortcuts; the exported `wp` function
rejects PHP preloads; file snapshots and loaded-file tracking reject
planted PHP/config files. Read-only `wp db prefix` and `wp db tables`
remain available. These are benchmark policy checks, not a shell
security boundary; Docker provides containment. Network downloads,
MySQL-only operations, and multisite are outside this suite.

## Why

The existing PHP suite does not measure how models operate WordPress
through WP-CLI. These tasks grade realistic workflows and preservation
constraints using the SQLite runtime from #68 and per-candidate
isolation/concurrency from #69. Equivalent native command workflows are
accepted; the grader does not require the reference script's exact
spelling.

Five overlapping tasks that all four sampled models passed were removed:
`e-cli-content-001`, `e-cli-taxonomy-meta-003`, `e-cli-users-001`,
`e-cli-options-002`, and `e-cli-reporting-003`. Retained IDs are
unchanged. Repeated model runs to assess saturation are separate
follow-up work; four successful answers alone do not establish that a
test is permanently saturated.

## Review follow-ups

Preserve and verify all four contributor commits:

- Runtime enforcement catches quoted `eval`, PHP preload flags, and
temporary mu-plugin shortcuts, including files removed after WordPress
loads them and fixture rewrites in the same second.
- Reserve five seconds of the grader timeout so slow scripts can report
their captured output before the outer timeout.
- Default Hugging Face filtering to the selected run suite. A further
regression found during review is also fixed: an explicit
`dataset.suite` now sets the result-suite label, with `--suite`
overriding both.
- Tear down fixture plugins and their activation entries. Remove the
integration case for the deleted plugin-status reporting task, while
retaining cleanup checks for the cron and bootstrap-recovery fixtures.

The proposed shared preservation helper is deferred to a subsequent PR.
It is reusable test infrastructure, and changing every preservation
assertion is a separate refactor from adding CLI support.

## Verification

- [x] `ruff check python` — passed
- [x] `mypy python` — passed, 48 files
- [x] `WP_BENCH_TEST_IMAGE=wp-bench-grader:cli-review pytest python -q`
— **307 passed**, including all Docker integration checks
- [x] Built the updated runtime image; all **9 runtime PHP files**
passed `php -l`
- [x] Real SQLite reference run with eight workers — **27/27 CLI
references passed**
- [x] CLI exploit audit with eight workers — **127 generic/authored
scripts rejected; 0/27 tasks exploitable** by this battery
- [x] `python datasets/export_dataset.py` — **377 rows** exported,
preserving shell contracts and excluding authored exploit candidates
- [x] `git diff --check` — clean

The timeout regression checks retained stdout/stderr through the real
harness timeout path. Policy integration checks cover quoted commands,
config overrides, PHP preload flags, and planted or rewritten PHP files.
Hugging Face regressions cover default selection, explicit filters,
conflicting run labels, and CLI overrides.

## Score impact

- [x] This PR changes scoring/dataset/runtime behavior — adds
**`wp-cli-v1`** and bumps the runtime/plugin from **2.1.0 to 2.2.0**.

A CLI pass requires every behavioral assertion, the expected exit status
on every repeat, completion within execution/output limits, and no
authored forbidden error pattern. Existing PHP contracts and scoring
version are unchanged. Compare CLI scores only for the same task set and
execution profile; the current 27-task suite is not directly comparable
to the earlier 24- or 32-task runs. Publishing the generated export to
the Hub is separate from this PR.

Historical model results for the eight advanced additions (October 12,
before the latest runtime-policy and timeout changes; one answer per
model/test, no skills):

| Model | Passed | Execution pass rate |
| --- | ---: | ---: |
| Claude Opus 5.5 | 7/8 | 87.5% |
| Claude Sonnet 5.5 | 6/8 | 75% |
| GPT-6.1 Sol | 6/8 | 75% |
| GPT-6 Astra | 6/8 | 75% |

All seven failed outputs were reproduced against unchanged assertions;
targeted maintainer corrections passed. Those corrected scripts are
diagnostic only and excluded from model scores. These historical scores
are not a new model run of the current 27-task suite.

---------

Co-authored-by: Miguel Lezama <lezama@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants