Time the benchmark's scaling rows on CPU time, and run its counts in parallel - #1352

tests/v2/test_benchmark.py's clock-timed tests failed at random on a laptop running several agent sessions at once. Tests and docs only; nothing under nameparser/ changes.

What changed

• The 34 clock-timed tests read the thread's CPU time (time.thread_time, _CLOCK) instead of time.perf_counter(). These are 23 _SHAPES rows, 8 _PREFIXED_SHAPES rows, 2 _POLICY_SHAPES rows and the thousand-names backstop. The wall clock counts time the scheduler spends running other processes. CPU time doesn't, but it still counts everything the parse does on its own thread, C-level work and memory stalls included.
• _best became _ratio. It interleaves the small and large input (one parse of each per round), so a slow stretch is charged to both. It takes _ROUNDS = 7 rounds. Where those read within 20% of the bound or over it (_NEAR = 0.8), it takes all _MAX_ROUNDS = 35 and judges the last minimums.
  • Each round can only lower the two minimums toward the true costs, so the estimate moves toward the true ratio in either direction: load can deflate a ratio as well as inflate it.
  • It is judged once, never at the first reading that passes.
  • test_ratio_takes_more_rounds_only_near_the_bound holds this on a scripted clock.
  • A quiet clean run (~4x) pays for no extra rounds. A loaded one does: the serial pass took 78–115s at 24 busy loops, against 47s for the first version of this PR.
• No threshold or baseline moved. _MAX_RATIO, _BASE, _PREFIXED_BASE and every frame baseline are unchanged. The one unit change is the thousand-names bound: still 5.0, now seconds of CPU time instead of wall time. CPU time can't exceed the wall time of a single-threaded parse, so the bound is no looser.
• The 28 untimed tests (frame, call and shift counts plus the table checks), and the new _ratio test, run in the parallel suite now, so a plain uv run pytest holds the _CALL_BASELINE band, the link pin, the reread ratios and the shift guards. Only the four timed test functions keep the serial mark (-n0 -m serial, unchanged in CI).
• Docs updated: the module docstring, the _CLOCK/_ratio comments, the thousand-names docstring, tests/conftest.py, the marker description in pyproject.toml, the workflow comment, and the root AGENTS.md Commands block and the "Perf regressions" gotcha. CONTRIBUTING.md's command line didn't need to change. decisions/parse-cost gets a dated entry and its Standing backstop line is amended.

Before and after (py3.11, 12-core Apple silicon: 6 performance + 6 efficiency cores)

"Busy N" means N shell while :; do :; done loops started beside the run. The machine was never idle: other sessions held load at roughly 5–45 throughout, so "quiet" means no busy loops of mine.

Five runs of uv run pytest -n0 -m serial tests/v2/test_benchmark.py per row:

Busy 12 barely loaded the machine (macOS spreads 12 loops over 12 cores, leaving the test its own core). Busy 24 is the case that reproduces the reported failures.

Per-row ratios on the same samples, three runs at busy 24, load 18–75:

• wall clock: 27 of 33 rows read ≥ 6.0 at least once, max 14.0x
• CPU clock, sequential best-of-7, measured back to back with the wall clock: max 5.76x. It read 5.76x with no busy loops too.
• CPU clock, interleaved best-of-7: max 5.59x over 99 more readings (load 30–59)

CPU time is still noisy under contention. A clean row (~4.0x) read up to 5.6x, about +40%. The likely cause is a thread moved onto an efficiency core or a cold cache, which is charged the slower run. Setting the thread's QoS to user-interactive didn't help, so it isn't in the PR.

The noise runs both ways. With extra rounds taken only at or over the bound (this PR's first push), a planted 6.7x quadratic (given_part_titles) read 5.93x once at busy 24 and passed. The first push's loop could also stop on a dip under the bound. Review found both; the final _ratio fixes both:

• the weak plant's 11 rows read 7.18x or more in all 5 runs at busy 24
• clean master reads ≤ 5.06x

Every guard still fails on its regression

Each control below was run through the shipped estimator (_ratio, or a verbatim copy for trees whose nameparser predates this module's tables) and through the old _best on perf_counter, quiet and at busy 24, three runs each. Historical controls ran on scratch worktrees of the named commit. Plants went into a scratch copy of master; nothing in this branch's nameparser/ was touched.

Quiet, the two clocks fail the same rows under every plant, within 0.35x. Rows reading under 6.0 quiet with a given plant mean the plant is too weak for that row, and the wall clock reads them the same way.

Two notes on coverage:

• 962301f (Fix the quadratic behind the title_chain and link_run CI failures #1036's memory move) reads 4.07–4.43 on CPU and 4.07–4.50 on the wall clock, quiet, as decisions/parse-cost recorded for the wall clock. That defect was only ever visible on CI's runners, and _ShiftCountingList holds it now. CPU time does count memory stalls, so CI should see it as before.
• bracket_joined_members and maiden_clause have no recorded control I could re-run. Neither the word-count plant nor the cut plant changes how their cost grows (each token is short), so they're covered only by the general result that the clocks agree when quiet.

Frame tests in the parallel run

They pass:

• each alone in a fresh process (-n0 path::test, all 28)
• after test_cases.py and test_match.py in one process (21,065 tests before them)
• in three full parallel suites (load 8–45)
• under --cov on 3.11 (sys.settrace) and on 3.14 (sys.monitoring, CI's coverage interpreter), in parallel and -n0

Decisions/parse-cost (#1170) records that an absolute count depends on what the process parsed before. The guards hold anyway because each one is one of:

• a ratio
• a band (the reference band averages over 50 parses; the link pin read 2,395–2,405 against 2,370–2,466 in the test review's probe)
• an exact count of one named function's calls (only=/caller=, == 0 up to == 6); none of those functions is lru_cache'd

Their code is unchanged; only the mark moved. The source-tree and installed-package CI jobs, which run only the parallel pass, now run these 28 tests too.

Reviews

• Tests (pr-test-analyzer): no test dropped from either pass (62 → 34 serial + 28 untimed, disjoint); conftest deselection still behaves. Its optional-stopping finding is fixed (above). Two findings were addressed without changing code:

  • CPU time misses off-thread or blocked time; the thousand-names docstring now says so.
  • bound is passed separately from the assertion. Both read _MAX_RATIO and there is one caller, so this was left as is.

• Comments (comment-analyzer): wording findings fixed.

  • The guard forms include exact pins up to == 6.
  • "read the same" became "passed".
  • The extra rounds are described as not a retry.
  • The wall-clock tables before 2026-11-11 are labelled as such.
  • Stale "one-second bound" and "800 times" wording fixed.

• Design docs (design-docs-reviewer): the decisions entry was rewritten from the raw outputs: the 0.05x claim became 0.35x, 962301f's wall clock reads 4.07–4.50, the count after the 21,065 tests is corrected, and the deflation miss is recorded. That reviewer's deflation finding is what led to the _NEAR rule.

• Fix commit (code-reviewer): no case the first version refused now passes. It found three things, all fixed:

  • the boundary row passed only by float drift; it now uses 4.7 and 4.9
  • AGENTS.md still said 21,093
  • the prose claimed the ratio moves monotonically toward the truth; it converges, not monotonically, which is why it is judged once

Checks

• uv run pytest: 45,465 passed, 38 skipped, 4 xfailed (3.11)
• uv run pytest -n0 -m serial: 34 passed (3.11)
• uv run mypy, uv run ruff check: clean
• 3.14 with --cov: both passes green (the module re-run after the review fixes)
• 3.15 in its own UV_PROJECT_ENVIRONMENT: both passes green

🤖 Generated with Claude Code