Skip to content

feat(ENG-RECORD-ANCHOR-RATCHET): parse the citation form the checker never read, and ratchet its rot - #851

Open
localai-bot wants to merge 1 commit into
mainfrom
row/anchor-ratchet-impl
Open

feat(ENG-RECORD-ANCHOR-RATCHET): parse the citation form the checker never read, and ratchet its rot#851
localai-bot wants to merge 1 commit into
mainfrom
row/anchor-ratchet-impl

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Closes the implementation half of #632, row ENG-RECORD-ANCHOR-RATCHET
(SPIKE -> ACTIVE), against the committed spec
.agents/specs/record-anchor-ratchet.md.

The defect, re-derived

check-agent-record.py looked like it validated the record's file.cpp:123
citations. LINK_RE matches markdown links, so local_line_anchors only ever
saw [label](path#L505). Re-derived over the five matrices at 0e8b15d56 by
counting path:line tokens inside backtick spans:

Matrix link anchors (seen) bare path:line (invisible)
engine-matrix.md 19 1100
model-matrix.md 14 658
kernel-matrix.md 143 179
quantization-matrix.md 182 52
backend-matrix.md 121 145
total 479 2134

17.2% of citation forms were examined. Three gaps compounded: the bare form
was unparsed; is_code_anchor answers with any(), so one good link covered
every rotted citation beside it; and EVIDENCED_STATES omits ACTIVE and
READY, so 83 live rows got no anchor check at all. Inside the fraction that
was checked, only the RANGE was checked, which is why three stale anchors
survived the 2026-08-13/14 campaign and had to be caught by a human reading.
All three were in range.

What this adds

  • Parser for bare `path:line` and `path:line-line` beside the
    existing link form (BARE_CITATION_RE, cell_citations).
  • Classifier per resolvable citation: OK, STALE (line exists, symbol not on
    it), BROKEN (out of range, or a filename gone from a directory we own).
  • Baseline scripts/record-anchor-baseline.json, gated in both directions
    like scripts/device-leakage-baseline.json: above it fails, below it fails
    until the baseline is lowered in the same commit.
  • --report printing every offender with file, line and expected symbol.
  • Gap 3: ACTIVE and READY join the counted states.

is_code_anchor's any() stays for the STATE gate. A row is still evidenced by
one good anchor; the ratchet counts every citation independently, which is the
half any() was hiding.

The measured backlog

40: 33 STALE, 7 BROKEN, over 800 OK. Every one of the 40 was verified by
hand against the cited file before the number was written down. Examples:

  • scripts/check-commit-trailers.py:38 names FORGE_ACCOUNT_EMAIL, which is at
    :47.
  • src/vllm/entrypoints/openai/server_main.cpp:289 names kAcceptedInertArgs,
    which is at :277.
  • src/vllm/model_executor/models/registry.cpp:10-20 is gone; the file is now
    model_registry.cpp.
  • Five rows cite examples/server/main.cpp at lines 96 to 247. That file has 23
    lines.

They are deliberately not repaired here. The ratchet exists so they are
fixed by whoever next touches each row, which is the design the spec chose over
a bulk cleanup.

Where the conservative line is drawn

Precision over recall at five measured points:

  1. Only the code and tests cells are read, never upstream. That is what
    keeps 1446 upstream vllm/...py:123 references out of the count
    structurally; tests/, cmake/, docs/, src/ and tools/ all collide
    with upstream references here, so a path heuristic cannot do it.
  2. A bare citation must be the whole of a backtick span.
  3. A missing file is BROKEN only at depth 3+ with an existing parent directory,
    so llama.cpp's src/llama-model.cpp is skipped rather than blamed on us.
  4. A symbol comes only from an immediately adjacent backtick span.
  5. That span must look like a symbol: 4+ chars carrying _, ::, () or an
    uppercase letter, and not starting with _. The leading-underscore rule was
    added after a measured false positive on the text-only `_ModelInfo`,
    where the anchor was right.

About six citations in seven yield no symbol and are OK by construction.

Tests and mutation

Nine cases in RecordAnchorRatchet, all captured RED before implementation
(AttributeError: module 'agent_record' has no attribute 'scan_record_anchors'
x7, then 2 more). Five mutants red them:

Mutant Cases it reds
ratchet becomes report-only test_new_rot_fails_the_gate, test_a_repair_fails_until_the_baseline_is_lowered
RECORD_ANCHOR_STATES back to EVIDENCED_STATES + test_active_row_anchors_are_counted, test_baseline_matches_the_tree_exactly
links-only parser 7 of 9
stop at the first citation per cell (any restored) + test_one_good_link_does_not_cover_a_rotted_bare_citation
range-only classifier (no symbol test) 4 of 9

The load-bearing case is the cell holding one good link beside one rotted bare
citation, the exact shape the rot hid in. Links-only, first-citation-only and
range-only all red it. The tree was restored byte-for-byte after each mutant
(sha256sum -c OK, five times plus a final check).

Wiring

scripts/agent-preflight.sh gates it through check-agent-record (a red there
names the bucket that moved, and run only shows 12 lines, which is why the
offender list is not wired there). The agent-record CI job now calls
--report, the way check-device-leakage --report is wired below it.

The ratchet is a mode of the checker rather than a new script because it needs
parse_claim_rows and field_index. A separate checker would have re-derived
both and dragged a CREATION_MUTATIONS entry through check-pr-size.py for no
gain.

Rebase onto #846, and the ratchet catching its own author

main moved the issue index out of roadmap_v1.md while this was in flight.
Resolved by taking main's roadmap_v1.md wholesale and re-appending #632 to
.agents/issue-index.md; tests/scripts/test_agent_record.py was a union (both
sides appended a new class at the end, neither replaces the other).

The rebase then shifted every line this row's own Our code cell cites, and the
ratchet reported all seven as STALE and reddened the gate. That is the first
thing it did on landing, so the row is repaired and the count is back at 40.

Gates

Host and load stated, because one of them is load-dependent.

Gate Host Result
scripts/check-agent-record.py mudler-ubuntu-box, 20 cores OK ... ANCHOR-ROT=40
python3 tests/scripts/test_agent_record.py same OK
python3 tests/scripts/test_check_gate_commands.py same 36 tests, OK
scripts/agent-preflight.sh same, load 20-30 All gates green on the pre-rebase tree; on the rebased tree, 1 red: test_cpu_x86_llamacpp_floor
check-commit-trailers.py --range <merge-base>..HEAD same OK, run explicitly per #653
check-commit-style.py --range <merge-base>..HEAD same OK
check-pr-size.py --base origin/main --head HEAD same OK, including executable red-before/green-after for both changed checkers
tests/scripts/test_cpu_x86_llamacpp_floor.py Thor kairos-4db2, 14 cores, load 3.6 10 tests, OK
RecordAnchorRatchet Thor, load 3.5 9 tests, OK

test_cpu_x86_llamacpp_floor is #618 and is not this change: the failing case
is test_the_quiet_gate_does_not_see_the_harnesss_own_process_tree, which exits
NO_QUIET_WINDOW because the harness's own quiet-window precondition never
arrives on a loaded box (busy=150% load=28.98). It passed on this same tree
content at load 20 on the dev box, and the identical tree was shipped to Thor
with git archive | ssh tar -x and passes there at load 3.6. No changed path
touches that harness.

Note for anyone re-running the record gate from a git archive tree: it has no
.git, so commit_exists() reports every DONE row's closing commit missing.
That is the instrument, not the record.

No GPU, no build, no network: this is a Python checker and its suite.

🤖 Generated with Claude Code

…never read, and ratchet its rot

`check-agent-record.py` looked like it validated the record's `file.cpp:123`
citations and did not. `LINK_RE` matches markdown links, so `local_line_anchors`
only ever saw `[label](path#L505)`. Re-derived over the five matrices at
`0e8b15d56`: 479 link anchors carrying an `#L` fragment against 2134 bare
citations inside backtick spans, so 17.2% of the citation forms a reader sees
were examined as citations at all.

Three gaps compounded. The bare form was unparsed. `is_code_anchor` answers with
`any()`, so one good link in a cell covered every rotted citation beside it.
And `EVIDENCED_STATES` omits `ACTIVE` and `READY`, so 83 live rows got no anchor
check. Inside the fraction that was examined, only the RANGE was checked --
which is why three stale anchors survived the 2026-08-13/14 campaign and had to
be caught by a human reading. All three were in range.

This parses both forms, classifies each resolvable citation as OK, STALE (the
line exists but does not contain the symbol named beside it) or BROKEN (out of
range, or a filename gone from a directory we own), and gates the STALE+BROKEN
count against `scripts/record-anchor-baseline.json` the way the DSR ratchet in
`check-device-leakage.py` gates leakage: a bucket above its baseline fails, and
a bucket below it fails too, so a repair must lower the baseline in the same
commit. Measured backlog at adoption is 40 -- 33 stale, 7 broken, over 800
correct -- and every one of the 40 was verified by hand against the cited file
before the number was written down. They are deliberately NOT repaired here;
the ratchet exists so they are fixed by whoever next touches each row.

`is_code_anchor`'s `any()` stays for the STATE gate: a row is still evidenced by
one good anchor. The ratchet counts every citation independently, which is the
half `any()` was hiding.

The symbol test is conservative by measurement, not by taste. It reads only the
`code` and `tests` cells, never `upstream`, which is what keeps 1446 upstream
`vllm/...py:123` references structurally out of the count instead of relying on
a path heuristic that `tests/`, `cmake/`, `docs/` and `src/` all defeat. A
symbol is inferred only from an immediately adjacent backtick span that looks
like an identifier -- 4+ characters carrying `_`, `::`, `()` or an uppercase
letter, and not starting with `_`, which was added after a measured false
positive on `` the text-only `_ModelInfo` ``. About six citations in seven yield
no symbol and are OK by construction. That polarity is the point: a checker that
cries wolf gets disabled, and this one has to survive a four-figure backlog.

Nine cases in `RecordAnchorRatchet`, all captured red before implementation --
the six the spec names plus the two gate directions and the tree/baseline pin.
Five mutants red them: report-only, `EVIDENCED_STATES` restored, links-only,
first-citation-only, and range-only. The load-bearing case is the cell holding
one good link beside one rotted bare citation, which is the exact shape the rot
hid in; links-only, first-citation-only and range-only all red it.

The ratchet is a mode of the checker rather than a new script because it needs
`parse_claim_rows` and `field_index`; a separate checker would have re-derived
both and dragged a `CREATION_MUTATIONS` entry through `check-pr-size.py` for no
gain. It runs in `agent-preflight.sh` through `check-agent-record`, and the
`agent-record` CI job now calls `--report` so the offender list is printed on
every run, the way `check-device-leakage --report` is wired below it.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants