Keeps the hand-written docs (content/docs/** minus content/docs/references/**)
in sync with the actual implementation in packages/** as the platform evolves.
Generated references (content/docs/references/) are produced from packages/spec
and are out of scope here — regenerate those separately.
The system has four parts, layered cheapest-and-earliest first:
Maps a set of packages/** changes to the hand-written docs that name something the
change touched, so an audit can be scoped to what actually changed.
# docs affected by changes on this branch vs origin/main
node scripts/docs-audit/affected-docs.mjs origin/main
# JSON (with changed packages + per-doc "why")
node scripts/docs-audit/affected-docs.mjs --json origin/main
# every hand-written doc (full audit scope)
node scripts/docs-audit/affected-docs.mjs --all
# pin the classifiers, package-root and anchor derivations (needs no repo state; CI runs this before the mapping)
node scripts/docs-audit/affected-docs.mjs --self-test
# how much of the DECLARED client-bound route surface the `sdk` bridge can reach (diff-free)
node scripts/docs-audit/affected-docs.mjs --bridge-coverage
node scripts/docs-audit/affected-docs.mjs --bridge-coverage --json # + the unreachable rows themselves, each with WHY and its witnessDerivation (#9192): a doc is affected when it NAMES something the change touched.
Not when it mentions the changed package — that predicate is a dependency-graph proxy
answering a semantic question, and it was measured wrong in both directions on PR #9191
(three read verbs in @objectstack/metadata-protocol): 3 pages listed of which 1 was
relevant, while the 2 pages that actually document the changed surface —
api/client-sdk.mdx and kernel/contracts/metadata-service.mdx — were absent, because
they document it through the SDK surface, which does not depend on the implementing
package at all.
Over-inclusion is not free, and that is the correction. A wrong-both-ways advisory trains its reader to skip it, and then it fails on the PR where it is right — the same bill exclusion 1 below already paid. The derivation is therefore precision-first: a shorter right list beats a longer noisy one.
Three anchor kinds, each exact:
| anchor | what it is | how it is derived |
|---|---|---|
symbol |
a documentable declaration the diff touched | the top-level declaration, or a member of a top-level container (class / interface / type / enum / schema object), enclosing each changed line — on both sides of the diff, so a removed export still anchors the pages naming it. A member that is a data property is additionally qualified by its declaring container against the authorable surface (see below) |
route |
a wire path the change touched | a path literal on a changed line, plus every route whose route-source handler references a changed symbol |
sdk |
the client method bound to an anchor route | the declared route ⟷ client rows in the repo's route ledgers |
The route and sdk hops are what carry the derivation across the surface boundary the
package graph cannot cross: auditMetaItem (changed) → GET /api/v1/meta/:type/:name/audit
(rest-server.ts, a registration call site) → meta.getAudit (rest-route-ledger.ts) → the token
api/client-sdk.mdx actually contains.
A local variable is not documentable surface. That one rule is what drops the measured
false positive: const singular = request.type; inside a method body is not an anchor, so
kernel/services-checklist.mdx — whose only singular is a service slot name — is no
longer listed. A const object is a container (its keys are metadata property names,
which docs do name); a function body is not.
A row used to read organizationId (symbol), and a reader had no way to tell the single
most on-target anchor the tool mints from pure noise. Now it reads:
- `content/docs/deployment/seed-tenancy-repair.mdx` _(via organizationId (symbol, a field of interface MetaOverlayCacheKey))_
- `content/docs/data-modeling/objects.mdx` _(via userActions (symbol, a field of const object ObjectSchemaBase))_
Those two are the same syntactic form — name: inside an object or interface — and
that is the point. One is a field of an internal cache struct and lands 10 pages that
document an unrelated organizationId; the other is the canonical authorable key, and its
row is the best one this tool produces. No syntactic test separates them (measured: the
declaring package does not either — schemaMode is authorable and lives in
packages/objectql). The declaring container does, and it is the field no row printed
before.
Each anchor kind names its own origin, from the same field the JSON publishes as
anchors[].from — one derivation, never a second spelling:
| kind | the clause |
|---|---|
symbol |
a field of interface MetaOverlayCacheKey · a method of class RestServer · a top-level function |
route |
a path literal in RestServer · bridged from symbol enforceEnvironmentOwnership — its route-source handler names it |
sdk |
the route ledger binds it to GET /api/v1/ui/view/:object/:type |
literal |
a string literal in cacheKeyOf |
command |
read off packages/cli/src/commands/environments/bind.ts |
rule |
a @docs-rule block in packages/objectql/src/engine.ts |
The bridge clauses are load-bearing for the same reason: an sdk row on a diff that never
came near that route is the amplification this machinery is measured to produce, and it is
only judgeable when the row names the hop it rode.
⛔ This is publication, not discrimination, and the difference is the ruling. The
maintainer's decision of 2026-08-31 took this (option C) and ruled OUT filtering on it
(option B): dropping data-property anchors buys a 17.9% shorter list and pays in the
userActions → data-modeling/objects.mdx and schemaMode → data-modeling/drivers.mdx
rows. A false positive costs a reader a minute; a false negative ships a falsified page.
So no guard, no threshold and no bridge hop reads from — --self-test pins that the
provenance key set is exactly the anchor set, in both directions, so a future change
cannot start deciding with it without going red. Container-qualified discrimination is
the separate, later step below (option D), and it decides from the declarations directly,
never by parsing this clause back out.
Option D of the same ruling, and the half that does move rows. It landed once the spec
side (#13712) published the mapping it needs: packages/spec/declaration-map/*.json maps a
TS declaration name to a spec type name (ObjectSchemaBase → data/Object), and
the authorable surface is keyed as container:property (data/Object:userActions). All of
it is generated and covered by check:generated; this script only ever reads it, and ⛔
never carries a local mapping table of its own — a container the map lacks is a spec-lane
card, not a local fix.
packages/spec/authorable-surface/
(the live per-category ratchet, read as one set) and
packages/spec/authorable-surface.base.json (the artifact #12824 names). The second is an
anchor, pinned at a fixed baseRev for the deletion gate, so it is missing every key
authored since — check:authorable-surface prints that lag on every run, and ⛔ a
count copied into prose here rots (that gate is the live instrument; this sentence is
not). What the lag is does not rot, and is what this rule needs: data/Object:editMode
and every key of security/OrgScopingEntitlement and api/ProvenanceWaiver are absent
from the anchor and present in authorable-surface/. Reading the anchor alone would
suppress anchors on genuinely authorable keys, and that class grows with every key added
after baseRev. A union can only ever keep an anchor one source vouches for, never drop
one more. [RETIRED] is stripped for the same reason: a tombstoned key still rejects with
an upgrade prescription, so it is still surface a page documents.
For a data property prop (name: / name?: / name =) whose most specific enclosing
declaration is the container C:
C resolves to |
C:prop authorable |
verdict | |
|---|---|---|---|
| (a) | cat/T |
yes | mint — today's behaviour, now justified. The row also says so: a field of const object ObjectSchemaBase, an authorable key of data/Object |
| (b) | cat/T |
no | drop the property anchor and fall through to the container branch that already existed — no new fallback |
| (c) | nothing, or two different types | — | mint, unchanged |
⛔ (c) must never become "drop". That is option B (blanket prefer-container) by another name, and it is vetoed on two rounds of measurement: its "zero recall loss" was measured against a ground truth of 10 of 46 pages, most of which were never in this tool's corpus at all (the real recall is 48.8%, re-derived), and the rows it drops are the most valuable ones the tool mints. Fail toward noise, never toward silence. A method, a nested interface, a namespaced type — anything whose winning declaration is not a data property — is outside the rule entirely.
Two consequences worth stating plainly, because both are easy to mistake for bugs:
- Both ruled true positives survive, and they survive by different routes.
userActionsis a key ofObjectSchemaBase, which the map carries, so it is kept by (a) —data-modeling/objects.mdxis still listed and the row now names the spec key.schemaModeis declared onDatasourceDef, an objectql-local interface the generated map does not carry, so it is kept by (c) — the unmapped keep, not the authorable lookup.--self-testpins that absence as a fact rather than papering over it; the day the map carriesDatasourceDef,data/Datasource:schemaModeis already an authorable key and the row survives via (a) instead. - The realised reduction is far smaller than the −17.4% the projection quoted, and for
the same reason: the projection hand-classified the 20 dropping containers, and 18 of
them are internal implementation types (
MetaOverlayCacheKey,LocalizationCacheEntry,AUTH_MODEL_TO_PROTOCOL, …) that the generated map does not carry either. Under the ruled rule those are case (c) and they keep. Only containers the map resolves can drop. Widening the drop set to reach the projected number would be option B; the number moves when the map's coverage grows, which is spec-lane work.
Every drop is published in containerQualifiedDrops, naming the property, its container
and the spec key that failed to be authorable — the same discipline as the two guards
below. A deliberate false negative that no field names is one no reviewer can audit.
If either artifact is unreadable (this script runs in contexts where packages/spec may be
absent) every container reads as unmapped, the run behaves exactly as it did before this
existed, and one note goes to stderr. --self-test pins that degradation too.
(Three narrowings publish now — the container qualification above reports
containerQualifiedDrops on the same terms. These two are the ones that run over the
whole anchor set, in every kind, after it is derived.)
The first build of this derivation was, on some PRs, noisier than the proxy it replaced (134 rows where the old tool gave 26). Two guards fixed that, and both run before the route bridge — a name left in the set does not merely add a noisy row, it mints noisy route and SDK anchors from every route-source handler that mentions it:
- Shape — an anchor must be code-shaped (camelCase / PascalCase / snake_case /
dotted).
label,object,start,localeandsectionsall arrived as real declarations and matched 82, 113, 43, 13 and 10 of 178 pages; confining them to code spans does not help, because those words live in code spans too. Reported asweakAnchorsDropped. The recall cost is a genuinely lowercase export (parse,mask). - Corpus share — an anchor matching more than 15% of the corpus is a hub term, not an
identifier.
ObjectQLis code-shaped, genuinely changed, and named by 59 of 178 pages; it cannot tell an author which page to re-read. Reported asoverbroadAnchors, with the count that condemned it.
Plus a cap on the route bridge itself: a symbol wired into more than 3 routes is a
cross-cutting helper, and "which routes mention this name" then answers every route.
Reported as crossCuttingSymbols. SCREAMING_SNAKE constants are kept out of the bridge
entirely — a data table is consulted by handlers, it is not their implementation.
The sdk hop needs a route source's path: tail to select a route-ledger row. Measured on
9ff11921a: 45 of the 221 client-bound ledger rows are reachable, 176 are not. An
unreachable row is not "unlisted this time" — no symbol change bridges to it, ever.
8f10a79f7a the same command reads 47 of 219, and the difference is not recognizer
drift — each move was measured row by row:
--bridge-coverage |
a6eca9223 |
8f10a79f7a |
why |
|---|---|---|---|
| route sources | 12 | 12 | two ADR-0049 ledger entries were admitted in between and are excluded again here; they produced 0 tails and 0 reachable rows, so they never moved the figures below |
| route tails | 43 | 44 | rest-server.ts unrolled for (const publishedPath of […]) into a literal path: — a variable path yields no tail, a literal one does |
| client-bound rows | 222 | 219 | three :type/:section/:name rows deleted from rest-route-ledger.ts, all three already unreachable |
| reachable | 45 | 47 | the one new tail /:type/:name/published selects meta.getPublished on the rest ledger and on the runtime ledger |
⭐ 45 → 47 is the bridge reaching more of its population, not losing track of it, so the figure stands at 47. ⛔ Do not "restore" 45: the only recognizer spelling that reproduces it drops ten of the fourteen matched files, including a tail-producing call site — the control appears to recover exactly when the recognizer stops working.
That number now travels with the answer. bridgeCoverage is emitted on every run whose
change carried a bridgeable symbol ({ measured: false, reason } when it did not — never a
fabricated zero), the drift comment renders it in What this run could not see, and
--bridge-coverage answers it with no diff at all.
Two things it deliberately is not:
- Not a verdict. The ratio is reported. Failing CI on it would widen the recognizer
under CI pressure, which the #9747 family declines explicitly. What does exit non-zero
is
brokenScan— no ledger found, no tail produced, a ledger file matching the convention that the row recognizer parses to zero rows, or a partial read (next section). Those cannot fire on a tree where the scan works at all, and each of them otherwise reports0 of 0 unreachable, which is arithmetically true and reads exactly like a healthy bridge. - Not a fix for the 176. The causes are unrelated to each other and each needs its own before/after measurement per #9432's standard. They are no longer estimated: the run splits them (next section).
Half the blind spot is load-bearing: 88 of the 176 unreachable rows name a client
method that at least one hand-written page carries (31 distinct pages, api/client-sdk.mdx
among them), so the silence is not an empty region.
56 of 56 and 46 of 87 used to print in the same words, and they are not the same
finding. Every unreachable row is now attributed against a ceiling — every path: any
packages/** file declares, with CALL_SITE_FILE_RE ignored entirely, built by
maximalTailsFrom from the same parseRouteSource over the same walk. Measured on
589758d22, the 177 unreachable rows partition as:
| cause | rows | what it means |
|---|---|---|
discovery-gap |
14 | an in-repo file declares this exact path; the filename convention did not scan that file. The JSON names the witness. |
no-in-repo-declaration |
56 | on a ledger where not one row is declared in-repo — declared upstream and catch-all-mounted. No discovery change reaches it. |
undecided |
107 | no in-repo declaration for the row, on a ledger that has in-repo route sources. Absence and an unreadable spelling are not distinguishable here, so neither is claimed. |
Exactly one of the seven ledgers is no-in-repo-declaration today: auth-route-ledger.ts,
whose own header has said so since #3656 — better-auth declares those routes inside
node_modules and the plugin mounts them with a single rawApp.all(`${basePath}/*`),
which routeTailOf cannot and should not turn into a tail. That is why widening
CALL_SITE_FILE_RE to admit auth-plugin.ts was measured to move route sources scanned 12 → 13 and nothing else.
⛔ This changes no discovery and moves no reach. CALL_SITE_FILE_RE is byte-identical,
the bridge still rides on routeSourceByTail alone, and reachable is 45 before and after —
pinned in --self-test. The ceiling only explains the number; it never participates in it,
and because it is a superset by construction a ceiling that misses a reachable row is a
brokenScan verdict rather than a quieter result.
The classification is derived, never listed. Control on 589758d22: adding one
in-repo file that declares one auth route — under a filename the convention does not match
— moves the auth ledger out of no-in-repo-declaration on its own (structural 56 → 0,
reachable still 45), and removing it restores 56.
Maintainer ruling A, 2026-09-04 decision batch #31. Until this card the recognizer's name,
docblock and --self-test all meant the file that registers the route — while the
measured widening that motivated the card admits five packages/spec Zod contract
declarations, which register nothing. The ruling made the rename a condition of
admitting them: nothing here is called a "registrar" any more, because the word would
otherwise denote two constructs.
A route source is a file whose source declares a route. Two kinds:
| kind | admitted by | today |
|---|---|---|
| registration call site | the CALL_SITE_FILE_RE filename convention (unchanged) |
12 |
| spec contract declaration | evidence: a non-test packages/**/*.ts whose masked source declares a route beside the HTTP method it answers |
5 |
Why a declaration counts: the drift check exists to say "the contract changed, re-verify
the manual", and a Zod API declaration in packages/spec is the contract. No widening
of the filename convention would ever reach it — storage.zod.ts is not going to be
renamed storage-routes.ts.
The guard is the HTTP method, not a file list. A route declaration names the verb it
serves; a data payload carrying a path: key does not. Measured on 460134af8: of the 61
literal path: sites in packages/spec/src/conversions/registry.ts — whose
/api/v1/health is a connector-action input inside an automation fixture — zero
carry an HTTP method, while the five contract declarations carry one at 61 of their 62
sites. (The 62nd has its method: five lines but zero properties up, behind a JSDoc
that maskComments blanks; the lookaround skips blank lines for exactly that reason.) The
separation is total at every lookaround from 1 to 6, so the constant is a margin, not a
threshold.
⛔ The fixture and the benchmark are not re-excluded here. The card also named
test/fixtures/*.ts and *.bench.ts; isTestFile grew those arms in #12965, so the walk
never offers them to either kind. --self-test pins that over the two real repo paths
rather than restating the exclusion — the day either arm is loosened, they come back as
contract declarations and the self-test says so.
Measured on 460134af8, before → after:
--bridge-coverage |
before | after |
|---|---|---|
| route sources scanned | 12 | 17 (12 call sites + 5 contract declarations) |
| route tails produced | 44 | 78 |
| client-bound rows reachable | 47 | 61 |
discovery-gap rows |
14 | 0 |
storage ledger unreachable |
7 of 7 | 0 of 7 |
i18n ledger unreachable |
3 of 3 | 1 of 3 |
rest ledger unreachable |
42 of 84 | 40 of 84 |
runtime ledger unreachable |
64 of 69 | 61 of 69 |
plugin-auth unreachable |
56 of 56 | 56 of 56 (unchanged — declared upstream, see above) |
No reach regression is possible by construction: selectsFrom is a some() over the
tail list, so added tails can only add selections. discovery-gap reaching 0 is the
strong reading — every row the ceiling said a widening could reach is now actually
reached, and what remains unreachable is unreachable for a reason no discovery change
touches.
45 → 59 figure from the card is not what landed. The card measured
on a6eca9223; both endpoints have since moved with the tree (see the #9572 table above),
and the guard admits five files rather than the card's eight. The figure this PR
establishes is 47 → 61 on 460134af8, re-measured rather than copied.
The row recognizer reads single-quoted values only, and the rowsParsed === 0 guard
above is the all-or-nothing case. The likelier shape is a ledger that parses almost
completely — and it used to render exactly like a complete one. Measured on a718ee3dd by
respelling one row of i18n-route-ledger.ts:
| ledger written as | client-bound rows | verdict | exit |
|---|---|---|---|
| all single-quoted (today, all 7 ledgers) | 221 | none | 0 |
one row backtick-quoted route: |
220 | none (now: PARTIAL read … 2 of 3) |
0 → 1 |
one row backtick-quoted client: |
220 | none (now: PARTIAL read … 2 of 3 client) |
0 → 1 |
one route: backtick-quoted in rest-route-ledger.ts |
221 |
none (now: PARTIAL read … 95 of 96) |
0 → 1 |
The last row is why the verdict keys on the declined spelling and not on a shortfall:
the row window is delimited by the same single-quote-only lead, so a declined row does not
close the previous row's window and the row before it inherits the declined row's
client. Backtick-quoting GET /api/v1/meta moved meta.getTypes onto the server-only
GET /api/v1/docs while clientRows stayed 221 — a count comparison is blind to it.
A template literal is the realistic spelling: the formatter rewrites double quotes back to
single, but leaves route: `${base}/locales` alone, and that is the natural shape the
moment anyone interpolates a base path into a row.
So both halves of the fraction now print on every run — ledger rows read … 259 of 259 declared — and a declined declaration is a brokenScan verdict that names the entry
(line 96: route: \GET /api/v1/i18n/locales`). The recognizer itself is untouched: this reports its narrowness, it does not widen it, and --self-test` pins the population at its
old value so a silent widening fails there.
Values in no quote at all are counted too. A route: / client: whose value is not a
string literal (route: ROUTES.health, route: BASE + '/x') is read by neither the
recognizer (which needs a leading ') nor the declined-spelling counter (which needs a
leading " or backtick). Such a row used to leave the population with no verdict at all —
not read, not declined, absent from the denominator. It is now reported as unread, so the
row ends up in a verdict either way.
The route: string; member each ledger's own entry interface declares is not a row, and
is excluded on a structural discriminator rather than on the quote: a type member sits inside
an interface/type declaration and a table row never does. Comments and string payloads
are masked out for the same reason — runtime/src/route-ledger.ts contains the English
sentence "It never named a mounted route: the branch". Both exclusions are pinned in
--self-test in both directions; removing the type-declaration one turns all seven of
today's accurate ledgers red (259 of 266), which is why a bare "count every route:"
check was never an option.
The row recognizer reads through both of those exclusions too — comments and string
payloads since #10683, type declarations since #10793. Before each, the recognizer read a
source the rest of the file had already ruled out, and a lead it should not have read did
not merely mis-count: it became a row. Both were silent for the same reason — rows
and the first term of routesDeclared moved together, so the partial-read verdict, which
keys on the gap between them, had no gap to see. The type-declaration case is the one a
quote test cannot catch on its own: a literal-union member (route: 'GET /a' | 'GET /b')
opens with the very quote the recognizer reads. Both moves were priced on a tree carrying
no instance of the shape, and 259 of 259 / 221 of 221 / 176 unreachable came out
byte-identical across each.
And the key itself is anchored — since #11542, in one place rather than eight. Eight
scans ask "is a route: / client: declaration written here?"; declLead has spelled the
colon and the run after it once since #11494, but the key stayed each call site's own
argument and only one of the eight anchored it with \b. So subroute: 'GET /api/v1/gone'
was a declaration to seven of them and not to the eighth, and it minted a phantom
row — silent for the same reason as the two above, rows and the first term of
routesDeclared moving together. Not all seven behaviours it moved are counting: the row
window is delimited by that same lead, so a subroute: written between a real route: and
its client: closed the real row's window and handed the binding to the phantom, which
then joined the unreachable population — a wrong binding no count comparison can see. Priced
on a tree carrying no instance: zero leads across the seven ledgers that the unanchored
spelling reads and the anchored one does not, and 269 of 269 / 222 of 222 / 45 reachable /
177 unreachable came out byte-identical row for row across the change. $route: is still
read as a declaration — \b fails only against a word character — but all eight agree on it
now, and --self-test pins that residue (#11630) where the next card will find it.
A skipped type member is reported nowhere, and that is the intended answer rather than
a new silence: it is a correct declaration of a type, not a table row — the same rule
under which the route: string; member of all seven entry interfaces has always produced
nothing. What the prose case gets instead is prose-quoted leads, because a lead sitting
where the mask says code is not is a would-be row and worth naming.
anchorlessChanges lists changed files that yielded no anchor at all; a non-empty value
means the list is incomplete by a known amount, and an empty docs beside it must
never be read as "no page documents this change". The superseded coarse set is still
computed and emitted as packageMentionDocs, labelled — an audit that deliberately wants
the wide net can still ask for it, and keeping it visible is how a reader tells a narrow
list from a blind one. The PR comment renders all of this in a collapsed section — and,
since #11357, renders the anchorless count in the headline as well, because a limit a
reader has to expand a fold to find is one a reader does not find. The failure #9192
records was never the tool lying; it was the tool never signalling its own limits at the
point of use.
Since #11356 the same posture reaches the ✅ itself — see The verdict when nothing names the anchors below.
Ten real PRs, each re-derived at its own merge base with its own docs corpus. docs rows:
| PR / commit | old (package-mention) | new (anchor) |
|---|---|---|
| #9191 — the three metadata read verbs (the filing card's specimen) | 4 | 3 |
0668f02a6 fix(rest): closed ErrorCode union on the error responder |
26 | 14 |
75b7c240a feat(spec): master_detail + controlled_by_parent |
113 | 32 |
07ad42463 fix(cli): os meta resync skip-count explanation |
22 | 0 |
7a537ce90 feat(spec): strict top-level stack keys |
113 | 13 |
445ae4deb fix(auth): auth emails follow the deployment locale |
13 | 3 |
30b1c636a feat(spec): register 9 REST wire codes |
113 | 4 |
650cd3daa fix(objectql): delete-cascade registry reads |
14 | 0 |
3851f87f0 feat(spec,plugin-security): partial field masking |
116 | 19 |
d5156b965 refactor(metadata-protocol): drop dead objects tolerances |
4 | 4 |
The #9191 row reads 4 where the filing card says "the bot listed three pages": docs is
the full set and the comment partitions content/docs/releases/v9.mdx into its own
read-only section (#6893), so 3 editable rows + 1 release-owned row = 4.
On #9191 the change is qualitative, not just smaller: all three previously-listed pages
are gone and the two pages the filing card measured as missing are back, each with the
anchor that put it there (getAudit/getReferences for client-sdk.mdx, getHistory
for metadata-service.mdx).
The two zeroes are the honest shape of the trade, not a bug: 07ad42463 derives
MetaResync and resyncSkipExplanationLine, and no hand-written page names either, so the
run says so and points at the coarse set — where the old tool's 22 rows were every page
mentioning @objectstack/cli. A CLI command name (os meta resync) is exactly the
recall class the shape guard costs us: it is a lowercase word, so it cannot anchor.
That state — anchors derived, nothing left unanchored, no page naming any of them — used to end the headline in a ✅. It no longer does (#11356). The sentence reports the naming relation, and the relation only ever lists a page that ALREADY names a changed token, so a PR that widens an enumerable vocabulary is invisible to it by construction: the new members' absence from the page is precisely the defect, and an absence names nothing.
Measured on #11347 — six new flow-expression functions (round/floor/ceil/abs/
min/max) — the run derived 9 anchors, matched 0 pages, and rendered the ✅, while
content/docs/automation/flows.mdx carried a table enumerating the available bindings
that the same PR had just made incomplete. A reviewing seat almost passed the PR on that
tick.
check-drift-comment.mjs, which asserts the verdict byte-exact on that state and asserts
it ABSENT on all four neighbouring states.
How often it renders, re-derived over the 40 first-parent commits ending at e43b18fd9:
3 of the 17 package-touching runs (18%) — 20a452e664, f213793ddb, dd4113ec0b — so it
is a rare notice rather than a per-PR banner, which is what keeps it readable.
Cost (the card's open question): the anchor derivation reads the same 178-page corpus
the old one did, plus the 18 route-source/ledger files (~875 KB) and one git show
per changed file per side. Measured end-to-end on the ten PRs above, node affected-docs.mjs
went from 85-195 ms to 114-582 ms. The heaviest case is the widest diff; every case stays
well under a second, against a job that already spends seconds checking out the repo and
setting up Node. It is the right default for every PR.
How a changed file maps to its package: the package root is the deepest ancestor
directory with a package.json, resolved from the filesystem — never a hand-kept
list of container directories. (The mapper once special-cased only
packages/plugins/*; the 30 packages nested under the other six containers collapsed
into packages/services et al., whose missing package.json disabled the npm-name
matching arm entirely, so a doc naming @objectstack/service-automation but not the
repo path was a guaranteed miss — #4162.) A deleted package falls back to the coarse
packages/<x> token, which still substring-matches any doc naming the deleted path.
Three exclusions: change classes that cannot make an implementation-accuracy doc stale are dropped before the changed-package roots are derived:
-
Test files (
*.test.*/*.spec.*at any depth, plus__tests__/__mocks__/__fixtures__): a test observes behaviour rather than defining it — yet counting them made every tests-only PR light up its packages' whole doc set, a class of finding that is always false. That is the one place over-inclusion actively hurt: a comment a reader learns to skip stops working on the PR where it is right. -
Package tooling scripts (
<packageRoot>/scripts/**): build/verification tooling, not the runtime behaviour docs describe (#4183 flagged 106 docs for a diff whose only code change was a new check script). Narrow on purpose:src/scripts/**is runtime code and stays counted. No package publishes runtime code fromscripts/(checked against everyfilesallowlist; three plugins ship a lonei18n-extract.config.tsonly for lack of afilesfield). -
Dev-only manifest edits (#6893): a
<packageRoot>/package.jsonwhose changed top-level keys are all in{scripts, devDependencies}. This is the only field-level exclusion —package.jsonas a file stays counted, becauseexports/main/dependencies/files/versionchanges ARE implementation.It is the residue of exclusion 2: #4183 dropped the check script but kept the
package.jsonline registering it, so the same PR still lit up the same doc set through the manifest. Measured over 400 merged commits, five had apackage.jsonas their onlypackages/**implementation change, and all five touched nothing but those two keys — 152 doc-rows in total, none of which could be stale:commit keys changed docs flagged df0605ba5scripts12 2672f855fscripts113 — #6893's headline number a64315556devDependencies10 77d9001c7devDependencies13 466bd9285devDependencies4 The last three are
test(...)commits: exactly the class exclusion 1 exists to kill, leaking through the manifest instead. The allowlist is an allowlist on purpose — an unknown or newly-invented key falls on the counted side — and unparseable, added or deleted manifests are counted too.Why it cannot narrow the net: the classifier is per file. A PR that also touches that package's
src/**derives the package root from those files anyway, so this arm only ever decides the case where the manifest is the package's sole change. Verified both directions on the real diffs (#6893): adding anexportsentry topackages/spec/package.jsonstill flags 113 docs, and ascriptsentry alongside asrc/edit also still flags 113 — with the manifest itself reported as skipped.
The excluded counts are reported in the summary line and as testFilesSkipped /
scriptFilesSkipped / devOnlyManifestsSkipped in --json, so the narrowing is never
silent. --self-test pins the classifiers, the package-root derivation and the anchor
derivation against inputs that must and must not match (commands/test.ts is implementation;
foo.conformance.test.ts is not; a container directory must never come out as a package
root; dependencies is never dev-only).
And one deliberate non-exclusion: packages/*/CHANGELOG.md stays counted, even though
release notes define behaviour no more than a test does. Extending the exclusion there
looks like the obvious next step and is a provable no-op, for two independent reasons:
- The only PR class that mass-touches those files is
chore: version packages, and it runs no GitHub Actions at all —changesets/actionopens it with the repo'sGITHUB_TOKEN, and GitHub does not trigger workflow runs fromGITHUB_TOKEN-authored events. Measured on #3910: one check run, from Vercel's own app. So this gate never sees a release PR to be noisy on. (The bump is still verified —ci.ymlandlint.ymlboth run onpush: main, andrelease.ymlgates publish on a green build.) - Even if it did run,
changeset versionwritespackage.jsonnext to everyCHANGELOG.mdit appends to — 45 of the former against 46 of the latter on the first page of #3910's diff — so dropping the CHANGELOGs would leave the derived package-root set bit-identical. Exclusion 3 does not undercut this: whatchangeset versionrewrites isversion(and workspacedependenciesranges), neither of which is in the dev-only allowlist, so those manifests stay counted.
A hand-edited CHANGELOG outside a release is also close to nonexistent in practice. Left counted, and recorded here so the idea is not rediscovered as a gap.
node scripts/docs-audit/check-audit-scope.mjs # verify (also: pnpm check:docs-audit-scope)
node scripts/docs-audit/check-audit-scope.mjs --write # regenerate the list from the filesystem
node scripts/docs-audit/check-audit-scope.mjs --self-testTHE single source is scripts/docs-audit/handwritten-docs.json — one generated file,
with two consumers: this gate, and whoever invokes the docs-accuracy-audit workflow
(part 3), who reads it and hands its docs array in as args.handwritten.
The workflow body cannot read it itself. A workflow script runs inside a node:vm
context whose only globals are log/phase/console/budget/timers plus
agent/parallel/pipeline/workflow/args, with code generation disabled — no
require, no import, no filesystem. It can neither walk content/docs/ nor open a
JSON artifact. But it does not have to: args is an injection channel, delivered
verbatim from the invocation, so the read happens in the caller, outside the sandbox,
and the list arrives as data.
The list used to live inline in the workflow body, as ALL_HANDWRITTEN. .claude/** is
a governed surface (human-merge-only, never armed, never queued), so adding one customer
documentation page forced a governed edit — through a bookkeeping list that merely
happened to live there, and invisibly: nothing in such a card's file list showed a
governed path until this gate ran. Maintainer ruling, 2026-09-01, verbatim 「同意」: move
it off. .claude/**.
The list is derived at generation time: --write rewrites the artifact from
affected-docs.mjs --all (one definition of "hand-written doc", not two), and the plain
run is a CI gate in lint.yml that fails when the artifact and content/docs/ disagree
in either direction. It also runs the workflow against stub agents and checks that the
body still consumes what it is handed — an artifact in sync with content/docs/ proves
nothing about a body that has stopped reading it, and that failure would be silent.
Both directions matter, and only one had ever been noticed (#4851):
- listed but missing — the 10
content/docs/protocol/objectos/**paths left behind by the rename toprotocol/kernel/, plus 6 others. An audit agent pointed at a non-existent file reads nothing and reportsfixCount: 0, which in the run summary is indistinguishable from a doc that was checked and found accurate. That is how the accuracy defects in #4781 and #4817 sat inprotocol/kernel/for ~2 months while full audits reported green. - exists but unlisted — 48 docs, including all of
protocol/kernel/**and the wholecapabilities/directory. A run loggingFULL audit (no args.docs given)was auditing 130 of 178 docs.
The workflow additionally preflights its resolved scope — including a caller-supplied
args.docs, which no CI gate can see — and aborts naming any path that does not exist;
and each audit agent reports docExists from the read path itself, so a preflight that
was wrong cannot be laundered into a green summary. The gate covers the default list,
the preflight covers the caller's list, and the read path checks both.
The derived scope contains content/docs/releases/** (9 pages), and AGENTS.md's
Documentation Guardrails forbid a code PR from editing those pages at all. Since the
audit's deliverable is an in-place mdx rewrite, a full audit used to walk straight into
that prohibition — and open exactly the PR the guardrail exists to stop.
They are not excluded. Excluding them would leave some of the most-read pages in the
docs permanently unaudited, and would put a second definition of "docs this workflow
covers" next to the generated block — #4851 is the bill for one subject with two
hand-kept lists. Instead the deliverable forks, on a path prefix (content/docs/ releases/, which is the guardrail's own path column, decidable inside the workflow VM):
| editable docs | release-owned pages | |
|---|---|---|
| prompt | audit + fix in place | review, never edit |
| output schema | fixesApplied / fixCount |
findings[] + filesEdited |
| adversarial verifier | yes — re-checks applied edits | n/a, nothing was applied |
| deliverable | the diff | findings → file as issues |
Each finding carries kind (never-true / no-longer-true / ambiguous — a release
page is a historical record, so "the current API differs" is not automatically an
error), where on the page it is, what it should say instead, and file:line evidence.
The run summary reports them under releaseOwnedReadOnly and logs
releases (read-only): N finding(s) — file issues, do not edit.
Three failure modes are made loud rather than silent, because "audited nothing" and "audited, found nothing" must never look alike:
- a release page whose review returns no result fails the run by name — that is the exclusion option arrived at by accident;
- a review agent reporting
filesEdited: truefails the run naming the file to revert; - the read-only headline is logged whenever release pages are in scope, including at zero findings (reviewed-and-clean is a result, absence is not).
pnpm check:docs-audit-scope enforces the whole contract: AGENTS.md must still mark
that exact path RELEASE-OWNED, the workflow's RELEASE_OWNED_PREFIX must still match
that row, the scope must still contain release pages, and the fork must still work —
checked by running the workflow against stub agents and inspecting which prompt and
schema each doc gets, not by grepping for a keyword. --self-test then mutates the fork
out of an in-memory copy and requires that check to go red.
On any PR that touches packages/**, runs affected-docs.mjs against the base branch
and posts/updates a single advisory PR comment listing the docs that name something the
change touched — each row carrying the anchor that put it there, so a wrong row is
reportable rather than merely annoying. Never fails the build — it only flags drift at
the source, before it lands on main. Reviewers (or an on-demand audit run) decide whether
to re-verify.
The comment also carries a collapsed "What this run could not see" section:
anchorless files, cross-cutting symbols, over-broad anchors, the coarse package-mention
count, the sdk bridge's reach over the client-bound ledger rows (#9572), and the
rule-carrying pairing no run can reach (#11434, below). That is the point-of-use half of
#9192 — every one of the three derived-list failures in that shift was caught only because
a dev widened the probe past what the tool offered, never because the tool signalled its
own limits where it was read.
Every other entry in that fold is a report about the run — a count it produced, a set it derived. One is not, and renders on every run:
a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run.
A page can document a rule in the rule's own vocabulary — the values it maps from —
while the code carrying the rule is named for what it does. The two share no token, so
precision-first anchoring has nothing to join them with. This is not a tuning miss with a
threshold behind it: the absence of a shared identifier is the defect, and an absence
anchors nothing, so no per-run signal exists to compute. It is reported because it cannot
be detected — the same posture as the sdk bridge line above.
Measured on the specimen (#11434, from PR #11430 / card #11374), re-run through today's mapper:
| the diff | packages/drivers/driver-sql/src/sql-driver.ts — createColumn, the text family's column mapping |
| derived | 9 anchors (SqlDriver, createColumn, keyableTextLength, …) |
| listed | 7 pages, every one via a SqlDriver-family symbol it merely mentions |
| not listed | content/docs/protocol/objectql/types.mdx — the page that diff falsified, in four places |
| coarse fallback | misses it too: the page never mentions the changed package |
| existing limit lines | all silent — anchorlessChanges, crossCuttingSymbols, overbroadAnchors, weakAnchorsDropped all empty |
The last row is the reason this is stated rather than left to the reader: with every limit
line silent, the run reads as narrow, not blind. Positive control on the same page and
the same tree — a change to FieldSchema, which the page does name, lists it via
FieldSchema (symbol). The page is reachable; that pairing is not.
⛔ This states the gap; it does not close it. Buying real coverage would take a curated
code↔page ledger, a per-type anchor vocabulary, or authored enumeration marks — the open
question escalated as #11817, whose ruling also owns the population question (how wide
this class is is deliberately not measured here; the sentence's truth does not depend
on the count). check-drift-comment.mjs pins the line byte-exact on every fixture
case, which is the only meaningful pin for a statement whose contract is that it is always
true — making it conditional, moving it to the headline, or dropping it all go red.
A README carries no @docs-rule block, exports no symbol, mounts no route and declares
no SDK method, so a README-only diff derives nothing — and the comment answered it
with "this run has no opinion about the docs", which a reviewer reads as "nothing to
check". The file it could not anchor was named, honestly, in the collapsed section
above; GitHub renders that section shut. Two real README defects landed inside that gap
on one day: #11180 (packages/cli/README.md advertising os studio, a command the CLI
does not ship) and #11262 (packages/console/README.md asserting a @object-ui/console
fallback the code no longer performs). Neither was detectable by this check on any run.
So whenever anchorlessChanges is non-empty the headline itself now names the count and
the files, says those pages are not covered by this run, and withholds the ✅ —
a green tick is the clean-bill glyph and a partial look is not a clean bill. Anchor
derivation is untouched: with nothing unanchored, every headline is byte-identical to
what it was, and no run's verdict moves either way (this check never fails a build).
check-drift-comment.mjs pins it in both directions — a fixture diff touching only a
README must produce the not-covered wording, and one touching an anchorable source file
must not. It runs the workflow's own inline github-script block against real
affected-docs.mjs output from throwaway git repos, because a source grep passes just as
happily on text that never renders and on text that renders unconditionally, and neither
is a report. Zero dependencies, like the mapper: this job never runs pnpm install.
Same ruling as 1b, one level
down. The comment used to list content/docs/releases/v17.mdx in the same bulleted list
as editable pages — so a reader treating the advisory as a worklist was being pointed at
the one edit AGENTS.md forbids outright. The specimen that made it concrete: PR #6921
changed two diagnostic strings in packages/lint and got back three rows, one of them
that release page.
They are not filtered out. docs in --json stays the full set (it is what scopes
the audit, and #4920 rejected excluding these pages for good reasons); releaseOwnedDocs
is a partition of it — releaseOwnedDocs ⊆ docs, always — and the comment renders it
under its own ⛔ heading telling the reader to file an issue instead of editing.
affected-docs.mjs therefore holds a third literal copy of RELEASE_OWNED_PREFIX,
alongside AGENTS.md's guardrail row and the audit workflow's own const. Copies, because
the workflow is evaluated in a sandbox VM that cannot import and a shared module would
leave it the only unanchored one. check-audit-scope.mjs iterates
RELEASE_OWNED_CONSUMERS and fails if any copy stops matching the guardrail row — add
a consumer, add it to that list.
A Claude Code multi-agent workflow (.claude/workflows/docs-accuracy-audit.js). For each
doc: an agent reads it, locates the real implementation, and applies evidence-backed
fixes in place; a second adversarial verifier re-checks every fix against the code and
repairs over-corrections. Scope it with args.docs; for a full audit hand in the whole
set as args.handwritten.
// scoped to the docs a code change touched:
Workflow({ name: 'docs-accuracy-audit', args: { docs: [/* output of affected-docs.mjs */] } })
// full audit of all hand-written docs — read the artifact first, OUTSIDE the sandbox:
// node -e "console.log(JSON.stringify(require('./scripts/docs-audit/handwritten-docs.json').docs))"
Workflow({ name: 'docs-accuracy-audit', args: { handwritten: [/* that array */] } })⛔ There is no "omit args and audit everything" invocation, and there cannot be: the
body has no filesystem, so with nothing handed in it does not know what "everything" is.
It refuses by name rather than inventing a scope — the two shapes it could invent are a
silent audit of nothing and a stale list, and both report success.
It edits files in place (frontmatter preserved, no moves) and returns a per-doc log of
fixes, verifier repairs, and residual items that couldn't be confirmed against code —
except for content/docs/releases/**, which is reviewed read-only and returns
findings to file as issues (see 1b).
Always follow a run with the docs build gate:
pnpm --filter @objectstack/docs build # must compile all pages cleanMeasured 2026-08-18: no live schedule runs this audit. This section previously described the backstop in the present tense; it does not exist, so it cannot be cited as coverage for what the part-1 scope leaves out. Recorded rather than fixed on the spot: standing up a periodic LLM audit spends real budget on a cadence, which is the maintainer's call.
The intended design, for whoever stands it up: a cron routine on a cadence (default
monthly / per-release) catches drift the CI gate missed — it runs the
docs-accuracy-audit workflow, runs the build, and opens a PR when there are fixes.
Two independent defects in the old text, both worth keeping in view:
-
It never ran. Checked four ways, all negative — repeat these rather than re-deriving them:
- GitHub Actions — no workflow runs this audit at all. Of the 31 workflows
registered on the repo, the 10 carrying a
schedule:trigger (codeql,coverage-nightly,engine-split-metric,prerelease-pin-watch,publish-smoke,rerun-safety-nightly,scaffold-e2e,showcase-smoke,stale,validate-deps) are all unrelated. Derive that list withgrep -rl '^\s*schedule:' .github/workflows/rather than copying it: an unanchored grep forschedule:also matchescut-rc.yml, whose only hit is a comment saying it deliberately has no schedule.⚠️ Read the registered workflow list and its run history, not the YAML on disk. The two disagree in both directions: a workflow can be registered in Actions with no file onmain(matrix-aggregate-experiment.ymlis, today), and a scheduled workflow that GitHub auto-disabled after 60 days of repo inactivity leaves its file byte-identical. "The file is there" cannot answer "is it running" — the same read-a-conclusion-off-a-field-that-cannot-carry-it failure this docs-audit subsystem keeps paying for. - Routines — the Claude Code Remote
list_triggerstool lists every Routine the agent-seat account owns, and this is the surface that answers the question (it was once written off as unreadable by any agent; it is not). The only cron Routine is the hourly triage seat,18 * * * *. There is no docs-audit Routine, enabled or disabled. A cron Routine is never hidden by theinclude_completedfilter, so the absence is real rather than a listing artifact. - The creation mechanism this section named is gone — there is no
scheduleskill in.claude/skills/. - No trace of a periodic run. Repo history holds exactly three docs-accuracy audit PRs (#3243, #4219, #4312), every one hand-initiated against a named issue family, the most recent 2026-07-31. Nothing on a cadence.
- GitHub Actions — no workflow runs this audit at all. Of the 31 workflows
registered on the repo, the 10 carrying a
-
A change-scoped run is not a backstop. The old text had the routine compute "the change-scoped doc list since the last audit", while the cost note below calls part 4 the "periodic full backstop". Those are two different runs and only the second is a backstop — a change-scoped list is derived by the very anchor heuristic whose misses the backstop exists to catch, so scoping it that way re-inherits the blind spot it is meant to cover. A backstop has to run
--all: all 178 hand-written docs (runcheck-audit-scope.mjsfor today's number rather than trusting this one). -
And the obvious cheap substitute is not a backstop either. When the backstop is missing, the tempting one-line fix is to widen the audit's scope back to the coarse
packageMentionDocsset part 1 still emits. Measured across the 8 most recentpackages/**-touching commits onmain: it is wider (21 pages vs 5 ona4331227b) but it is not a superset — in every one of the 8, between 3 and 7 pages present in the precisedocsset are absent frompackageMentionDocs(api/error-catalog.mdx,automation/approvals.mdx,api/plugin-endpoints.mdx,data-modeling/relationships.mdxamong them). That is #9192's finding restated: the two sets miss in different directions, because the coarse one is a dependency-graph proxy and pages documenting a change through the SDK surface never name the implementing package. Swapping to it trades one incomplete set for another and loses pages the precise scope gets right. Only--allis a backstop.
Cost note: a full audit is ~2 agents per doc — measured at ~2.8M output tokens /
~160 agents when the scope was 128 docs, and the hand-written set is 178 today (run
check-audit-scope.mjs for the current number; don't trust a count written down here).
Always prefer the change-scoped list (affected-docs.mjs) over --all; reach for --all
only for a deliberate full audit. (This sentence used to name the periodic full backstop as
the exception — see part 4: that backstop is not running.)