Skip to content

Latest commit

 

History

History
31 lines (20 loc) · 2.63 KB

DOCS_TEST_SUITE.md

File metadata and controls

31 lines (20 loc) · 2.63 KB

Docs regression tests

Run the executable suite after building the site:

bun run build
bun run test:docs

CI runs test:docs after the production build. It uses Node's test runner and exits nonzero on a failed assertion. The experience checks read the built Markdown in dist/, so a passing source edit alone cannot satisfy them. Existing unit tests for the docs corpus, LLM index, and composed-site audit run in the same command.

Reader-facing contracts

scripts/docs-experience.test.mjs checks that:

  • Authored pages open with a subject or action instead of the repeated “After this page…” promise.
  • The homepage and Quickstart offer dashboard Drop before CLI installation, and the linked Publishing page describes the Drop flow.
  • The homepage links to a glossary that defines its recurring product terms.
  • Database links the full Zero runtime declaration and names its required server file before teaching queries.
  • Troubleshooting gives readers a first diagnostic step before listing error codes.
  • The opening paragraphs on Versions, Crons, Environment variables, Access, Caching, and Traffic stats answer specific reader questions. Each test names the question and the evidence expected near the top of the built page.

These are regression contracts for the changed entry paths and first-screen explanations. They make a future edit fail if it hides those answers again. They do not prove product behavior, grade tone, prove that shorter text is clearer, or measure whether a person can complete a real task. Check each expected answer against the public contract and the page's exceptions before encoding it in a test.

The October 6 review corrected assertions that reinforced overbroad access and traffic claims. The checks now distinguish views from requests and unique visitors, retained ready versions from other versions, and removal of one public grant from removal of all access. Cache refresh is a request, not an unconditional freshness guarantee.

Other gates

The CI build also runs verify:generated, command-example verification, type checking, strict link validation, the composed-site audit, public-safety verification, Vale, and route verification. The existing evals.yaml asks an agent factual-retrieval questions from the built docs; it is a separate evaluation and is not part of test:docs.

Known content gaps remain documented in the point-in-time QA comparison: no confirmed support channel and conflicting API-key preset guidance. That comparison records manual observations, not test-suite results. The writing audit records before/after editorial signals and its limits.