Latest nightly report: https://cinderloop.github.io/playwright-framework-template/
A production-grade Playwright + TypeScript test automation framework. It is demonstrated against Practice Software Testing (Toolshop) and built to be reused as a GitHub template for other applications.
- Lint-enforced core/app boundary.
src/coreis reusable and knows nothing about Toolshop; ESLint and a separate tsconfig keep it that way. - Fixtures, not base classes. Page objects, API clients, test data and sessions are injected
with
test.extendandmergeTests. - Typed configuration. It is selected by
TEST_ENVand validated with Zod, so a bad value stops the run before a browser opens. - API testing. A typed
BaseApiClientparses every response through a Zod schema, andtoMatchSchemaasserts response contracts. Failed tests get a redacted log of their API calls. - Advanced API layer (opt-in). A data-driven authorization matrix, security-adjacent tests, OpenAPI drift checks, property-based tests, concurrency and async flows. See docs/api-testing.md.
- Fast, self-cleaning setup. Tests sign in through the API with a fresh token, and per-test data is created through the API and always deleted.
- Test layers: UI, API, hybrid e2e (including a full checkout), network mocking and HAR replay, the Clock API, axe accessibility scans with a rule baseline, ARIA snapshots, and visual regression in Docker.
- Browsers: Chromium, Firefox, WebKit and Pixel 7.
- CI on GitHub Actions:
- PR checks: lint, format, typecheck, the boundary check, unit tests, the Docker image tag and
@smoke. - Nightly: the full suite in 4 shards, plus jobs for the advanced API suites and Docker visual tests, with a merged report on GitHub Pages.
- PR checks: lint, format, typecheck, the boundary check, unit tests, the Docker image tag and
- Flaky-test handling. CI retries a failing test with a trace,
@quarantinekeeps known flaky tests out of blocking runs, and a stability summary is added to every job. - Optional extras: Allure reports, Renovate dependency updates, and the Playwright test agents (planner, generator, healer) for Claude Code.
Requirements: Node.js 22+ (24 LTS recommended, see .nvmrc) and Docker.
npm ci
npx playwright install --with-deps
cp .env.example .env # fill in the accounts below
npm run toolshop:start # Toolshop in Docker: UI :4200, API :8091, seeded
npm testThe local stack uses Toolshop's seeded
demo accounts. Put a
customer and the admin account in .env.
In Docker (no local browsers needed, and the way to run WebKit on hosts where it is unsupported):
npm run toolshop:start
npm run test:docker # builds the Playwright image and runs the suite over host networkingAgainst the hosted demo: TEST_ENV=public npm run test:smoke. That environment is shared, so
keep runs small. Demo accounts there are sometimes locked by other users; if so, point
CUSTOMER_EMAIL at another seeded account.
src/core/ reusable framework code, knows nothing about Toolshop (lint-enforced)
api/ BaseApiClient: typed requests on APIRequestContext, bearer token, Zod parsing;
ApiRecorder and redaction
authorization/ data-driven authorization matrix: rows, expansion and the test runner
contract/ compares live responses with an OpenAPI spec
matchers/ expect.toMatchSchema(schema), expect.toNotExposeFields(fields)
a11y/ axe-core scan with a known-violation baseline
reporting/ flaky/quarantine summary reporter
fixtures/ base test: uniqueId, cleanup, apiRecorder
utils/ pollUntil, checkProperty, markKnownIssue, localOnly, cleanup queue, ...
config/ data/
src/app/ Toolshop-specific code
api/ one client per resource (auth, products, categories, brands, carts, invoices, ...)
schemas/ Zod response schemas; types are inferred from them
data/ Faker builders and TestData (create via API, delete after the test)
authorization/ the Toolshop authorization matrix and its actors
contract/ the operations the drift check samples, and the reviewed spec hash
pages/ components/ fixtures/ config/ auth/ a11y/
tests/
setup/ health check
api/ API tests (no browser)
api/advanced/ opt-in, local-only: authorization, security, properties, concurrency, async
contract/ OpenAPI drift check
ui/ UI tests, including network mocking, HAR replay and the Clock API
e2e/ hybrid tests: arrange via API, act and verify in the UI, clean up via API
a11y/ axe scans against a rule baseline, ARIA snapshots
visual/ screenshot baselines (Playwright Docker image only)
agents/ seed test for the Playwright test agents
specs/ test plans written by the planner agent
docs/ architecture, API testing, findings, adapting guide, flaky tests, ADRs
src/core may never import from src/app or tests. Its pure functions have Vitest unit tests
beside them (*.unit.test.ts). See
docs/architecture.md for how the layers, fixtures, auth, data and CI fit
together.
| Layer | Project(s) | Tests | What it covers |
|---|---|---|---|
tests/api |
api (no browser) |
30 | Auth, products, categories, brands, cart, invoices: happy paths, 422/409 validation, 401/403, 404s, pagination |
tests/api/advanced |
api (no browser), local only |
199 | Authorization matrix (124 rows), mass assignment, sensitive data, tokens, account security, rate limiting, error format, property-based, concurrency, idempotency, async flows |
tests/contract |
contract (no browser), local only |
2 | OpenAPI spec fingerprint; 39 live responses compared with the spec and the framework's schemas |
src/core/**/*.unit.test.ts |
Vitest | 150 | Pure core logic: redaction, recording, matrix expansion, schema comparison |
tests/ui |
every browser project | 23 | Browsing, search, filters, cart, login, role access, contact form; mocked API failures, HAR replay, Clock API |
tests/e2e |
every browser project | 4 | API-created data verified in the UI, including a full checkout by a newly registered customer |
tests/a11y |
a11y (Chromium) |
10 | axe-core WCAG 2.1 AA scans of six pages against a rule baseline; ARIA snapshots of key components |
tests/visual |
visual (Docker only) |
4 | Screenshot comparison of stable components |
UI and e2e specs run on Chromium, Firefox, WebKit and Pixel 7. Counts are per browser. The HAR
re-record test runs only with npm run test:har:update. In tests/api/advanced, tests with a
known-issue annotation assert the correct behaviour and are expected to fail until Toolshop
changes; docs/api-findings.md lists what they found.
Test data. Tests create their own data through testData, using Faker-built payloads that are
unique per test. The data is deleted through the API when the test ends, even when it fails.
Toolshop cannot delete some data:
- orders, and therefore any customer or product an order references;
- contact messages.
That data is kept and recorded on the test as a retained-data annotation. CI starts every run
from a freshly seeded database, and npm run toolshop:seed resets a local one.
Network mocking (tests/ui/network). page.route replaces Toolshop API responses to reach
states a healthy backend will not produce on demand:
- an empty catalog;
- a 500 from the product API and from add-to-cart;
- a slow product API.
Slow responses are simulated by holding the request until the test has checked the loading state
(holdRequests in src/core/utils), not with timers. One scenario replays the product API from a
recorded HAR. Re-record it against a freshly seeded local stack with npm run test:har:update.
Accessibility (tests/a11y). Each key page is scanned with axe-core for WCAG 2.1 A/AA.
src/app/a11y/baseline.ts lists known failures per page, each with a
reason:
- a scan fails on any rule not in the baseline;
- baseline rules that stopped failing are annotated, so the list only shrinks;
- the full axe output is attached to every test.
ARIA snapshots (toMatchAriaSnapshot) pin the accessibility tree of the main menu, sort control,
login form, checkout steps and cart table.
Visual regression (tests/visual). toHaveScreenshot covers the login form, the guest
header, a product detail page and the cart table, all from seeded data.
- Floating widgets and the footer build stamp are hidden by
tests/visual/screenshot.css, and related products are masked. - Baselines are generated and compared only inside the official Playwright image, so rendering is identical on every machine.
- That image is the
Dockerfile, pinned to the@playwright/testversion and checked bynpm run check:docker-imagein CI. - Outside the image, the visual tests are skipped. Nightly CI runs them in the image.
- ADR 0004 explains why.
To update baselines after an intended UI change:
- Start and seed Toolshop:
npm run toolshop:start. - Run
npm run test:visual:update. It builds the image and writes new PNGs totests/visual/__screenshots__/. - Review the image diffs in the pull request, then commit them with the change.
- When upgrading Playwright, update the
Dockerfiletag in the same PR and regenerate all baselines. Renovate groups the two updates into one PR.
Clock API (tests/ui/checkout/card-expiry.spec.ts). The payment step uses the browser clock
to reject card expiry dates before the current month.
page.clockpins the date to 15 March 2031. February is rejected and March accepted.- The clock then moves into April, and the same card is rejected as expired.
- The test runs as a guest. With the clock years ahead, a signed-in session's JWT would look expired to the app.
Specs are tagged with the tag option: @smoke, @regression, @api, @api-advanced,
@contract, @a11y, @visual and @quarantine. The health check is untagged and always runs first, as a project dependency.
| Script | Purpose |
|---|---|
npm test |
Everything, all projects |
npm run test:smoke |
@smoke tests (what PRs run), excluding @quarantine |
npm run test:regression |
@regression UI and e2e tests |
npm run test:api / test:a11y |
API tests, without the advanced suites / accessibility tests |
npm run test:api:advanced |
Advanced API suites (local only; includes a five-minute token test) |
npm run test:unit |
Vitest unit tests for src/core |
npm run check:openapi |
Compare live responses with the OpenAPI spec and print the drift report |
npm run check:openapi:update |
Record the current spec as reviewed |
npm run test:quarantine |
Only @quarantine tests |
npm run test:ui / test:headed |
Playwright UI mode / headed run |
npm run test:docker |
Whole suite inside the official Playwright image (incl. WebKit, visual) |
npm run test:visual |
Visual tests in the Playwright image |
npm run test:visual:update |
Regenerate visual baselines in the Playwright image |
npm run test:har:update |
Re-record the product API HAR (local Toolshop) |
npm run test:allure |
Run with Allure results enabled (ALLURE=1) |
npm run report |
Open the last HTML report |
npm run allure:generate / allure:open |
Build / open the Allure report from allure-results/ |
npm run lint / lint:fix |
ESLint (zero warnings) / ESLint with autofix |
npm run lint:boundary |
ESLint on src/core only (the core/app boundary) |
npm run typecheck / typecheck:core |
tsc for the whole repo / for src/core alone |
npm run format / format:check |
Prettier |
npm run check:docker-image |
Dockerfile image tag matches @playwright/test |
npm run docker:build |
Build the test image (toolshop-tests) without running it |
npm run toolshop:start |
Start Toolshop and seed it (up + seed) |
npm run toolshop:seed |
Reset Toolshop data, generated PDFs and the cache |
npm run toolshop:down |
Stop Toolshop and delete its volumes |
Pass extra Playwright options after --, e.g. npm run test:smoke -- --project=firefox.
- HTML report: written on every run (
npm run report). The nightly merges every shard and the Docker visual job into one report and publishes it to GitHub Pages. - Test stability summary: written after every run to the console and, in CI, to the job summary. It lists tests that only passed on retry and every quarantined test. See docs/flaky-tests.md for how flakiness is tracked.
- Allure (optional):
npm run test:allure, thennpm run allure:generateandnpm run allure:open. It is off by default, so normal runs write only the HTML report. It uses Allure 3, which needs no Java.
| Workflow | Trigger | What it does |
|---|---|---|
| PR checks | pull request, push to main |
Lint, format, typecheck, core/app boundary, unit tests, Docker image tag; @smoke (minus @quarantine) on API, Chromium and Pixel 7 against Toolshop in Docker Compose |
| Nightly | 02:00 UTC daily, manual | Full suite on every browser in 4 shards, advanced API suites and OpenAPI drift, visual tests in the Playwright image, quarantined tests (non-blocking); one merged report published to GitHub Pages |
What every workflow does:
- Toolshop: each test job starts it from
docker-compose.yml, waits for every service to be healthy and seeds the database. It runs the tests, then tears the stack down. - Artifacts: the HTML report is uploaded on every run, and
test-results/(traces, screenshots, videos) on failure. - Caching: npm packages and Playwright browsers are cached.
- Concurrency: a newer run on the same branch or PR cancels the older one.
- Pinning: third-party actions are pinned to commit SHAs.
The Playwright test agents (planner, generator, healer) are set up for Claude Code. With Toolshop
running, open Claude Code in this repository and ask for an agent by name, for example: "Use the
playwright-test-planner agent to plan tests for the registration page, using project 'agents' and
seed tests/agents/seed.spec.ts". Treat what they produce as a draft. The agents get app behaviour
wrong at times, and the generated code needs rewriting onto the page objects.
docs/ai-assisted-testing.md covers the setup, a worked example and
what review changed.
- Pre-commit hook: Husky runs ESLint and Prettier on staged files (lint-staged), then both
typechecks, so a commit cannot break lint or types. It is installed by
npm ci. - Pull requests: they use the template in
.github/PULL_REQUEST_TEMPLATE.mdand must pass the PR checks. Record user-facing changes in CHANGELOG.md (Keep a Changelog). - Issues: use the forms for framework bugs and flaky tests. Bugs in Toolshop itself belong in its repository.
- Commits: follow Conventional Commits
(
feat:,fix:,docs:,chore:).
They are documented in .env.example and validated with Zod at start-up. An invalid or missing value stops the run before any browser starts.
| Variable | Required | Description |
|---|---|---|
TEST_ENV |
no | local (default) or public |
BASE_URL, API_URL |
no | Override the per-environment UI/API URLs |
CUSTOMER_EMAIL, CUSTOMER_PASSWORD |
yes | Account used for the customer role |
ADMIN_EMAIL, ADMIN_PASSWORD |
yes | Account used for the admin role |
ALLURE |
no | Set to 1 to also write Allure results |
PROPERTY_SEED |
no | Repeat a property test run with this seed |
UPDATE_OPENAPI |
no | Set by check:openapi:update; records the spec |
PW_AGENTS |
no | Set by .mcp.json; adds the agents' seed project |
- Architecture: the core/app boundary, fixture composition, auth, data and CI flow.
- API testing: the API layer, each advanced technique and how to adapt them.
- API findings: what the advanced suites found in Toolshop.
- API reconnaissance: the endpoint inventory the advanced suites were planned from.
- Adapting to your app: use this as a template; add a page object, an API client, a role or an environment.
- Flaky tests: retries, quarantine and how flakiness is tracked.
- AI-assisted testing: the Playwright planner, generator and healer agents, what they produced, and what review changed.
- Decisions: ADRs for fixtures, API auth, Zod, Docker visual baselines, Toolshop in Compose, Renovate, the advanced API layer and signing in per test.
- Changelog