Skip to content

About

Production-grade Playwright + TypeScript framework template with API, UI, a11y and visual testing, Docker-based CI, and a reusable core. Demo app: Toolshop

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

24 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Playwright Toolshop Framework

PR checks Nightly License: MIT Playwright

Latest nightly report: https://cinderloop.github.io/playwright-framework-template/

A production-grade Playwright + TypeScript test automation framework. It is demonstrated against Practice Software Testing (Toolshop) and built to be reused as a GitHub template for other applications.

Features

  • Lint-enforced core/app boundary. src/core is reusable and knows nothing about Toolshop; ESLint and a separate tsconfig keep it that way.
  • Fixtures, not base classes. Page objects, API clients, test data and sessions are injected with test.extend and mergeTests.
  • Typed configuration. It is selected by TEST_ENV and validated with Zod, so a bad value stops the run before a browser opens.
  • API testing. A typed BaseApiClient parses every response through a Zod schema, and toMatchSchema asserts response contracts. Failed tests get a redacted log of their API calls.
  • Advanced API layer (opt-in). A data-driven authorization matrix, security-adjacent tests, OpenAPI drift checks, property-based tests, concurrency and async flows. See docs/api-testing.md.
  • Fast, self-cleaning setup. Tests sign in through the API with a fresh token, and per-test data is created through the API and always deleted.
  • Test layers: UI, API, hybrid e2e (including a full checkout), network mocking and HAR replay, the Clock API, axe accessibility scans with a rule baseline, ARIA snapshots, and visual regression in Docker.
  • Browsers: Chromium, Firefox, WebKit and Pixel 7.
  • CI on GitHub Actions:
    • PR checks: lint, format, typecheck, the boundary check, unit tests, the Docker image tag and @smoke.
    • Nightly: the full suite in 4 shards, plus jobs for the advanced API suites and Docker visual tests, with a merged report on GitHub Pages.
  • Flaky-test handling. CI retries a failing test with a trace, @quarantine keeps known flaky tests out of blocking runs, and a stability summary is added to every job.
  • Optional extras: Allure reports, Renovate dependency updates, and the Playwright test agents (planner, generator, healer) for Claude Code.

Quick start

Requirements: Node.js 22+ (24 LTS recommended, see .nvmrc) and Docker.

npm ci
npx playwright install --with-deps
cp .env.example .env          # fill in the accounts below
npm run toolshop:start        # Toolshop in Docker: UI :4200, API :8091, seeded
npm test

The local stack uses Toolshop's seeded demo accounts. Put a customer and the admin account in .env.

In Docker (no local browsers needed, and the way to run WebKit on hosts where it is unsupported):

npm run toolshop:start
npm run test:docker           # builds the Playwright image and runs the suite over host networking

Against the hosted demo: TEST_ENV=public npm run test:smoke. That environment is shared, so keep runs small. Demo accounts there are sometimes locked by other users; if so, point CUSTOMER_EMAIL at another seeded account.

Project structure

src/core/        reusable framework code, knows nothing about Toolshop (lint-enforced)
  api/           BaseApiClient: typed requests on APIRequestContext, bearer token, Zod parsing;
                 ApiRecorder and redaction
  authorization/ data-driven authorization matrix: rows, expansion and the test runner
  contract/      compares live responses with an OpenAPI spec
  matchers/      expect.toMatchSchema(schema), expect.toNotExposeFields(fields)
  a11y/          axe-core scan with a known-violation baseline
  reporting/     flaky/quarantine summary reporter
  fixtures/      base test: uniqueId, cleanup, apiRecorder
  utils/         pollUntil, checkProperty, markKnownIssue, localOnly, cleanup queue, ...
  config/ data/
src/app/         Toolshop-specific code
  api/           one client per resource (auth, products, categories, brands, carts, invoices, ...)
  schemas/       Zod response schemas; types are inferred from them
  data/          Faker builders and TestData (create via API, delete after the test)
  authorization/ the Toolshop authorization matrix and its actors
  contract/      the operations the drift check samples, and the reviewed spec hash
  pages/ components/ fixtures/ config/ auth/ a11y/
tests/
  setup/         health check
  api/           API tests (no browser)
  api/advanced/  opt-in, local-only: authorization, security, properties, concurrency, async
  contract/      OpenAPI drift check
  ui/            UI tests, including network mocking, HAR replay and the Clock API
  e2e/           hybrid tests: arrange via API, act and verify in the UI, clean up via API
  a11y/          axe scans against a rule baseline, ARIA snapshots
  visual/        screenshot baselines (Playwright Docker image only)
  agents/        seed test for the Playwright test agents
specs/           test plans written by the planner agent
docs/            architecture, API testing, findings, adapting guide, flaky tests, ADRs

src/core may never import from src/app or tests. Its pure functions have Vitest unit tests beside them (*.unit.test.ts). See docs/architecture.md for how the layers, fixtures, auth, data and CI fit together.

Test layers

Layer Project(s) Tests What it covers
tests/api api (no browser) 30 Auth, products, categories, brands, cart, invoices: happy paths, 422/409 validation, 401/403, 404s, pagination
tests/api/advanced api (no browser), local only 199 Authorization matrix (124 rows), mass assignment, sensitive data, tokens, account security, rate limiting, error format, property-based, concurrency, idempotency, async flows
tests/contract contract (no browser), local only 2 OpenAPI spec fingerprint; 39 live responses compared with the spec and the framework's schemas
src/core/**/*.unit.test.ts Vitest 150 Pure core logic: redaction, recording, matrix expansion, schema comparison
tests/ui every browser project 23 Browsing, search, filters, cart, login, role access, contact form; mocked API failures, HAR replay, Clock API
tests/e2e every browser project 4 API-created data verified in the UI, including a full checkout by a newly registered customer
tests/a11y a11y (Chromium) 10 axe-core WCAG 2.1 AA scans of six pages against a rule baseline; ARIA snapshots of key components
tests/visual visual (Docker only) 4 Screenshot comparison of stable components

UI and e2e specs run on Chromium, Firefox, WebKit and Pixel 7. Counts are per browser. The HAR re-record test runs only with npm run test:har:update. In tests/api/advanced, tests with a known-issue annotation assert the correct behaviour and are expected to fail until Toolshop changes; docs/api-findings.md lists what they found.

Test data. Tests create their own data through testData, using Faker-built payloads that are unique per test. The data is deleted through the API when the test ends, even when it fails. Toolshop cannot delete some data:

  • orders, and therefore any customer or product an order references;
  • contact messages.

That data is kept and recorded on the test as a retained-data annotation. CI starts every run from a freshly seeded database, and npm run toolshop:seed resets a local one.

Advanced coverage

Network mocking (tests/ui/network). page.route replaces Toolshop API responses to reach states a healthy backend will not produce on demand:

  • an empty catalog;
  • a 500 from the product API and from add-to-cart;
  • a slow product API.

Slow responses are simulated by holding the request until the test has checked the loading state (holdRequests in src/core/utils), not with timers. One scenario replays the product API from a recorded HAR. Re-record it against a freshly seeded local stack with npm run test:har:update.

Accessibility (tests/a11y). Each key page is scanned with axe-core for WCAG 2.1 A/AA. src/app/a11y/baseline.ts lists known failures per page, each with a reason:

  • a scan fails on any rule not in the baseline;
  • baseline rules that stopped failing are annotated, so the list only shrinks;
  • the full axe output is attached to every test.

ARIA snapshots (toMatchAriaSnapshot) pin the accessibility tree of the main menu, sort control, login form, checkout steps and cart table.

Visual regression (tests/visual). toHaveScreenshot covers the login form, the guest header, a product detail page and the cart table, all from seeded data.

  • Floating widgets and the footer build stamp are hidden by tests/visual/screenshot.css, and related products are masked.
  • Baselines are generated and compared only inside the official Playwright image, so rendering is identical on every machine.
  • That image is the Dockerfile, pinned to the @playwright/test version and checked by npm run check:docker-image in CI.
  • Outside the image, the visual tests are skipped. Nightly CI runs them in the image.
  • ADR 0004 explains why.

To update baselines after an intended UI change:

  1. Start and seed Toolshop: npm run toolshop:start.
  2. Run npm run test:visual:update. It builds the image and writes new PNGs to tests/visual/__screenshots__/.
  3. Review the image diffs in the pull request, then commit them with the change.
  4. When upgrading Playwright, update the Dockerfile tag in the same PR and regenerate all baselines. Renovate groups the two updates into one PR.

Clock API (tests/ui/checkout/card-expiry.spec.ts). The payment step uses the browser clock to reject card expiry dates before the current month.

  • page.clock pins the date to 15 March 2031. February is rejected and March accepted.
  • The clock then moves into April, and the same card is rejected as expired.
  • The test runs as a guest. With the clock years ahead, a signed-in session's JWT would look expired to the app.

Running subsets

Specs are tagged with the tag option: @smoke, @regression, @api, @api-advanced, @contract, @a11y, @visual and @quarantine. The health check is untagged and always runs first, as a project dependency.

Script Purpose
npm test Everything, all projects
npm run test:smoke @smoke tests (what PRs run), excluding @quarantine
npm run test:regression @regression UI and e2e tests
npm run test:api / test:a11y API tests, without the advanced suites / accessibility tests
npm run test:api:advanced Advanced API suites (local only; includes a five-minute token test)
npm run test:unit Vitest unit tests for src/core
npm run check:openapi Compare live responses with the OpenAPI spec and print the drift report
npm run check:openapi:update Record the current spec as reviewed
npm run test:quarantine Only @quarantine tests
npm run test:ui / test:headed Playwright UI mode / headed run
npm run test:docker Whole suite inside the official Playwright image (incl. WebKit, visual)
npm run test:visual Visual tests in the Playwright image
npm run test:visual:update Regenerate visual baselines in the Playwright image
npm run test:har:update Re-record the product API HAR (local Toolshop)
npm run test:allure Run with Allure results enabled (ALLURE=1)
npm run report Open the last HTML report
npm run allure:generate / allure:open Build / open the Allure report from allure-results/
npm run lint / lint:fix ESLint (zero warnings) / ESLint with autofix
npm run lint:boundary ESLint on src/core only (the core/app boundary)
npm run typecheck / typecheck:core tsc for the whole repo / for src/core alone
npm run format / format:check Prettier
npm run check:docker-image Dockerfile image tag matches @playwright/test
npm run docker:build Build the test image (toolshop-tests) without running it
npm run toolshop:start Start Toolshop and seed it (up + seed)
npm run toolshop:seed Reset Toolshop data, generated PDFs and the cache
npm run toolshop:down Stop Toolshop and delete its volumes

Pass extra Playwright options after --, e.g. npm run test:smoke -- --project=firefox.

Reporting

  • HTML report: written on every run (npm run report). The nightly merges every shard and the Docker visual job into one report and publishes it to GitHub Pages.
  • Test stability summary: written after every run to the console and, in CI, to the job summary. It lists tests that only passed on retry and every quarantined test. See docs/flaky-tests.md for how flakiness is tracked.
  • Allure (optional): npm run test:allure, then npm run allure:generate and npm run allure:open. It is off by default, so normal runs write only the HTML report. It uses Allure 3, which needs no Java.

Continuous integration

Workflow Trigger What it does
PR checks pull request, push to main Lint, format, typecheck, core/app boundary, unit tests, Docker image tag; @smoke (minus @quarantine) on API, Chromium and Pixel 7 against Toolshop in Docker Compose
Nightly 02:00 UTC daily, manual Full suite on every browser in 4 shards, advanced API suites and OpenAPI drift, visual tests in the Playwright image, quarantined tests (non-blocking); one merged report published to GitHub Pages

What every workflow does:

  • Toolshop: each test job starts it from docker-compose.yml, waits for every service to be healthy and seeds the database. It runs the tests, then tears the stack down.
  • Artifacts: the HTML report is uploaded on every run, and test-results/ (traces, screenshots, videos) on failure.
  • Caching: npm packages and Playwright browsers are cached.
  • Concurrency: a newer run on the same branch or PR cancels the older one.
  • Pinning: third-party actions are pinned to commit SHAs.

AI-assisted test authoring (optional)

The Playwright test agents (planner, generator, healer) are set up for Claude Code. With Toolshop running, open Claude Code in this repository and ask for an agent by name, for example: "Use the playwright-test-planner agent to plan tests for the registration page, using project 'agents' and seed tests/agents/seed.spec.ts". Treat what they produce as a draft. The agents get app behaviour wrong at times, and the generated code needs rewriting onto the page objects. docs/ai-assisted-testing.md covers the setup, a worked example and what review changed.

Contributing

  • Pre-commit hook: Husky runs ESLint and Prettier on staged files (lint-staged), then both typechecks, so a commit cannot break lint or types. It is installed by npm ci.
  • Pull requests: they use the template in .github/PULL_REQUEST_TEMPLATE.md and must pass the PR checks. Record user-facing changes in CHANGELOG.md (Keep a Changelog).
  • Issues: use the forms for framework bugs and flaky tests. Bugs in Toolshop itself belong in its repository.
  • Commits: follow Conventional Commits (feat:, fix:, docs:, chore:).

Environment variables

They are documented in .env.example and validated with Zod at start-up. An invalid or missing value stops the run before any browser starts.

Variable Required Description
TEST_ENV no local (default) or public
BASE_URL, API_URL no Override the per-environment UI/API URLs
CUSTOMER_EMAIL, CUSTOMER_PASSWORD yes Account used for the customer role
ADMIN_EMAIL, ADMIN_PASSWORD yes Account used for the admin role
ALLURE no Set to 1 to also write Allure results
PROPERTY_SEED no Repeat a property test run with this seed
UPDATE_OPENAPI no Set by check:openapi:update; records the spec
PW_AGENTS no Set by .mcp.json; adds the agents' seed project

Documentation

  • Architecture: the core/app boundary, fixture composition, auth, data and CI flow.
  • API testing: the API layer, each advanced technique and how to adapt them.
  • API findings: what the advanced suites found in Toolshop.
  • API reconnaissance: the endpoint inventory the advanced suites were planned from.
  • Adapting to your app: use this as a template; add a page object, an API client, a role or an environment.
  • Flaky tests: retries, quarantine and how flakiness is tracked.
  • AI-assisted testing: the Playwright planner, generator and healer agents, what they produced, and what review changed.
  • Decisions: ADRs for fixtures, API auth, Zod, Docker visual baselines, Toolshop in Compose, Renovate, the advanced API layer and signing in per test.
  • Changelog

License

MIT

About

Production-grade Playwright + TypeScript framework template with API, UI, a11y and visual testing, Docker-based CI, and a reusable core. Demo app: Toolshop

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages