Skip to content

chore: add verify-coder skill for local end-to-end checks - #31782

Open
ThomasK33 wants to merge 47 commits into
mainfrom
verification-gp0d
Open

ThomasK33 wants to merge 47 commits into
mainfrom
verification-gp0d

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Why

Agents that change Coder need a repeatable way to prove the change on the real app, not only in tests. This adds a repo skill, verify-coder, that starts an isolated local Coder from the checkout, checks that it is the instance the agent started, drives it through the web UI, the CLI and the API, and keeps the evidence after cleanup.

What changed

  • .agents/skills/verify-coder/scripts/ has launch.sh, a read-only doctor.sh, cleanup.sh (stops only what this launch started), lib.sh helpers and a selftest.sh that needs no instance.
  • launch.sh starts scripts/develop.sh with --port-offset and --bind-address 127.0.0.1 (feat: add a --bind-address flag to scripts/develop #31815) in its own process session, and reads the ports from develop's log. The API, Vite and metrics listen on 127.0.0.1 only: develop creates the owner through the unauthenticated /api/v2/users/first route, and Vite serves checkout files, .coderv2/session included. When Docker works, docker-forward.py carries only the API port from the Docker bridge gateway to 127.0.0.1, so the docker template's workspace agents connect through host.docker.internal. A new database gets a random admin password, never the published default.
  • Four journeys each drive one feature end to end and write screenshots, API and CLI read-backs and a result file: create a user, templates, a Docker workspace, and a Coder Agents chat against a local llm-mock.
  • features/ maps five user-facing features to their entry points and proofs. SKILL.md keeps the short path (launch, doctor, drive, cleanup), and reference.md holds the detail.
  • The dogfood skill keeps PR scoping and reporting and points to verify-coder for the run.
  • scripts/develop/main.go logs a resolved listen ports line before preflight. scripts/coder-dev.sh accepts CODER_DEV_URL to target a dev instance on other ports.

Scope

  • Local development only. It adds no CI job and changes no product code.
  • It maps five features: sign in, users, templates, workspaces and Coder Agents. Other features have no recipe yet.
  • Design rule for later changes: nothing but the authenticated API may listen on a non-loopback address. Review rounds 1 to 5 kept finding exposure through the listen address (the first-user route, then Vite's /@fs/ route).

Blast Radius

Most files are new under .agents/skills/. The two script changes are additive: develop prints one more log line, and coder-dev.sh reads CODER_DEV_URL only when it is set.

Verification

  • A fresh launch.sh --fresh passed doctor 12/12 on b10ae6e824, whose skill files equal this head's. The listeners were 127.0.0.1:3320, 127.0.0.1:8400 and 172.17.0.1:3320, the last being the API forward for Docker. Journeys create-user 9/9, templates 4/4, workspace 10/10 (Docker agent connected) and coder-agents 7/7 passed, and cleanup passed 7/7 and stopped the forwarder.
  • After develop was killed with SIGKILL during setup, launch.sh printed the admin password 0 times. Cleanup kept .coderv2 and named the Postgres pid that still ran.
  • selftest.sh passed 20/20. Each fix in this PR has a mutation that turns a selftest check red.
  • Verifier agents that did not write the code checked successive heads, with live runs and mutation tests. I fixed every gap they found.
  • shellcheck, shfmt and typos are clean on the changed files. I did not run the repo git hooks locally.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

…l Coder

Adds verify-coder, a project skill that launches an isolated Coder from the
checkout with scripts/develop.sh, checks it with a read-only doctor, drives
it through the web UI with agent-browser and through the coder CLI, and
keeps evidence under build/verify-coder/evidence after cleanup. It seeds a
feature map for sign-in, users, templates, workspaces and Coder Agents, and
ships journeys/create-user.sh as the first end-to-end journey.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
Adds a doctor check for a missing docker template, before and filtered
after screenshots in the create-user journey, a rule for hand-driven
evidence, and exact wait, jq and redaction steps in the feature files.
… with journeys

Adds journeys/workspace.sh (CLI and UI create, agent, SSH, web terminal,
stop, start, delete) and journeys/coder-agents.sh (llm-mock provider,
default model, chat reply), rewrites both feature files from the live
runs, stops recorded helpers in cleanup.sh, captures a screenshot on any
failed journey step, and waits out the post-sign-in hard reload.
Waits for the login form and the chat composer before driving them,
raises agent-browser's timeout to 90 s for loaded hosts, and saves page
errors, console and network requests for a failed journey step.
cleanup.sh deletes only workspaces created after the launch and keeps a
developer's older ones, --keep-data hands .coderv2 back to the developer,
launch.sh passes flags to develop.sh unchanged, the token and passwords stay
off command lines, journeys drive the embedded production frontend, and a
templates journey and a role check cover the recipes that never ran.
--trace enables OpenTelemetry tracing and adds no log lines.
lib.sh no longer changes the caller's shell options, vc_check restores the
errexit state, vc_api_status checks HTTP codes without exposing the token,
and the user, template, workspace and sign-in recipes keep credentials and
developer data safe.
Documents vc_check semantics and every lib.sh helper, names the workspaces
cleanup deletes, and spells out the template push, CLI stop and start, and
sign-in evidence steps.
Bounds the agent wait recipe, names the whoami command for a separate
config folder, logs cleanup's full output with workspace names, and adds a
CLI stop and start check to the workspace journey.
The coder-agents journey restores a developer's default model and deletes
its own provider, model and chat. Cleanup fails and names containers when
the server is already down, and keeps .coderv2 while another process runs
Postgres from it. launch.sh refuses --agpl. The dogfood skill points to
verify-coder for the scripted run.
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-09T17:42:55.439952Z 2081e79 Manual request
🔒 Security Review ✅ Completed 2026-11-09T17:46:49.171551Z 2081e79 Manual request

Security findings

Advisory findings (6)

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

CI's shellcheck reports SC2119 on vc_browser_login calls without
arguments. Releases 0.9 and 0.10 reproduce it, 0.11 does not.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c44d2d9640

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit: c44d2d9640

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Give a harness-created database a random admin password, keep the state
folder private, and fill the password through agent-browser stdin.
Quote run/env values, refuse CODER_BUILD_AGPL and --password, and record
the launch password for later journeys.

The users journey now creates a CLI user and deletes both users. The
templates journey stops early without Docker. The workspace journey
checks the CLI home volume. The coder-agents journey restores the model
on exit.
…g on a closed pipe

grep -q closed the pipe before curl finished writing the login page, so
pipefail failed the check in 26 of 100 runs.
…x fixes

Use the random admin password whenever the launch creates the database,
not only in a .coderv2 the harness created. The doctor now checks the
state folder mode and that the default password is refused. A new
selftest.sh checks the launch refusals and the stdin-only password fill.
create-user proves both users are gone and names the coder users create
exit code.
… the first-user log line

The check skipped whenever launch.sh chose the default password, so a
regression there passed. It now follows develop.log, which says when the
launch created the database. The doctor docs no longer call it read-only,
and a failed coder users delete names its output file.
… new database

launch.sh now decides the password before it builds and refuses the
published default for a database it creates, so the doctor needs no
exception for it. The selftest checks the refusal and that a new
database gets past the password step without CODER_DEV_ADMIN_PASSWORD.
@ThomasK33

Copy link
Copy Markdown
Member Author

Review-limit waiver: the PR owner asked to keep running the Codex review until it reports no issues on this PR. The six-round limit is waived for this PR.

The first Codex pair raised 10 findings on c44d2d9. All are fixed on c60c818, and an independent verifier passed that head. Each thread has its own reply.


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c60c818b9a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit: c60c818b9a

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

launch.sh now takes only --fresh and allowlisted server flags, and starts
develop without the caller's CODER_* variables, so no reset, database
URL, port or env file slips past it. The password file changes only
after develop creates the first user. coder.sh always targets the
recorded URL. cleanup.sh deletes only workspaces that a journey or
vc_record_workspace recorded, and matches the Postgres path literally.
…ministic

The stand-in zombie's child could exit before its parent became sleep, so
bash reaped it and no zombie existed. The child now outlives the exec, and
the check waits for the Z state. launch.sh saves the password file once
more after the ready wait, as before.
…op starts

The waits in launch.sh could read the log before the background develop
opened it, and sed then failed the launch. The deleted-binary selftest case
now removes the stand-in's binary only after it runs it.
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e0eb1741a9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .agents/skills/dogfood/SKILL.md Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit: e0eb1741a9

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…esign

- launch.sh removes a saved CLI session before setup, so develop signs in
  as the admin instead of reusing another user's token.
- vc_docker_since fails without a docker CLI while a daemon is configured.
- The workspace journey stops waiting for an agent once the workspace is
  missing or its build failed.
- sign-in.md says the member has the admin password only on data that
  develop created, and dogfood's migration steps keep the workspaces that
  must survive the version switch.
…hole run

After the restart on every interface, Vite's /@fs/ route served checkout
files, .coderv2/session included, to any host on the network. launch.sh now
runs develop once with --bind-address 127.0.0.1 and no restart.
…r workspaces

develop now listens on 127.0.0.1 only. When docker works, launch.sh runs
docker-forward.py on the Docker bridge gateway for the API port, so the
docker template's agents connect through host.docker.internal while Vite
and metrics stay on loopback. doctor checks the forward and that Vite does
not answer on the gateway, and cleanup stops the forwarder as a helper.
ethanndickson pushed a commit that referenced this pull request Nov 9, 2026
#31829)

## Why

`test-go-race-pg` fails on PRs that touch no Go product code, for
example two runs on #31782. Since #30124, two parallel tests in
`coderd/coderdtest` call swag's `Spec.ReadDoc`, which writes shared
state, so the race detector fails the whole package (`WARNING: DATA
RACE` in `swag.(*Spec).ReadDoc`, `swagger_test.go:41` and `:77`).

## What changed

`swagger_test.go` renders the spec once through `sync.OnceValue`, and
both tests read that string.

## Verification

- On `main`, `go test -race -count=20 -run
'TestSwaggerParametersValid|TestChatModelPathParametersFormatted'
./coderd/coderdtest/` reported 4 data races and failed.
- With this change, the same command passed with 0 races.
- `golangci-lint` reports 0 issues for `./coderd/coderdtest/` with both
the main and the new-code configs.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_
@ThomasK33
ThomasK33 temporarily deployed to main November 9, 2026 13:33 — with GitHub Actions Inactive
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2081e799b3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit: 2081e799b3

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…indings

- Loopback requests bypass any configured proxy (NO_PROXY), so the admin
  password and session token never reach it.
- create-user fills the new password with vc_fill_secret, on stdin.
- launch.sh keeps the caller's saved CLI session in the run state, and
  cleanup.sh puts it back when it keeps .coderv2.
- The Agents journey gives the default back only while its own model
  still has it.
- Journey evidence folders get a unique suffix, and the terminal check
  polls for the command's file.
- vc_record_workspace is for local Docker workspaces only.
- vc_wait_agent moves to lib.sh with a selftest, and doctor's port check
  reads the 127.0.0.1 listener, not the Docker forward.
- The docs name the API, Vite and metrics as loopback only: coder server's
  tailnet still opens WireGuard UDP sockets.
@ThomasK33
ThomasK33 temporarily deployed to main November 9, 2026 18:50 — with GitHub Actions Inactive
… fix

- selftest creates each stand-in log before its server starts, so a poll
  under load no longer reads a missing file.
- coder-agents restore keeps a newer default that another admin picked and
  passes in that case.
- lib.sh merges the caller's NO_PROXY and no_proxy instead of overwriting
  no_proxy.
- vc_fill_secret no longer sits inside the vc_browser_login comment.
- New selftest checks cover the restore guard, the web terminal poll, the
  journeys' mktemp evidence folders, the doctor's gateway requests under a
  proxy, how lib.sh sets NO_PROXY, and the restored session's file mode.

_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
@ThomasK33
ThomasK33 temporarily deployed to main November 10, 2026 08:44 — with GitHub Actions Inactive
…vioral

The journeys now take their evidence folder from a vc_new_evidence helper in
lib.sh. The selftest calls it twice with one stamp and checks that the
folders differ and have mode 755, instead of grepping each journey for its
mktemp line.

_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
- create-user waits for the new-user form before it reads the login type.
  The page sets its title first, and get text does not wait.
- vc_start_develop drops every ANTHROPIC_* and OPENAI_* variable, as the
  docs say.
- lib.sh splits NO_PROXY on newlines too, and keeps a lone "*".
- coder-agents restore accepts the server's pick of a default when no old
  default existed. The exit traps call restore and delete_users directly.
- The selftest runs loaded functions through with_errexit, so a failing
  step inside them fails the check, and it checks the session mode after
  launch.sh moves it. A new form-wait check covers the create-user fix.

_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
- coder-agents restore PATCHes the old default only while that model still
  exists. When it is gone, the server's pick of a new default passes.
- The restore selftest stand-in fails a PATCH of a missing model, as the API
  does, checks which cases send a PATCH, and covers a deleted old default.
- The doctor proxy check stops its stand-in server when the stand-in proxy
  does not start, and start_proxy waits up to 20 seconds.

_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
agent-browser 0.33.2 opens a Chrome debugging port and a stream port on
127.0.0.1 TCP, so any local account can drive a page that the harness
signed in as the admin. The skill now drives the instance only through the
coder CLI and the API:

- Remove the four journeys and the agent-browser helpers in lib.sh, the
  browser session in launch.sh and cleanup.sh, and their selftest checks.
- Feature files keep their CLI and API recipes and name each web UI
  sub-feature as not verified. Templates create from a starter with
  coder templates init. Coder Agents had only UI recipes, so its file goes.
- Hand-driven checks take their evidence folder from vc_new_evidence.

_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
@ThomasK33
ThomasK33 temporarily deployed to main November 10, 2026 11:49 — with GitHub Actions Inactive
- selftest works from a canonical temp folder, because /proc/<pid>/exe
  holds canonical paths, and its Postgres stand-in checks stop their
  stand-ins before they fail, so a failure no longer hangs the run.
- A new check proves vc_check fails a step on any failed command or
  pipeline stage. The unused vc_on_fail hook goes.
- Docs name the evidence folder with its random part, say when cleanup
  deletes every workspace, use vc_wait_agent for the agent wait, and give
  sign-in its Gotchas section back.

_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
@ThomasK33
ThomasK33 temporarily deployed to main November 10, 2026 13:14 — with GitHub Actions Inactive
scripts/coder-dev.sh ran make in the caller's folder, so vc_coder (and the
shim itself) failed silently outside the repo root: make found no Makefile,
or mise picked a Go toolchain that does not link the binary. It now runs
make from the project root.

The verify-coder selftest checks that with a stand-in make, and its
check-errexit check now catches a vc_check that drops pipefail. The
self-test list names both checks.

_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_

This branch was previously deployed

1 inactive deployment
main — 4aece1ec Deployed Nov 10, 2026 by ThomasK33 via cla #253335
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants