Skip to content

feat: store experiment rules with revisions and audit types - #29898

Merged
ThomasK33 merged 10 commits into
thomask33/experiment-rules-1-evaluatorfrom
thomask33/experiment-rules-2-storage
Sep 29, 2026
Merged

ThomasK33 merged 10 commits into
thomask33/experiment-rules-1-evaluatorfrom
thomask33/experiment-rules-2-storage

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Slice 2 of 4 for runtime experiment rules: persist one rule per experiment in site_configs, with revisions that only increase, and add an audit record type. Nothing calls this code yet, so behavior does not change.

What's in it

  • Queries in siteconfig.sql, all checked by dbauthz against ResourceDeploymentConfig:
    • GetExperimentRules (read) lists experiment_rule: keys with starts_with, not LIKE, because _ is a LIKE wildcard.
    • GetExperimentRule (read) reads one row.
    • UpsertExperimentRule (update) writes one row.
  • experiments.WriteRule(ctx, db, actor, ex, rule, expectedRevision) validates the rule first, then does everything in one READ COMMITTED transaction:
    • It takes an advisory lock for that experiment, waiting at most 5 s.
    • It reads the current row; a missing row counts as revision 0.
    • It returns *RevisionConflictError when the expected revision is stale.
    • An identical mode and condition writes nothing and returns changed=false. The first inherit on a missing row is a change and stores revision 1.
    • Otherwise it writes revision+1 with the actor and time, and returns the old and new rule as read and written inside the transaction.
  • experiments.NewDBStore(db) implements the evaluator's Store. It reads rules by prefix, plus the user's id, username, email, explicit site roles, member org names, and <org>/<group> group names (including <org>/Everyone). It runs as the system.
  • Audit:
    • A new resource_type enum value experiment_rule (migration 000602; the down migration is a comment because PostgreSQL cannot drop enum values).
    • A new struct database.ExperimentRule. experiments.AuditRecord derives its ID from the experiment name, so audit history groups by experiment.
    • In the audit table, experiment, mode revision and condition are tracked (ActionTrack), so the audit diff shows the old and new condition. In Enterprise, the slog audit backend also writes that diff to the server log. Admins and auditors can already read the condition through the rules API.
    • The audit page filter includes the new resource type.

Testing

  • WriteRule and NewDBStore tests run against real PostgreSQL. They cover stale revisions after a reset and a recreate, two concurrent writers with the same expected revision (exactly one wins), no-ops, and rollback after a forced error.
  • dbauthz method tests and an audit diff test that shows both condition texts.

Remote dogfood UAT through Coder Agents passed the upgrade from slice 1 (existing state kept, enum value added), a fresh install, unchanged /api/v2/experiments, and the WriteRule tests under -race. The licensed audit-page check could not run: the test workspace has no premium license. That round tested 015979ca96. The commit after it only adds AuditRecord, and a follow-up UAT round will cover the final head.

Note

Migration 000604 was the next free number when this was written. Recheck it before merging.

Stack

  1. feat: add per-user experiment rule evaluator #29869: evaluator library
  2. This PR: storage and audit types
  3. Enforcement and discovery (/api/v2/experiments, meta tag, chatd mcp-tool-search, ruleguard)
  4. Control plane (experimental API, coder exp experiment-rules CLI, docs)

Note

This PR was written by an AI agent (Xum).


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

@ThomasK33
ThomasK33 added this pull request to stack #29899 September 25, 2026 13:06
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-28T14:52:12.863856Z 6c09577 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 22e95eb7fd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/audit/request.go Resolved
Comment thread coderd/experiments/dbstore.go Resolved
Comment thread coderd/experiments/write.go Resolved
Comment thread coderd/database/dbauthz/dbauthz.go Resolved
Comment thread coderd/experiments/dbstore.go Resolved
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 31e895a934

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: 6ca7041cda

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
ThomasK33 marked this pull request as ready for review September 25, 2026 14:54
@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-2-storage branch from 6ca7041 to 85835b9 Compare September 28, 2026 08:33
@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-2-storage branch from 85835b9 to cb8486a Compare September 28, 2026 10:28
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: cb8486af35

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-2-storage branch from cb8486a to 9e7eafa Compare September 28, 2026 11:54
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 9e7eafa87e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread enterprise/audit/diff_internal_test.go Outdated
Comment on lines +395 to +396
// Experiment rule conditions can name users, so audit diffs
// record that the condition changed without its text.

@ethanndickson ethanndickson Sep 28, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intentional? sounds like nonsense

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, that reason didn't hold up: auditors can already read the condition through the rules API, and hiding it only removed the history of what changed. condition is now ActionTrack, and ConditionTracked checks that the audit diff carries both the old and the new text (6c09577). The condition text still stays out of server logs, /deployment/config and support bundles.

Reply drafted by an AI agent (xum).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction to my previous reply: the condition text does reach the server log, through the audit lines that Enterprise's slog audit backend writes for every audit entry (and the line logged when an audit export fails). It stays out of every other log line, /deployment/config and support bundles. #29971 documents this and tests it (944c6ea).

Reply drafted by an AI agent (xum).

@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-2-storage branch from 9e7eafa to 6c09577 Compare September 28, 2026 14:48
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 6c095775fb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
ThomasK33 requested a review from jakehwll September 29, 2026 09:45
Store one runtime rule per user-scoped experiment in site_configs under
the key experiment_rule:<experiment>. Add GetExperimentRules (prefix
listing via starts_with), GetExperimentRule and UpsertExperimentRule,
authorized against DeploymentConfig read and update.

experiments.WriteRule validates the rule, then in one READ COMMITTED
transaction takes a per-experiment advisory lock, checks the expected
revision (returning *RevisionConflictError on mismatch), skips identical
rules and otherwise writes revision+1 with the actor and time. It
returns the old and new decoded rules; the old rule of a missing row is
the zero Rule (revision 0).

experiments.NewDBStore reads rules and the D1 user attributes as the
system for the Evaluator.

Add the experiment_rule audit resource type with database.ExperimentRule,
whose condition is audited as a secret. Nothing calls WriteRule or
NewDBStore outside tests yet.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…Record

The audit comments referred to an ID derived from the experiment name, but nothing derived it. AuditRecord builds the audited database.ExperimentRule for a rule, with ID = uuid.NewSHA1(fixed namespace, experiment) so audit history groups by experiment.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
Exclude groups of soft-deleted organizations from condition attributes (their names can be reused), sort roles, organizations and groups so list-indexing conditions are deterministic, let WriteRule replace a stored value whose revision is readable but whose other fields are malformed, and cover the Experiment Rule audit filter option with a story.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
…on in Vitest

Per FE1, story play functions must not assert behavior. Replace the ExperimentRuleFilter story with a hook test that checks the resource-type menu offers experiment_rule with its label and emits it on selection.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
…e type migration to 000604

main added its own 000602 and 000603 migrations.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
…e revision

A stored value with invalid JSON or a missing or non-positive revision
made WriteRule fail, so an experiment failing closed on such a row could
not be repaired through the API. Treat it as revision 0: expected
revision 0 replaces it at revision 1, any other expected revision
conflicts with current revision 0.
The oxlint no-restricted-types rule on main forbids importing FC and
PropsWithChildren from react.
Conditions are admin-written settings that auditors can already read
through the rules API, so masking them only hid what changed. Audit the
condition text like the other rule fields.
@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-2-storage branch from 6c09577 to 697d678 Compare September 29, 2026 11:50
@ThomasK33
ThomasK33 merged commit a0b00b5 into main Sep 29, 2026
63 of 84 checks passed
@ThomasK33
ThomasK33 deleted the thomask33/experiment-rules-2-storage branch September 29, 2026 12:23
ThomasK33 added a commit that referenced this pull request Sep 29, 2026
Slice 3 of 4 for runtime experiment rules: evaluate stored rules per
user in discovery, the frontend experiments query, and the pilot
`mcp-tool-search` gate in chatd. There is still no way to edit rules
through the product; that comes in slice 4.

With no rules stored and a healthy database, backend decisions match
today. This PR deliberately changes three things:

1. **Fail closed on rule reads.** If the rules read fails,
`mcp-tool-search` is off, even when it is enabled at startup.
2. **Frontend freshness.** The experiments query now refetches.
3. **Caching.** `/api/v2/experiments` and rendered HTML are sent with
`Cache-Control: no-store`. Static assets keep their long-lived caching.

## What's in it

- **coderd** builds `api.ExperimentEvaluator` from
`experiments.NewDBStore` plus the startup `--experiments` list.
- **Discovery:** `GET /api/v2/experiments` and the `<meta
property="experiments">` tag both use `EnabledExperiments(ctx, caller)`.
- **chatd:** `mcp-tool-search` is now user-scoped.
`chatd.Config.ExperimentEvaluator` is required, so `chatd.New` returns
an error without it. `prepareGeneration` decides once per turn for the
chat owner (`chat.OwnerID`); child chats inherit the owner.
- **Ruleguard** (`userScopedExperimentStaticCheck` in
`scripts/rules.go`): rejects static `Experiments.Enabled(...)` and
`httpmw.RequireExperiment*(...)` checks of user-scoped experiments in
server packages. A codersdk test keeps its list equal to
`codersdk.ExperimentsUserScoped`.
- **Frontend:** the experiments query is keyed by user ID and seeded
from the meta tag only when the embedded user matches. It is stale 60 s
after page load and refetches on mount, window focus and reconnect.
Logout also clears the experiments metadata. The dashboard keeps the
last loaded list when a background refetch fails, instead of replacing
the page with an error.
- **Once per turn:** chatd decides `mcp-tool-search` on a turn's first
step and reuses that decision for the turn's later steps, so a rule
change applies from the next turn and never withdraws a `find_tools`
call the model already made. The decision lives on the chat runner,
keyed by the turn's prompt row, like the stop-nudge tracker. A runner
handoff in the middle of a turn evaluates again.
- **Reference audit:** the only non-test server check of
`mcp-tool-search` is in `generation_preparer.go`, and it now goes
through the evaluator.

## Testing

- **Go:** per-user results and `no-store` on `/api/v2/experiments`; the
per-user meta tag and the HTML cache headers in `site`; group and org
conditions end to end in enterprise; chatd owner eligibility, on → off →
on across turns including a child chat, and a failed rules read.
- **Vitest:** user-keyed seeding, staleness from page load after a
remount, refetch on focus, logout clearing, and the dashboard surviving
a failed refetch.

Remote dogfood UAT through Coder Agents passed on a licensed test
deployment:
- per-user discovery in both the API and the meta tag;
- widening a rule, and turning it off, with no restart;
- `off` beating a startup default, and `inherit` restoring it;
- fail closed on malformed JSON and on an unknown mode, with no
condition text in the logs;
- the `no-store` headers;
- real-model chatd enforcement: `find_tools` offered on turn 1, gone
after `off`, back on turn 3.

That round tested `b57207d4ee`. The later commits change only tests, and
a follow-up round will cover the final head.

**For the author:** `coderd/x/chatd/ARCHITECTURE.md`'s list of the
runner's local state doesn't mention the per-turn trackers (debug turn,
session start, stop nudges, and now this experiment decision). I did not
edit that document.

**Known behavior:** a broken rule logs one error line per evaluation
(per request or turn), with only the experiment, revision, category and
position.

## Stack

1. #29869: evaluator library
2. #29898: storage and audit types
3. **This PR:** enforcement and discovery
4. Control plane (experimental API, `coder exp experiment-rules` CLI,
docs)

> [!NOTE]
> This PR was written by an AI agent (Xum).

---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_

---------

Signed-off-by: Thomas Kosiewski <tk@coder.com>
ThomasK33 added a commit that referenced this pull request Sep 29, 2026
Slice 4 of 4 for runtime experiment rules: an experimental management
API and a hidden CLI, so admins can target, widen, or kill a user-scoped
experiment at runtime without a redeploy. This PR also adds a
startup-default note on the Deployment Overview page and a docs section.

## What's in it

- **API** under `/api/experimental/experiments/rules` (behind
`apiKeyMiddleware` and `httpmw.NoStore`):
- `GET` returns, per user-scoped experiment, the startup default and the
stored rule (mode, condition, revision, actor, time). Stored rules for
other experiments follow, marked `ignored`.
  - `PUT /{experiment}` takes `{mode, condition?, expected_revision}`.
- **Write path:**
- It authorizes read and update on `DeploymentConfig` before parsing
anything. A denied request gets 403 and is not audited.
- It validates next. An invalid condition returns 400 with the full CEL
diagnostics, is not audited, and leaves the rule unchanged.
- It then starts the audit and calls `experiments.WriteRule`. A stale
revision returns 409 and a transaction error returns 500; both are
audited with an empty diff.
- An identical write returns 200, creates no audit entry, and keeps the
revision.
- The audit export happens after commit. If it fails, that is logged and
the committed change stands (the existing audit contract).
- **SDK:** `ExperimentalClient.ExperimentRules` and
`ExperimentalClient.PutExperimentRule`.
- **CLI:** hidden `coder exp experiment-rules list|on|off|set|reset`.
- Write commands take `--expected-revision`. Without it they read the
current revision first, which only protects the gap between the two
calls; the help text says so.
  - A 409 prints the current rule and exits non-zero, with no retry.
- `reset` warns when the startup default is on, because reset is not a
kill switch.
- **Deployment Overview:** a note says the Experiments values are
startup defaults that runtime rules can override per user.
- **Docs:** a new "Target experiments at runtime" section in
`docs/reference/feature-stages.md` covers the modes, roles, CEL fields
and limits, reset vs. off, revisions and audit, the kill-switch
guarantee (including the rolling-upgrade requirement), frontend
freshness, fail-closed behavior, and that rules are not an authorization
boundary.

## Testing

- **HTTP:** every mode applies on the next request with no restart;
rejected writes keep the rule; RBAC (members 403 on GET and PUT;
auditors can read but not write; an invalid body from an unauthorized
caller still gets 403); malformed and ignored stored rows.
- **HA:** two coderd instances share one database. A PUT through A is
seen by B's `/api/v2/experiments` and by B's evaluator, which is the one
chatd uses.
- **Audit (enterprise, licensed):** an effective change creates one
entry whose diff carries the condition, and the entry is found before
the diff is checked. No-ops, 403s and 400s create no entry. A 409 and an
injected 500 are audited with an empty diff. A failed export keeps the
change.
- **Redaction:** a sentinel condition appears only in the rules API,
audit entries, and the server log's audit lines (Enterprise's slog audit
backend writes every audit diff there). It is absent from every other
log line, `/deployment/config`, the support bundle, and telemetry. The
test wires the slog audit backend so this is checked as stated.
- **CLI:** write and list, a conflict that is not retried, and the reset
warning.

Remote dogfood UAT through Coder Agents passed on a licensed test
deployment:
- the CLI;
- per-user results in the browser;
- 403 for members, including an invalid body;
- group targeting, including removing the user from the group;
- sentinel redaction across logs, `/deployment/config`, the support
bundle and the audit API (at that head the audit still masked the
condition; it is now tracked);
- the audit page showing the condition as changed but masked (same
note);
- the Overview note;
- the static default with `off` and `reset`;
- chatd enforcement toggled between turns.

That round tested `135aaaa41e`. The later commits are a CLI refactor and
a test addition, and a follow-up round will cover the final head.

## Follow-ups (low, not in this PR)

- A broken rule logs one error line per evaluation. Consider
rate-limiting it per experiment and revision.
- There's no API or CLI path to clear a stored rule for an experiment
that isn't user-scoped; those rules are inert and listed as `ignored`.
- A malformed stored rule is listed with zero `updated_by` and
`updated_at` values instead of omitting them.
- CLI wording: the 409 message, conflict output showing a UUID instead
of a username, a no-op write not saying nothing changed, and the parent
help lacking the `user.*` fields.
- `PutExperimentRuleRequest.mode` is not in the Swagger `required` list.
An omitted mode already returns 400.
- The dashboard's experiments query (from #29964) refreshes on mount,
window focus or reconnect once 60 seconds old, not on a timer, and its
comment says it bounds staleness. Correct the comment, or add a refetch
interval if the UI must pick up rule changes without a reload.

## Stack

1. #29869: evaluator library
2. #29898: storage and audit types
3. #29964: enforcement and discovery
4. **This PR:** control plane

> [!NOTE]
> This PR was written by an AI agent (Xum).

---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_

---------

Signed-off-by: Thomas Kosiewski <tk@coder.com>
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 29, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants