Skip to content

feat: evaluate experiment rules per user in discovery and chatd - #29964

Merged
ThomasK33 merged 16 commits into
thomask33/experiment-rules-2-storagefrom
thomask33/experiment-rules-3-enforcement
Sep 29, 2026
Merged

ThomasK33 merged 16 commits into
thomask33/experiment-rules-2-storagefrom
thomask33/experiment-rules-3-enforcement

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

Slice 3 of 4 for runtime experiment rules: evaluate stored rules per user in discovery, the frontend experiments query, and the pilot mcp-tool-search gate in chatd. There is still no way to edit rules through the product; that comes in slice 4.

With no rules stored and a healthy database, backend decisions match today. This PR deliberately changes three things:

  1. Fail closed on rule reads. If the rules read fails, mcp-tool-search is off, even when it is enabled at startup.
  2. Frontend freshness. The experiments query now refetches.
  3. Caching. /api/v2/experiments and rendered HTML are sent with Cache-Control: no-store. Static assets keep their long-lived caching.

What's in it

  • coderd builds api.ExperimentEvaluator from experiments.NewDBStore plus the startup --experiments list.
  • Discovery: GET /api/v2/experiments and the <meta property="experiments"> tag both use EnabledExperiments(ctx, caller).
  • chatd: mcp-tool-search is now user-scoped. chatd.Config.ExperimentEvaluator is required, so chatd.New returns an error without it. prepareGeneration decides once per turn for the chat owner (chat.OwnerID); child chats inherit the owner.
  • Ruleguard (userScopedExperimentStaticCheck in scripts/rules.go): rejects static Experiments.Enabled(...) and httpmw.RequireExperiment*(...) checks of user-scoped experiments in server packages. A codersdk test keeps its list equal to codersdk.ExperimentsUserScoped.
  • Frontend: the experiments query is keyed by user ID and seeded from the meta tag only when the embedded user matches. It is stale 60 s after page load and refetches on mount, window focus and reconnect. Logout also clears the experiments metadata. The dashboard keeps the last loaded list when a background refetch fails, instead of replacing the page with an error.
  • Once per turn: chatd decides mcp-tool-search on a turn's first step and reuses that decision for the turn's later steps, so a rule change applies from the next turn and never withdraws a find_tools call the model already made. The decision lives on the chat runner, keyed by the turn's prompt row, like the stop-nudge tracker. A runner handoff in the middle of a turn evaluates again.
  • Reference audit: the only non-test server check of mcp-tool-search is in generation_preparer.go, and it now goes through the evaluator.

Testing

  • Go: per-user results and no-store on /api/v2/experiments; the per-user meta tag and the HTML cache headers in site; group and org conditions end to end in enterprise; chatd owner eligibility, on → off → on across turns including a child chat, and a failed rules read.
  • Vitest: user-keyed seeding, staleness from page load after a remount, refetch on focus, logout clearing, and the dashboard surviving a failed refetch.

Remote dogfood UAT through Coder Agents passed on a licensed test deployment:

  • per-user discovery in both the API and the meta tag;
  • widening a rule, and turning it off, with no restart;
  • off beating a startup default, and inherit restoring it;
  • fail closed on malformed JSON and on an unknown mode, with no condition text in the logs;
  • the no-store headers;
  • real-model chatd enforcement: find_tools offered on turn 1, gone after off, back on turn 3.

That round tested b57207d4ee. The later commits change only tests, and a follow-up round will cover the final head.

For the author: coderd/x/chatd/ARCHITECTURE.md's list of the runner's local state doesn't mention the per-turn trackers (debug turn, session start, stop nudges, and now this experiment decision). I did not edit that document.

Known behavior: a broken rule logs one error line per evaluation (per request or turn), with only the experiment, revision, category and position.

Stack

  1. feat: add per-user experiment rule evaluator #29869: evaluator library
  2. feat: store experiment rules with revisions and audit types #29898: storage and audit types
  3. This PR: enforcement and discovery
  4. Control plane (experimental API, coder exp experiment-rules CLI, docs)

Note

This PR was written by an AI agent (Xum).


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

@ThomasK33
ThomasK33 added this pull request to stack #29899 September 26, 2026 01:21
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-29T08:21:33.168503Z ce3d2a0 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a6d180a32d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/generation_preparer.go Resolved
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f3ccd3eab4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/options.go Resolved
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e6d6c758d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/runner.go Resolved
@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-3-enforcement branch from 86f9e26 to f8766c3 Compare September 28, 2026 08:33
@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-3-enforcement branch from 5194979 to a38de1b Compare September 28, 2026 10:28
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: a38de1bd98

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-3-enforcement branch from a38de1b to 89ef6c7 Compare September 28, 2026 11:54
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 89ef6c78cc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
ThomasK33 marked this pull request as ready for review September 28, 2026 12:18
@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-3-enforcement branch from 89ef6c7 to 287b980 Compare September 28, 2026 14:48
Comment thread coderd/x/chatd/generation_preparer.go Outdated
Comment on lines +732 to +742
if decideMCPToolSearch(mcpToolSearchInput{
experimentEnabled: server.experiments.Enabled(codersdk.ExperimentMCPToolSearch),
candidates: deferredCandidates,
dynamicToolNames: dynamicToolNames,
// The owner is the subject: only the owner posts turns and
// descendant chats inherit it. Preparation runs for every step,
// so the first step decides and later steps of the same turn
// reuse that decision; otherwise a rule change mid-turn would
// reject find_tools calls the model already issued.
experimentEnabled: input.TurnExperiments.mcpToolSearchEnabled(stopNudgeKey(input.Messages), func() bool {
return server.experimentEvaluator.Enabled(ctx, chat.OwnerID, codersdk.ExperimentMCPToolSearch)
}),
candidates: deferredCandidates,
dynamicToolNames: dynamicToolNames,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this evaluates the rule before decideMCPToolSearch checks for candidates, so a chat with no MCP tools still reads the rules on every turn and then throws the result away (find_tools is never offered without candidates).

  • Cost for each turn of such a chat: 1 site_configs read, or 4 reads plus a CEL run when the rule is a condition. The turn cache keeps it to once per turn, not once per step.
  • The larger effect is logging. A condition that errors at runtime for some users logs an Error on each evaluation (evaluator.go:224, :235). Today that happens on every turn of every chat in the deployment, not only on chats that use MCP.
  • A probe test (chat with no MCP servers, store that counts Rules calls) saw 1 read on this head and 0 with the change below. The 26 existing tool-search and rules tests in chatd still pass with the change.

The outcome does not change. If MCP tools appear mid-turn (for example, the agent creates a workspace at step 2), step 2 makes the decision instead of step 1. No find_tools call can exist before then, so nothing breaks. The ARCHITECTURE.md line "The first step decides" would become "The first step with MCP tools decides".

Suggested change
if decideMCPToolSearch(mcpToolSearchInput{
experimentEnabled: server.experiments.Enabled(codersdk.ExperimentMCPToolSearch),
candidates: deferredCandidates,
dynamicToolNames: dynamicToolNames,
// The owner is the subject: only the owner posts turns and
// descendant chats inherit it. Preparation runs for every step,
// so the first step decides and later steps of the same turn
// reuse that decision; otherwise a rule change mid-turn would
// reject find_tools calls the model already issued.
experimentEnabled: input.TurnExperiments.mcpToolSearchEnabled(stopNudgeKey(input.Messages), func() bool {
return server.experimentEvaluator.Enabled(ctx, chat.OwnerID, codersdk.ExperimentMCPToolSearch)
}),
candidates: deferredCandidates,
dynamicToolNames: dynamicToolNames,
// The owner is the subject: only the owner posts turns and
// descendant chats inherit it. Preparation runs for every step,
// so the first step with MCP candidates decides and later steps
// of the same turn reuse that decision; otherwise a rule change
// mid-turn would reject find_tools calls the model already issued.
// Without candidates find_tools is never offered, so skip the read.
mcpToolSearchEnabled := false
if len(deferredCandidates) > 0 {
mcpToolSearchEnabled = input.TurnExperiments.mcpToolSearchEnabled(stopNudgeKey(input.Messages), func() bool {
return server.experimentEvaluator.Enabled(ctx, chat.OwnerID, codersdk.ExperimentMCPToolSearch)
})
}
if decideMCPToolSearch(mcpToolSearchInput{
experimentEnabled: mcpToolSearchEnabled,
candidates: deferredCandidates,
dynamicToolNames: dynamicToolNames,
Probe test, if you want it as a regression test
// Inside TestMCPToolSearchExperimentRules:
t.Run("no MCP tools skips the rules read", func(t *testing.T) {
	t.Parallel()
	var reads atomic.Int64
	h := newHarness(t, codersdk.ExperimentsKnown, func(database.Store) experimentrules.Store {
		return countingRulesStore{reads: &reads}
	})
	ctx := testutil.Context(t, testutil.WaitLong)
	chat, err := h.server.CreateChat(ctx, chatd.CreateOptions{
		OrganizationID: h.org.ID,
		OwnerID:        h.owner.ID,
		Title:          "no mcp",
		ModelConfigID:  h.model.ID,
		InitialUserContent: []codersdk.ChatMessagePart{
			codersdk.ChatMessageText("hello"),
		},
	})
	require.NoError(t, err)
	_ = h.turnTools(t, 1)
	waitForChatStatus(ctx, t, h.db, chat.ID, database.ChatStatusWaiting)
	require.Zero(t, reads.Load())
})

type countingRulesStore struct {
	experimentstest.Store
	reads *atomic.Int64
}

func (s countingRulesStore) Rules(ctx context.Context) (map[codersdk.Experiment]experimentrules.StoredRule, error) {
	s.reads.Add(1)
	return s.Store.Rules(ctx)
}

Generated by xum.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, applied as suggested. The rule is now read only when the step has MCP candidates, so a chat without MCP tools never reads the rules or logs a condition error. Your probe is now TestMCPToolSearchExperimentRules/no_MCP_tools_skips_the_rules_read: it failed on 287b980 and passes now (d8264a9). ARCHITECTURE.md now says the first step with MCP candidates decides (ce3d2a0).

Reply drafted by an AI agent (xum).

Comment thread coderd/experiments.go
Comment on lines +21 to +23
// The result is personalized by the caller's rules, so it is served
// with no-store caching (see the route registration).
httpapi.Write(ctx, rw, http.StatusOK, api.ExperimentEvaluator.EnabledExperiments(ctx, apiKey.UserID))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not totally in love with this endpoint now only returning the user's experiments tbh, but im not sure what the alternative is...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I agree. Personally, I’d prefer folding it into the stable experiments endpoint. For now, I don’t want to introduce experimental changes to a stable API.

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ce3d2a0fe6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/ARCHITECTURE.md Resolved
Comment thread site/src/api/queries/experiments.ts Resolved
@ThomasK33
ThomasK33 requested a review from jakehwll September 29, 2026 09:45
experimentsQuery.error ||
// Experiments refetch in the background; keep the last list when a
// refetch fails instead of replacing the dashboard with an error.
(!experimentsQuery.data && experimentsQuery.error) ||

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Would isPending be more correct here?

Suggested change
(!experimentsQuery.data && experimentsQuery.error) ||
(!experimentsQuery.isPending && experimentsQuery.error) ||

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried it, and isPending would bring back the error this guards against. In react-query v5, a failed background refetch sets status to error while keeping data, so isPending is already false and !isPending && error shows the ErrorAlert. With the suggestion applied, keeps the dashboard value when an experiments refetch fails fails. !data expresses what we want: show the error only when there is no list to fall back on, so I'm keeping it.

Reply drafted by an AI agent (xum).

Discovery (/api/v2/experiments and the HTML experiments meta tag) and the
pilot mcp-tool-search check in chatd now decide user-scoped experiments
with experiments.Evaluator for the caller or chat owner, failing closed
when rules cannot be read. mcp-tool-search joins ExperimentsUserScoped.

- coderd builds API.ExperimentEvaluator on the DB store; startup fails on
  a constructor error. chatd.New requires an evaluator and returns an
  error without one.
- /api/v2/experiments and rendered HTML are served with no-store.
- The experiments query is keyed by user, seeded from embedded metadata
  only for the embedded user, stale after 60 s from page load, and
  refetches on mount, focus and reconnect. Logout clears the metadata.
- A ruleguard rule rejects static server-side checks of user-scoped
  experiments; a codersdk test keeps its list equal to
  ExperimentsUserScoped.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Reuse the resolved experiments list in newInternalTestServer and drop a
logout test comment that restated the production comment.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
…keeping the dashboard

The experiments query now refetches in the background, and DashboardProvider keeps the last list when a refetch fails instead of replacing the dashboard with an error. Add a Vitest for that guard; it fails without it.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
prepareGeneration runs for every step of a turn, and each step runs as
a separate runner task. Evaluating the mcp-tool-search rule there let a
rule change between steps drop find_tools after the model had already
called it, so the call was rejected as not active in this turn.

Keep the decision on the runner, keyed by the turn's prompt row, so the
first step decides and later steps of the same turn reuse it. The next
turn evaluates the rule again.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
…d experiments refetch

Replace the DOM-presence assertion (FE1) with a context consumer that
records the experiments it receives and its unmount. After a failed
background refetch the consumer must stay mounted and keep receiving
the previously loaded list.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
…tion

prepareGeneration now requires TurnExperiments; the internal tests that
call it directly supply an empty holder.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
stopNudgeKey returns 0 when the history has no user prompt, so caching under that key would pin one mcp-tool-search decision across such turns. Evaluate every step in that case.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
A canceled task of an older turn can finish evaluating after the next turn stored its decision. Prompt row IDs come from a sequence, so reject results for a smaller turn key instead of overwriting the newer decision.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
…eriment decision

Per AGENTS.md, the human author documents chatd architecture; this marks the runner's local-state section for that update.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
… mock fixtures

Inline the single-use stale time constant with its rationale, rename the
test fixtures to the mock prefix, replace the metadata factory with one
constant, and drop comments that restate the test.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
…tch test

Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I4a8e63bda9fc954a23ff2993770f3a1bf73e4426
Signed-off-by: Thomas Kosiewski <tk@coder.com>
The oxlint no-restricted-types rule on main forbids importing FC and
PropsWithChildren from react.
…andidates

The rule was evaluated before decideMCPToolSearch checked for candidates,
so a chat without MCP tools read the rules on every turn, and a condition
failing at runtime logged an error on every turn. Decide only on the
first step with candidates; without them find_tools is never offered.
@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-3-enforcement branch from ce3d2a0 to 0a167da Compare September 29, 2026 11:50
@ThomasK33
ThomasK33 merged commit 9aa7130 into main Sep 29, 2026
79 of 100 checks passed
@ThomasK33
ThomasK33 deleted the thomask33/experiment-rules-3-enforcement branch September 29, 2026 12:23
ThomasK33 added a commit that referenced this pull request Sep 29, 2026
Slice 4 of 4 for runtime experiment rules: an experimental management
API and a hidden CLI, so admins can target, widen, or kill a user-scoped
experiment at runtime without a redeploy. This PR also adds a
startup-default note on the Deployment Overview page and a docs section.

## What's in it

- **API** under `/api/experimental/experiments/rules` (behind
`apiKeyMiddleware` and `httpmw.NoStore`):
- `GET` returns, per user-scoped experiment, the startup default and the
stored rule (mode, condition, revision, actor, time). Stored rules for
other experiments follow, marked `ignored`.
  - `PUT /{experiment}` takes `{mode, condition?, expected_revision}`.
- **Write path:**
- It authorizes read and update on `DeploymentConfig` before parsing
anything. A denied request gets 403 and is not audited.
- It validates next. An invalid condition returns 400 with the full CEL
diagnostics, is not audited, and leaves the rule unchanged.
- It then starts the audit and calls `experiments.WriteRule`. A stale
revision returns 409 and a transaction error returns 500; both are
audited with an empty diff.
- An identical write returns 200, creates no audit entry, and keeps the
revision.
- The audit export happens after commit. If it fails, that is logged and
the committed change stands (the existing audit contract).
- **SDK:** `ExperimentalClient.ExperimentRules` and
`ExperimentalClient.PutExperimentRule`.
- **CLI:** hidden `coder exp experiment-rules list|on|off|set|reset`.
- Write commands take `--expected-revision`. Without it they read the
current revision first, which only protects the gap between the two
calls; the help text says so.
  - A 409 prints the current rule and exits non-zero, with no retry.
- `reset` warns when the startup default is on, because reset is not a
kill switch.
- **Deployment Overview:** a note says the Experiments values are
startup defaults that runtime rules can override per user.
- **Docs:** a new "Target experiments at runtime" section in
`docs/reference/feature-stages.md` covers the modes, roles, CEL fields
and limits, reset vs. off, revisions and audit, the kill-switch
guarantee (including the rolling-upgrade requirement), frontend
freshness, fail-closed behavior, and that rules are not an authorization
boundary.

## Testing

- **HTTP:** every mode applies on the next request with no restart;
rejected writes keep the rule; RBAC (members 403 on GET and PUT;
auditors can read but not write; an invalid body from an unauthorized
caller still gets 403); malformed and ignored stored rows.
- **HA:** two coderd instances share one database. A PUT through A is
seen by B's `/api/v2/experiments` and by B's evaluator, which is the one
chatd uses.
- **Audit (enterprise, licensed):** an effective change creates one
entry whose diff carries the condition, and the entry is found before
the diff is checked. No-ops, 403s and 400s create no entry. A 409 and an
injected 500 are audited with an empty diff. A failed export keeps the
change.
- **Redaction:** a sentinel condition appears only in the rules API,
audit entries, and the server log's audit lines (Enterprise's slog audit
backend writes every audit diff there). It is absent from every other
log line, `/deployment/config`, the support bundle, and telemetry. The
test wires the slog audit backend so this is checked as stated.
- **CLI:** write and list, a conflict that is not retried, and the reset
warning.

Remote dogfood UAT through Coder Agents passed on a licensed test
deployment:
- the CLI;
- per-user results in the browser;
- 403 for members, including an invalid body;
- group targeting, including removing the user from the group;
- sentinel redaction across logs, `/deployment/config`, the support
bundle and the audit API (at that head the audit still masked the
condition; it is now tracked);
- the audit page showing the condition as changed but masked (same
note);
- the Overview note;
- the static default with `off` and `reset`;
- chatd enforcement toggled between turns.

That round tested `135aaaa41e`. The later commits are a CLI refactor and
a test addition, and a follow-up round will cover the final head.

## Follow-ups (low, not in this PR)

- A broken rule logs one error line per evaluation. Consider
rate-limiting it per experiment and revision.
- There's no API or CLI path to clear a stored rule for an experiment
that isn't user-scoped; those rules are inert and listed as `ignored`.
- A malformed stored rule is listed with zero `updated_by` and
`updated_at` values instead of omitting them.
- CLI wording: the 409 message, conflict output showing a UUID instead
of a username, a no-op write not saying nothing changed, and the parent
help lacking the `user.*` fields.
- `PutExperimentRuleRequest.mode` is not in the Swagger `required` list.
An omitted mode already returns 400.
- The dashboard's experiments query (from #29964) refreshes on mount,
window focus or reconnect once 60 seconds old, not on a timer, and its
comment says it bounds staleness. Correct the comment, or add a refetch
interval if the UI must pick up rule changes without a reload.

## Stack

1. #29869: evaluator library
2. #29898: storage and audit types
3. #29964: enforcement and discovery
4. **This PR:** control plane

> [!NOTE]
> This PR was written by an AI agent (Xum).

---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_

---------

Signed-off-by: Thomas Kosiewski <tk@coder.com>
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 29, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants