Skip to content

feat: add experimental API and CLI to manage experiment rules - #29971

Merged
ThomasK33 merged 10 commits into
thomask33/experiment-rules-3-enforcementfrom
thomask33/experiment-rules-4-control-plane
Sep 29, 2026
Merged

ThomasK33 merged 10 commits into
thomask33/experiment-rules-3-enforcementfrom
thomask33/experiment-rules-4-control-plane

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

Slice 4 of 4 for runtime experiment rules: an experimental management API and a hidden CLI, so admins can target, widen, or kill a user-scoped experiment at runtime without a redeploy. This PR also adds a startup-default note on the Deployment Overview page and a docs section.

What's in it

  • API under /api/experimental/experiments/rules (behind apiKeyMiddleware and httpmw.NoStore):
    • GET returns, per user-scoped experiment, the startup default and the stored rule (mode, condition, revision, actor, time). Stored rules for other experiments follow, marked ignored.
    • PUT /{experiment} takes {mode, condition?, expected_revision}.
  • Write path:
    • It authorizes read and update on DeploymentConfig before parsing anything. A denied request gets 403 and is not audited.
    • It validates next. An invalid condition returns 400 with the full CEL diagnostics, is not audited, and leaves the rule unchanged.
    • It then starts the audit and calls experiments.WriteRule. A stale revision returns 409 and a transaction error returns 500; both are audited with an empty diff.
    • An identical write returns 200, creates no audit entry, and keeps the revision.
    • The audit export happens after commit. If it fails, that is logged and the committed change stands (the existing audit contract).
  • SDK: ExperimentalClient.ExperimentRules and ExperimentalClient.PutExperimentRule.
  • CLI: hidden coder exp experiment-rules list|on|off|set|reset.
    • Write commands take --expected-revision. Without it they read the current revision first, which only protects the gap between the two calls; the help text says so.
    • A 409 prints the current rule and exits non-zero, with no retry.
    • reset warns when the startup default is on, because reset is not a kill switch.
  • Deployment Overview: a note says the Experiments values are startup defaults that runtime rules can override per user.
  • Docs: a new "Target experiments at runtime" section in docs/reference/feature-stages.md covers the modes, roles, CEL fields and limits, reset vs. off, revisions and audit, the kill-switch guarantee (including the rolling-upgrade requirement), frontend freshness, fail-closed behavior, and that rules are not an authorization boundary.

Testing

  • HTTP: every mode applies on the next request with no restart; rejected writes keep the rule; RBAC (members 403 on GET and PUT; auditors can read but not write; an invalid body from an unauthorized caller still gets 403); malformed and ignored stored rows.
  • HA: two coderd instances share one database. A PUT through A is seen by B's /api/v2/experiments and by B's evaluator, which is the one chatd uses.
  • Audit (enterprise, licensed): an effective change creates one entry whose diff carries the condition, and the entry is found before the diff is checked. No-ops, 403s and 400s create no entry. A 409 and an injected 500 are audited with an empty diff. A failed export keeps the change.
  • Redaction: a sentinel condition appears only in the rules API, audit entries, and the server log's audit lines (Enterprise's slog audit backend writes every audit diff there). It is absent from every other log line, /deployment/config, the support bundle, and telemetry. The test wires the slog audit backend so this is checked as stated.
  • CLI: write and list, a conflict that is not retried, and the reset warning.

Remote dogfood UAT through Coder Agents passed on a licensed test deployment:

  • the CLI;
  • per-user results in the browser;
  • 403 for members, including an invalid body;
  • group targeting, including removing the user from the group;
  • sentinel redaction across logs, /deployment/config, the support bundle and the audit API (at that head the audit still masked the condition; it is now tracked);
  • the audit page showing the condition as changed but masked (same note);
  • the Overview note;
  • the static default with off and reset;
  • chatd enforcement toggled between turns.

That round tested 135aaaa41e. The later commits are a CLI refactor and a test addition, and a follow-up round will cover the final head.

Follow-ups (low, not in this PR)

  • A broken rule logs one error line per evaluation. Consider rate-limiting it per experiment and revision.
  • There's no API or CLI path to clear a stored rule for an experiment that isn't user-scoped; those rules are inert and listed as ignored.
  • A malformed stored rule is listed with zero updated_by and updated_at values instead of omitting them.
  • CLI wording: the 409 message, conflict output showing a UUID instead of a username, a no-op write not saying nothing changed, and the parent help lacking the user.* fields.
  • PutExperimentRuleRequest.mode is not in the Swagger required list. An omitted mode already returns 400.
  • The dashboard's experiments query (from feat: evaluate experiment rules per user in discovery and chatd #29964) refreshes on mount, window focus or reconnect once 60 seconds old, not on a timer, and its comment says it bounds staleness. Correct the comment, or add a refetch interval if the UI must pick up rule changes without a reload.

Stack

  1. feat: add per-user experiment rule evaluator #29869: evaluator library
  2. feat: store experiment rules with revisions and audit types #29898: storage and audit types
  3. feat: evaluate experiment rules per user in discovery and chatd #29964: enforcement and discovery
  4. This PR: control plane

Note

This PR was written by an AI agent (Xum).


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

@ThomasK33
ThomasK33 added this pull request to stack #29899 September 26, 2026 05:36
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-28T15:51:38.183623Z 944c6ea Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: be4dc00d0d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread codersdk/experimentrules.go Resolved
Comment thread codersdk/experimentrules.go Resolved
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 7391ba55ec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-4-control-plane branch from 7391ba5 to 18554df Compare September 28, 2026 08:33
@ThomasK33
ThomasK33 marked this pull request as ready for review September 28, 2026 09:22
@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-4-control-plane branch from 18554df to 365a83e Compare September 28, 2026 09:50
@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-4-control-plane branch from 365a83e to 52ba29c Compare September 28, 2026 10:28
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 52ba29c684

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/experiments/write.go Resolved
@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-4-control-plane branch from 52ba29c to b556287 Compare September 28, 2026 11:54
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b5562875b7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread codersdk/experimentrules.go Resolved
Comment thread codersdk/experimentrules.go Resolved
@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-4-control-plane branch from b556287 to 4511890 Compare September 28, 2026 14:48
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4511890a92

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/experiments.go Resolved
Comment thread docs/reference/feature-stages.md Resolved
@coderagents

coderagents Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Documentation Check

Updates Needed

  • docs/reference/feature-stages.md (open) - This was addressed in c9120bb834: the entry now says the audit entry records "the old and new condition" and that "Audit lines in the server log carry the same diff," matching enterprise/audit/table.go's condition: ActionTrack and the enterprise auditor's always-on backends.NewSlog export (enterprise/cli/server.go).

Automated review via Coder Agents

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 944c6eaf0e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-4-control-plane branch from 944c6ea to 9b504ad Compare September 29, 2026 08:14
@ThomasK33
ThomasK33 requested a review from jakehwll September 29, 2026 09:45
Add GET /api/experimental/experiments/rules and
PUT /api/experimental/experiments/rules/{experiment}. The handler
authorizes before parsing, validates before auditing, writes through
experiments.WriteRule, audits effective changes and failures after
commit, and skips auditing identical writes. Responses are no-store.

Add ExperimentalClient.ExperimentRules and PutExperimentRule, the hidden
coder exp experiment-rules list|on|off|set|reset command, a startup
default note on the Deployment Overview page, and a docs section in
feature-stages.md.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…ites are not audited

Signed-off-by: Thomas Kosiewski <tk@coder.com>
…names

The rules API returns an empty mode for a malformed stored rule and arbitrary experiment names for ignored rows, but the response types reused the closed ExperimentRuleMode and Experiment enums. Make ExperimentRule.Mode and ExperimentRuleEntry.Experiment plain strings so generated clients can represent both; the write request keeps the closed mode enum.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
… rules

parseStoredRule can no longer fail after WriteRule started treating a row
without a readable positive revision as revision 0, so drop its error and
unused parameter and update the ReadRules doc. Cover an unreadable row
listed at revision 0 and replaced through PUT with expected revision 0.
Revision 0 in a listed rule means the stored rule has no readable
positive revision and is malformed, not that it was never configured;
expected revision 0 also replaces such a rule. Regenerate the API docs
and TypeScript types.
…ule conditions

Audit entries now track the condition, and enterprise servers export every
audit entry to the server log. Document that, keep the promise that no
other log line carries the text, and make the redaction test export audit
lines to the captured log so it checks the promise as written.
@ThomasK33
ThomasK33 force-pushed the thomask33/experiment-rules-4-control-plane branch from 9b504ad to 077f05c Compare September 29, 2026 11:50
@ThomasK33
ThomasK33 merged commit 12be685 into main Sep 29, 2026
63 of 84 checks passed
@ThomasK33
ThomasK33 deleted the thomask33/experiment-rules-4-control-plane branch September 29, 2026 12:23
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 29, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants