Skip to content

fix: harden chat automation management audit, webhook size and docs - #30312

Closed
ThomasK33 wants to merge 7 commits into
mainfrom
tk/codagt-1226-automations-mgmt-hardening
Closed

ThomasK33 wants to merge 7 commits into
mainfrom
tk/codagt-1226-automations-mgmt-hardening

Conversation

@ThomasK33

Copy link
Copy Markdown
Member

Hardens the chat automations management API, following the security-review nits in CODAGT-1226 and the audit follow-up in CODAGT-1273.

Changes

  1. Audit before-image under the lock (CODAGT-1226 item 3, CODAGT-1273). chatd.UpdateAutomation now returns (old, updated, err), where old is the row it read under the row lock. The PATCH handler and the manage_automations tool (update, enable, disable) record old as the audit Old. Before, a concurrent owner edit between the caller's read and the lock showed up in the diff of the wrong action. The tool change is limited to the two audit call sites.
  2. Escaped webhook event cap (item 6). json.HTMLEscape turns each <, >, &, U+2028 and U+2029 inside a JSON string into a six-byte \u sequence, so a 256 KiB body could become about 1.5 MB of message text. The escaped event data is now capped at 256 KiB, the same as the request body limit (maxChatRequestBodyBytes). A larger event gets 413 Event data too large. with the escaped size in the detail, before admission: nothing is queued or created, and a single-use webhook is not consumed.
  3. Prompt edits keep queued inputs (item 5). Owner decision: no behavior change. The UpdateAutomation doc comment, the PATCH swagger description and docs/ai-coder/agents/automations.md now say that messages already queued keep the prompt they were sent with, and that disabling or deleting the automation removes them.
  4. dbauthz docs (item 2). UpdateChatAutomationByID and UpdateChatAutomationWebhookSecretByID now state that they authorize only RBAC update, and that the owner-only rule lives in chatd UpdateAutomation and RotateAutomationSecret, so callers must go through those services.
  5. Delete without update permission (item 1). Already fixed on main by feat: disable, re-enable, and rotate chat automations with schedule preview #30117: DeleteAutomation deletes the row first, so a denied delete changes nothing. This PR adds the missing partial-change test: a read + update role gets 403 on DELETE, and the automation stays enabled with the same queue generation and update time.
  6. Negative automation cap (item 4). Already fixed on main by feat: add chat automations management API behind chat-automations #30112: DeploymentValues.Validate rejects --chat-max-automations-per-owner below 1 at startup, covered by TestDeploymentValues_Validate_ChatLimits. No change here.

Tests

  • TestManageAutomationsToolAuditsLockedRow (CODAGT-1273 acceptance) and TestChatAutomations/UpdateAuditsLockedRow: the owner renames the automation between the caller's read and the lock; the audit diff has the changed field and not name. Both fail when the pre-read row is audited.
  • TestPublishAutomationWebhookEventSize (boundary) and TestChatAutomationEvents/EscapedEventTooLarge (413, no side effects, single-use not consumed).
  • TestChatAutomationsDeleteWithoutUpdate extension (denied delete changes nothing).

Remote dogfood UAT through Coder Agents passed: escaped-size boundary (262144 accepted, 262145 refused), single-use not consumed by a 413, a PATCHed prompt leaves the queued input on the old prompt and disable removes it, and coder server refuses --chat-max-automations-per-owner of -1 and 0.

Closes CODAGT-1226
Closes CODAGT-1273
Part of CODAGT-1135


📋 Implementation Plan

Plan: CODAGT-1226 + CODAGT-1273, automation management hardening (lane L2)

Contract: Linear CODAGT-1226 (6 items) and CODAGT-1273, plus owner decisions. Base origin/main (fcbf45c). Keep the manage_automations tool change minimal: lane L7 edits coderd/x/chatd/automations_tool.go and lane L1 edits other functions in coderd/x/chatd/automations.go. Do not touch site/.

Item 1: DELETE needs update permission (CODAGT-1226 #1)

Already fixed on main by #30117 (squash 5c1662e): DeleteAutomation deletes the row first (ActionDelete), then cleans queued rows as chatd. A 403 on delete therefore changes nothing.

  • Add the missing partial-change check to enterprise/coderd/exp_chat_automations_test.go TestChatAutomationsDeleteWithoutUpdate (or a sibling test): a member with a custom org role of read + update on chat_automation (no delete) gets an error on DELETE; the stored row afterwards still exists with Enabled true and unchanged QueueGeneration (and UpdatedAt).

Items 3 + CODAGT-1273: audit Old from the locked row (one fix)

  • Change func (p *Server) UpdateAutomation(...) (database.ChatAutomation, error) in coderd/x/chatd/automations.go to return (old, updated database.ChatAutomation, err error), where old is the row as read by GetChatAutomationsByIDsForUpdate under the lock (before the change). Update its doc comment: callers record old as the audit before-image.
  • coderd/exp_chat_automations.go patchChatAutomation: after a successful update set aReq.Old = old (keep the pre-read row as Old only for failures).
  • coderd/x/chatd/automations_tool.go: the disable case and manageAutomationsUpdate audit with old instead of the pre-read row. Change nothing else in that file.
  • Update all test call sites (grep -rn '\.UpdateAutomation(' --include=*_test.go).
  • Acceptance test (CODAGT-1273): in coderd/x/chatd/automations_tool_internal_test.go, using the automationReadHook store wrapper and the fixture pattern of TestManageAutomationsToolConcurrentRetarget, rename the automation through the service as the owner between the tool's read and the service lock, then run tool update (prompt change) and enable (on a disabled automation). Assert the tool succeeded, one audit entry was written, and its diff does not contain name (the locked row already had the new name) but contains the changed field (enabled; prompt is an ActionSecret field, so assert the key exists only if the diff records secret keys; otherwise assert on enabled / updated_at). Check how audit.MockAuditor exposes the diff (AuditLog.Diff JSON) first. Negative control: the test must fail if the tool audits the pre-read row (state this in a comment, verify by temporarily reverting once locally).

Item 2: owner-only rule lives only in chatd (CODAGT-1226 #2)

  • coderd/database/dbauthz/dbauthz.go: extend the doc comments of UpdateChatAutomationByID and UpdateChatAutomationWebhookSecretByID: these authorize only RBAC update, which org admins and site owners also hold; the rule that only the owner can change an automation (other than disabling) or rotate its secret lives in chatd UpdateAutomation and RotateAutomationSecret, so every caller (HTTP handlers, the manage_automations tool, run_now) must go through those services. The schedule scanner writes only through AdvanceChatAutomationScheduleCursor. Verified: no other non-test callers of either query exist.

Item 4: negative automation cap (CODAGT-1226 #4)

  • codersdk/deployment.go (*DeploymentValues).Validate: reject AI.Chat.MaxAutomationsPerOwner < 1 with an error naming --chat-max-automations-per-owner (style of the existing chat hook errors). cli/server.go already calls vals.Validate() at startup.
  • Test in codersdk/deployment_test.go using defaultDeploymentValues(t): 0 and -1 fail with the flag name in the error, 1 passes, the default passes.
  • Only this option is in scope; do not validate the other chat limits.

Item 5: prompt edit keeps queued inputs (owner decision: keep behavior, document it)

No behavior change. A prompt edit does not start a new queue generation, so messages already queued keep the prompt they were sent with; disabling (or deleting) the automation removes them.

  • Code comment in UpdateAutomation doc comment and at the req.Prompt branch.
  • API docs: the swagger @Description of patchChatAutomation (add one if missing) states it. Do not change codersdk struct comments (they regenerate site/src/api/typesGenerated.ts; site/ is out of scope for this lane).
  • docs/ai-coder/agents/automations.md: one sentence (one sentence per line, docs/AGENTS.md style) near the PATCH row or where editing is described.
  • Run make gen (or the narrow apidoc target) if swagger output changes; commit generated files.

Item 6: cap the escaped webhook body (owner decision)

Cap: the escaped event data must be at most 256 KiB, the same as the raw request body limit (maxChatRequestBodyBytes in coderd/exp_chats.go). This keeps the message a webhook produces no larger than what the endpoint reads.

  • coderd/x/chatd/automations_publish.go: exported MaxAutomationEventDataBytes = 256 * 1024 (comment: equals coderd's request body limit), ErrAutomationEventTooLarge plus AutomationEventTooLargeError{Size, Max int} wrapping it (pattern: AutomationQueueShareFullError). Split escaping into escapeAutomationEventData(body []byte) []byte (json.HTMLEscape). PublishAutomationWebhook escapes once after the JSON check and returns the typed error before publishAutomation when the escaped size exceeds the cap, so nothing is admitted, queued, created, or consumed. automationEventText(name, escaped) takes the escaped data; update TestAutomationEventText accordingly and add a size boundary unit test (exactly the cap passes the check, one byte more fails).
  • coderd/exp_chat_automations.go writeChatAutomationEventError: map it to 413 "Event data too large." with a Detail giving the escaped size and the max, and saying that <, >, & (and U+2028/U+2029) count as six bytes each. Update the swagger @Description / body param text of postChatAutomationEvent.
  • coderd/exp_chat_automation_events_test.go InvalidBody (or a new subtest): a valid JSON body well under 256 KiB raw whose escaped form exceeds 256 KiB (for example " + 50,000 < + ") returns 413 with the escaping detail, adds no message to the target chat, and does not consume a single-use webhook.
  • docs/ai-coder/agents/automations.md: update the body limit sentence (line ~179), the 413 response row, and the Limits table row.

Validation (targeted; CI is the gate)

  • go build ./... for touched packages, go vet.
  • umask 022 && go test ./coderd/x/chatd/ -run 'Automation|ManageAutomations' -count=1, go test ./coderd/ -run 'ChatAutomation' -count=1, go test ./enterprise/coderd/ -run 'ChatAutomations' -count=1, go test ./codersdk/ -run 'DeploymentValues_Validate' -count=1, go test ./coderd/database/dbauthz/ -run 'ChatAutomation' -count=1.
  • make gen drift check for apidoc; make lint/go or golangci-lint on touched packages; pnpm run format-docs/lint for the docs file if available.

Acceptance criteria

  1. Tool update/enable and HTTP PATCH audit entries use the locked pre-update row as Old (CODAGT-1273 test passes and fails when reverted).
  2. A 403 DELETE leaves the automation enabled and its queue generation unchanged (test).
  3. dbauthz doc comments state the service-only rule.
  4. coder server refuses --chat-max-automations-per-owner < 1 at startup (Validate test).
  5. Prompt-edit behavior documented in code comments, swagger, and user docs; no behavior change.
  6. Webhook events whose escaped data exceeds 256 KiB get 413 with no side effects; docs updated.

Dogfooding (remote UAT via Coder Agents)

Build the branch in a dev deployment with the chat-automations experiment enabled for the test user and model config dd70d5aa-01bc-4ef6-a2db-70a4cd8cc876 ("Opus 5.5 (Bedrock)").

  1. Create a webhook automation (existing_chat). POST an event body {"x":"<" * 50000} (about 50 KB raw, about 300 KB escaped): expect 413 with the escaping detail; the chat gets no message. POST a small body: 202 and the message arrives.
  2. Single-use webhook: the oversized escaped body returns 413 and a following small body still returns 202 (not consumed).
  3. PATCH the prompt while an input is queued behind a busy chat: the queued input keeps the old text (documented behavior); disabling removes it.
  4. Start coder server with --chat-max-automations-per-owner=-1 (or the env var): startup fails with the flag named in the error; with 0 likewise.
  5. Do not query /api/v2/audit; audit Old is verified through Go tests only.
    Evidence: terminal transcripts / screenshots under .mux-uat/round-/.

Lane rules (copied from lane_rules.md)

  • implement-flow fully; at most 6 Codex rounds (4-6 blockers only). Base origin/main; never force-push; update by merging origin/main; never merge the PR or enable auto-merge.
  • No site/ changes in this lane. No migrations.
  • PR title type(scope): message, scope a real path containing every changed file (likely no scope or coderd... use a scope that contains codersdk, coderd, enterprise, docs: no scope). Body: "Closes CODAGT-1226", "Closes CODAGT-1273", "Part of CODAGT-1135", xum footer.
  • UAT must not query /api/v2/audit.
  • Commit and push with --no-verify; unset ANTHROPIC_* for repo tests; umask 022.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

UpdateAutomation now returns the row it read under the lock before the
change. The PATCH handler and the manage_automations tool record it as
the audit before-image, so a concurrent change between their read and
the lock is not attributed to them (CODAGT-1273, CODAGT-1226 item 3).
The doc comments also state that a prompt edit keeps already queued
messages.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
…s nothing

Also document in dbauthz that the chat automation update and secret
rotation queries authorize only RBAC update, and that the owner-only
rule lives in the chatd services (CODAGT-1226 items 1 and 2).

Signed-off-by: Thomas Kosiewski <tk@coder.com>
HTML escaping turns each <, >, &, U+2028, and U+2029 into six bytes, so
a body under the 256 KiB request limit could add a much larger message.
PublishAutomationWebhook now refuses escaped event data above 256 KiB
before admission, and the events endpoint returns 413 with the escaped
size. A refused delivery queues nothing and does not consume a
single-use webhook (CODAGT-1226 item 6). The docs also state that a
prompt edit keeps already queued messages (item 5).

Signed-off-by: Thomas Kosiewski <tk@coder.com>
…utomation

The function doc comment already states that a prompt edit keeps the
queue generation and leaves queued messages unchanged.

Signed-off-by: Thomas Kosiewski <tk@coder.com>
The branch made the PATCH handler record the row read under the update
lock as the audit Old, but only the manage_automations tool path had a
test for it. A store hook renames the automation between the handler's
read and the lock; the disable's audit diff must not contain the rename.
The test fails when the handler records its pre-read row.
@linear-code

linear-code Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-02T23:50:11.803501Z b4a3132 Manual request
🔒 Security Review ✅ Completed 2026-10-02T23:51:31.137560Z b4a3132 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: b4a3132374

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: b4a3132374

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ThomasK33

Copy link
Copy Markdown
Member Author

Superseded: this work is consolidated into the single Automations branch automations-bcpb. Closing on the owner's request.

@ThomasK33 ThomasK33 closed this Oct 6, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 6, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant