Skip to content

feat: add start and message jitter to chat scaletest - #30264

Merged
cstyan merged 2 commits into
mainfrom
callum/chatd-01-scaletest-jitter
Oct 27, 2026
Merged

cstyan merged 2 commits into
mainfrom
callum/chatd-01-scaletest-jitter

Conversation

@cstyan

@cstyan cstyan commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Currently the chat scaletest synchronizes its load: runners launch together and the turn-start barrier releases all follow-up turns at once, so a run produces a thundering herd that spikes database CPU and then subsides as chats desynchronize, which hides sustainable steady-state throughput. This adds two opt-in knobs to coder exp scaletest chat: --start-jitter delays each runner's chat creation by a random duration in [0, start-jitter), and --message-jitter delays each follow-up send the same way. Both default to 0s (current behaviour) and honor context cancellation.

We should achieve the same theoretical throughput with this change but allow for a more realistic representation of user load against the system via chats.

Database impact: none by itself; this is harness-only. It is what made the lock-wait measurements behind the rest of this series repeatable, since the synchronized bursts otherwise dominate the run.

Combined with the other PRs: no merge-order interaction; it is independent of the coderd changes.

Series: #30264 harness jitter, #30265 transition metrics, #30266 lock-free stream reads, #30267 single-write transitions, #30268 commit-write result and RBAC cache, #30269 non-blocking acquisition, #30270 finish turn in the step commit.


Generated by Coder Agents on behalf of @cstyan.

Comment thread scaletest/chat/run.go
logger.Info(ctx, "starting chat runner")

// Stagger runner starts so the initial turns do not all begin at once.
if delay := jitterDelay(r.cfg.StartJitter); delay > 0 {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we excluding the jitter from latency metrics? Looks like this is within the timed period

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The jitter here, yes, as we call this immediately after:

r.resetConversation(time.Now(), markTurnStartReady)

createStartedAt := time.Now()

However the AI review did correctly identify that the message jitter needed to move to fix the same issue you suspected was present here for the start time jitter.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like you might have forgotten to push?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my bad, pushed now 👍

@ethanndickson

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-02T07:23:20.029701Z b2c28b1 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b2c28b10d4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scaletest/chat/run.go Outdated
Comment on lines +332 to +336
if delay := jitterDelay(r.cfg.MessageJitter); delay > 0 {
if err := sleepWithContext(ctx, delay); err != nil {
return xerrors.Errorf("wait message jitter for turn %d: %w", nextTurn, err)
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Start turn latency after applying message jitter

When --message-jitter is nonzero, r.turnStartTime has already been set before this sleep, so the random client-side delay is included in chat_time_to_running_seconds, chat_time_to_first_output_seconds, chat_time_to_terminal_status_seconds, and the corresponding duration logs. These metrics are intended to measure processing from when a turn starts, but enabling the new load-shaping option can inflate every follow-up latency by nearly the configured maximum and produce misleading experiment results. Apply the jitter before resetting turnStartTime, or reset it immediately after this sleep and before sending the request.

Useful? React with 👍 / 👎.

@cstyan
cstyan requested a review from ethanndickson October 5, 2026 22:30
The chat scaletest synchronizes load: runners launch together and the
turn-start barrier releases all follow-up turns at once, producing a
thundering herd that spikes database CPU and then subsides as chats
desynchronize. That makes it hard to measure sustainable steady-state
throughput.

Add two opt-in knobs to spread the load:

- --start-jitter: each runner waits a random duration in [0, start-jitter)
  before creating its chat, so initial turns do not all begin at once.
- --message-jitter: each follow-up turn send waits a random duration in
  [0, message-jitter), so follow-up turns spread across the run instead of
  releasing as a synchronized burst. Intended to be small (a few seconds).

Both default to 0s (current behavior). Sleeps honor context cancellation.
…trics

Apply message jitter before turnStartTime is reset so the client-side delay is not counted in follow-up turn latencies. Inline the jitter waits and cover both cancellation paths through the runner.
@cstyan
cstyan force-pushed the callum/chatd-01-scaletest-jitter branch from 1911bbd to 9f5ca64 Compare October 27, 2026 18:59
@cstyan
cstyan merged commit 4b6f2c1 into main Oct 27, 2026
27 checks passed
@cstyan
cstyan deleted the callum/chatd-01-scaletest-jitter branch October 27, 2026 19:34
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 27, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants