Repository navigation
feat: add start and message jitter to chat scaletest - #30264
Conversation
| logger.Info(ctx, "starting chat runner") | ||
|
|
||
| // Stagger runner starts so the initial turns do not all begin at once. | ||
| if delay := jitterDelay(r.cfg.StartJitter); delay > 0 { |
There was a problem hiding this comment.
Are we excluding the jitter from latency metrics? Looks like this is within the timed period
There was a problem hiding this comment.
The jitter here, yes, as we call this immediately after:
r.resetConversation(time.Now(), markTurnStartReady)
createStartedAt := time.Now()
However the AI review did correctly identify that the message jitter needed to move to fix the same issue you suspected was present here for the start time jitter.
There was a problem hiding this comment.
looks like you might have forgotten to push?
There was a problem hiding this comment.
my bad, pushed now 👍
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b2c28b10d4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if delay := jitterDelay(r.cfg.MessageJitter); delay > 0 { | ||
| if err := sleepWithContext(ctx, delay); err != nil { | ||
| return xerrors.Errorf("wait message jitter for turn %d: %w", nextTurn, err) | ||
| } | ||
| } |
There was a problem hiding this comment.
Start turn latency after applying message jitter
When --message-jitter is nonzero, r.turnStartTime has already been set before this sleep, so the random client-side delay is included in chat_time_to_running_seconds, chat_time_to_first_output_seconds, chat_time_to_terminal_status_seconds, and the corresponding duration logs. These metrics are intended to measure processing from when a turn starts, but enabling the new load-shaping option can inflate every follow-up latency by nearly the configured maximum and produce misleading experiment results. Apply the jitter before resetting turnStartTime, or reset it immediately after this sleep and before sending the request.
Useful? React with 👍 / 👎.
The chat scaletest synchronizes load: runners launch together and the turn-start barrier releases all follow-up turns at once, producing a thundering herd that spikes database CPU and then subsides as chats desynchronize. That makes it hard to measure sustainable steady-state throughput. Add two opt-in knobs to spread the load: - --start-jitter: each runner waits a random duration in [0, start-jitter) before creating its chat, so initial turns do not all begin at once. - --message-jitter: each follow-up turn send waits a random duration in [0, message-jitter), so follow-up turns spread across the run instead of releasing as a synchronized burst. Intended to be small (a few seconds). Both default to 0s (current behavior). Sleeps honor context cancellation.
…trics Apply message jitter before turnStartTime is reset so the client-side delay is not counted in follow-up turn latencies. Inline the jitter waits and cover both cancellation paths through the runner.
1911bbd to
9f5ca64
Compare
Currently the chat scaletest synchronizes its load: runners launch together and the turn-start barrier releases all follow-up turns at once, so a run produces a thundering herd that spikes database CPU and then subsides as chats desynchronize, which hides sustainable steady-state throughput. This adds two opt-in knobs to
coder exp scaletest chat:--start-jitterdelays each runner's chat creation by a random duration in[0, start-jitter), and--message-jitterdelays each follow-up send the same way. Both default to0s(current behaviour) and honor context cancellation.We should achieve the same theoretical throughput with this change but allow for a more realistic representation of user load against the system via chats.
Database impact: none by itself; this is harness-only. It is what made the lock-wait measurements behind the rest of this series repeatable, since the synchronized bursts otherwise dominate the run.
Combined with the other PRs: no merge-order interaction; it is independent of the coderd changes.
Series: #30264 harness jitter, #30265 transition metrics, #30266 lock-free stream reads, #30267 single-write transitions, #30268 commit-write result and RBAC cache, #30269 non-blocking acquisition, #30270 finish turn in the step commit.
Generated by Coder Agents on behalf of @cstyan.