Repository navigation
fix: classify quota errors as usage_limit instead of auth - #25676
Conversation
…nstead of auth Move quota/billing patterns out of authStrongPatterns into a dedicated usageLimitPatterns set. Map them to ChatErrorKindUsageLimit (which already existed) with a new priority rule that fires before auth. This fixes misleading 'Authentication failed' messages when the real issue is a provider quota/billing limit.
|
/coder-agents-review |
|
Review posted | Chat Review history
deep-review v0.5.0 | Round 5 | Last posted: Round 5, 6 findings (1 P2, 4 P3, 1 Nit), APPROVE. Review Finding inventoryFindings
Contested and acknowledgedCRF-2 (P3, signals.go:68) - "quota" pattern too broad
CRF-3 (P3, classify.go:200) - HTTP 402 not mapped to usage_limit
Round logRound 1Panel. 1 P2, 3 P3, 1 Nit posted. 1 Nit dropped. Reviewed against 4f1043a..74b77a1. Round 2Panel. CRF-1, CRF-5 verified fixed. CRF-3 contested, panel closed (5/7). CRF-4 fix incomplete (vacuous test). 0 new findings. Reviewed against 4f1043a..fa78792. Round 3BLOCKED. CRF-4 silent. Author claimed fix but no commit pushed. No review. Round 4Panel. CRF-4 verified fixed (dead code removed). 1 new P3 (CRF-7, test contradictory flag). Reviewed against 5d39c83..37a0500. Round 5Panel. CRF-7 verified fixed (vacuous test removed). 0 new findings. 0 open. Reviewed against be184a0..ee19a9d. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
Clean, well-scoped reclassification. The pattern move, priority insertion, message templates, and test coverage are all correct. The real-world OpenAI error test case is a strong anchor. One concern about frontend semantic overload, two pattern-breadth observations, one test coverage gap, one naming nit.
1 P2, 3 P3, 1 Nit.
Quote from Pariston: "I tried to build a case against this PR and couldn't find a substantive gap."
🤖 This review was automatically generated with Coder Agents.
| } | ||
| authWeakPatterns = []string{"forbidden"} | ||
| usageLimitPatterns = []string{ | ||
| "quota", |
There was a problem hiding this comment.
P3 [CRF-2] The bare "quota" pattern matches both billing-quota exhaustion (non-retryable) and per-minute rate-limit quota messages (retryable), suppressing retries for providers that use "quota" in rate-limit errors.
Google Gemini returns rate limits like "status 429: Resource has been exhausted (e.g. check quota)." Azure returns "exceeded token rate limit ... quota remaining: 0". Both contain "quota" and HTTP 429. With usageLimitMatch at priority 2 and rateLimitMatch at priority 5, the 429 is swallowed. The error classifies as usage_limit (non-retryable) instead of rate_limit (retryable).
Similarly, "billing" could match "rate limited: upgrade your billing plan for higher rate limits", producing the same non-retryable misclassification.
This was already broken before the PR (these patterns were in authStrongPatterns with the same non-retryable behavior). The PR improves the user-facing label but had the opportunity to disambiguate. More specific patterns like "insufficient_quota", "billing_error", or "exceeded.*billing" would reduce false positives.
"The ripeness grows with Gemini adoption. At low traffic nobody notices. At scale, the missing retries on Google 429s become visible." (Hisoka P3, Mafuuu P3)
🤖
There was a problem hiding this comment.
Acknowledged. The 429-with-"quota" scenario (Gemini's "check quota", Azure's "quota remaining: 0") is a valid concern, but as the comment notes, this is pre-existing behavior, not a regression from this PR. These patterns were in authStrongPatterns before (also non-retryable, also higher priority than rateLimitMatch).
I investigated the lift to fix this properly:
- The classifier input is exclusively
err.Error()from provider SDK calls, so the general false-positive risk from non-error content is negligible. - The real risk is exactly what the comment describes: rate-limit responses that incidentally contain "quota" text getting classified as non-retryable
usage_limitinstead of retryablerate_limit. - The cleanest fix would be a 429 exclusion:
usageLimitMatch := containsAny(lower, usageLimitPatterns...) && statusCode != 429. The rationale: true billing quota exhaustion comes with 401/403/200-with-error bodies, not 429. A 429 with "quota" text is almost certainly a rate limit. - An alternative is more specific patterns ("insufficient_quota", "exceeded.*billing"), but this risks missing future provider phrasings.
Deferring to a follow-up since it's pre-existing and the scope here is specifically the auth misclassification from CODAGT-484.
🤖 Generated with Coder Agents
There was a problem hiding this comment.
CODAGT-495 for follow-up.
| retryableHTTP2StreamReset, hasHTTP2StreamReset := classifyHTTP2StreamReset(err) | ||
| deadline := errors.Is(err, context.DeadlineExceeded) || strings.Contains(lower, "context deadline exceeded") | ||
| overloadedMatch := statusCode == 529 || containsAny(lower, overloadedPatterns...) | ||
| usageLimitMatch := containsAny(lower, usageLimitPatterns...) |
There was a problem hiding this comment.
P3 [CRF-3] HTTP 402 (Payment Required) is not mapped to usage_limit, leaving a status-code gap unique to this kind.
Every other error kind pairs text patterns with a status code trigger: 401/403 for auth, 429 for rate-limit, 408/502/503/504 for timeout, 500 for generic, 529 for overloaded. The new usageLimitMatch relies on text patterns only. If a provider returns a bare 402 without quota/billing text, it falls through to generic.
Practical likelihood is low (major providers use 429 for quota errors with descriptive body text), but the sibling pattern is clear. Fix: usageLimitMatch := statusCode == 402 || containsAny(lower, usageLimitPatterns...). (Chopper P3)
🤖
There was a problem hiding this comment.
Disagree. 402 is a non-standard status code.
There was a problem hiding this comment.
Panel closed this finding (5/7 accept).
Razor found that the original premise is inaccurate: configMatch also relies on text patterns only, with no status code trigger. The claim "every other kind pairs text patterns with a status code" does not hold. Hisoka traced that "payment required" in usageLimitPatterns serves as a de facto handler for any 402 that includes the standard reason phrase. No known AI provider returns bare HTTP 402 without descriptive body text.
The gap is theoretical with no real-world occurrence. Closing.
🤖
Provider quota errors (with provider field set) now render via ChatStatusCallout instead of the Coder analytics link, since the issue is on the provider's billing page.
|
/coder-agents-review |
There was a problem hiding this comment.
Round 2. CRF-1 fix verified by all reviewers: the !usageLimitStatus.provider guard correctly separates Coder vs provider quota rendering. CRF-5 fix verified. CRF-2 deferred to CODAGT-495.
CRF-3 (HTTP 402 gap): closed by panel vote (5/7). Razor found that configMatch also has no status code trigger, undermining the "every other kind" premise. Hisoka traced that "payment required" in usageLimitPatterns serves as a de facto 402 handler. No known provider returns bare 402 without descriptive text.
One remaining item: CRF-4 fix is incomplete. The new test (TestStreamRetryPayloadUsageLimitMessage) does not exercise the retryMessage UsageLimit switch case it claims to cover.
0 new findings. 1 open P3.
Quote from Razor on CRF-3: "configPatterns also has no status code trigger."
🤖 This review was automatically generated with Coder Agents.
There was a problem hiding this comment.
Pull request overview
This PR fixes misclassification of provider quota/billing failures by reclassifying them as usage_limit (instead of auth) and updates the UI to avoid showing Coder’s “View Usage” CTA for provider-originated quota errors.
Changes:
- Backend: moved quota/billing text matching into
usageLimitPatternsand prioritizedusage_limitclassification ahead of auth. - Backend: added usage-limit specific terminal/retry messaging and updated auth messaging to remove billing references.
- Frontend: gated the “View Usage” link so it only renders for Coder spend-cap usage-limit errors (no provider), and added a Storybook coverage case.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| site/src/pages/AgentsPage/components/ChatConversation/LiveStreamTail.tsx | Only show “View Usage” CTA for usage-limit errors without a provider (Coder spend cap), otherwise fall back to ChatStatusCallout. |
| site/src/pages/AgentsPage/components/ChatConversation/LiveStreamTail.stories.tsx | Adds a Storybook scenario validating provider quota errors do not render the “View Usage” link. |
| coderd/x/chatd/chaterror/signals.go | Splits quota/billing patterns out of auth patterns into usageLimitPatterns. |
| coderd/x/chatd/chaterror/classify.go | Adds usageLimitMatch and inserts a higher-priority rule mapping to ChatErrorKindUsageLimit. |
| coderd/x/chatd/chaterror/classify_test.go | Updates expectations for auth message text and adds coverage ensuring usage-limit beats auth when quota/billing signals are present. |
| coderd/x/chatd/chaterror/message.go | Adds usage_limit terminal and retry copy; simplifies auth terminal copy. |
| coderd/x/chatd/chaterror/message_test.go | Adds terminal-message test coverage for usage_limit. |
| coderd/x/chatd/chaterror/payload_test.go | Adds a stream-retry payload test for usage_limit output. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
/coder-agents-review |
There was a problem hiding this comment.
Round 3 blocked. CRF-4 remains unaddressed.
The R2 review identified that TestStreamRetryPayloadUsageLimitMessage does not exercise the retryMessage UsageLimit switch case (message.go:93-100). The reply stated "Removed the dead code and kept the test," and the thread was resolved. However, no commit was pushed between rounds: git diff fa78792..eeb1e94abd2c -- coderd/x/chatd/chaterror/message.go is empty. The UsageLimit case remains at message.go:93. The only commit is a merge from main.
Further review is blocked until the author either pushes the removal commit or reopens the thread with an alternative resolution.
🤖 This review was automatically generated with Coder Agents.
> 🤖 This PR was written by Coder Agents on behalf of Jake Howell. Linear: [DEVEX-355](https://linear.app/coder/issue/DEVEX-355) Third PR in a 5-PR stack splitting #25328. Adds the component-level pieces used by the provider management pages landing in the next PR of the stack. - `ProviderForm` + `CredentialField` + a provider type-to-form mapping for reading and editing the per-type credential and config fields, with the form API map covered by unit tests. - `ProviderIcon` resolves the bundled per-provider SVG icons and falls back to a building glyph for unknown types. - `ProviderRow` renders a single provider entry for the list view. - `useUnsavedChangesPrompt` hook intercepts unsaved-form navigation. - Storybook stories for `ProviderForm`, `ProviderIcon`, and `ProviderRow` exercise each provider type and form state and consume the mock providers from PR 2. Stories now consume `MockAIProviderOpenAI` / `Anthropic` / `Bedrock` so their per-mock `@lintignore` tags are removed; the `MockAIProviders` aggregate and the `addableProviderTypes` / `aiProviders` query modules keep their exclusions for the page stories in the next PR. <details> <summary>Stack</summary> 1. #25579 jakehwll/DEVEX-355/01-primitives, primitives 2. #25580 jakehwll/DEVEX-355/02-api, API client and query layer 3. **jakehwll/DEVEX-355/03-components, provider form components (this PR)** 4. jakehwll/DEVEX-355/04-pages, pages and routes 5. jakehwll/DEVEX-355/05-section, section reshuffle Replaces #25328 once the stack lands. </details>
> 🤖 This PR was written by Coder Agents on behalf of Jake Howell. Linear: [DEVEX-355](https://linear.app/coder/issue/DEVEX-355) Fourth PR in a 5-PR stack splitting #25328. Wires the new `/ai/settings` provider management UI. - `AISettingsLayout` hosts the section under `/ai/settings` with a sidebar outlet. - `AISettingsSidebar(View)` shows a single "Providers" nav entry. The remaining sidebar entries arrive with the broader AI settings section reshuffle in the next PR. - `ProvidersPage` lists configured AI providers via the queries added in PR 2. - `AddProviderPage` walks through provider-type selection and form submission, with type-specific credential fields. - `UpdateProviderPage` edits an existing provider with the same form components. - Storybook stories cover each view's loading, empty, populated, error, and form states using the mock providers from `testHelpers/entities.ts`. - `router.tsx` mounts the new `/ai/settings` layout with index, `add`, and `:providerId` child routes. The `governance` child route lands together with the dashboard navigation changes in the next PR. Removes the now-unused knip ignore entries for `src/api/queries/aiProviders.ts` and `src/pages/AISettingsPage/ProvidersPage/components/addableProviderTypes.ts`, and drops the matching `@lintignore` tags on `getProviderIcon` and `MockAIProviders` since the pages and page stories now consume them. <details> <summary>Stack</summary> 1. #25579 jakehwll/DEVEX-355/01-primitives, primitives 2. #25580 jakehwll/DEVEX-355/02-api, API client and query layer 3. #25581 jakehwll/DEVEX-355/03-components, provider form components 4. **jakehwll/DEVEX-355/04-pages, pages and routes (this PR)** 5. jakehwll/DEVEX-355/05-section, section reshuffle Replaces #25328 once the stack lands. </details>
> 🤖 This PR was written by Coder Agents on behalf of Jake Howell. Linear: [DEVEX-355](https://linear.app/coder/issue/DEVEX-355) Fifth and final PR in a 5-PR stack splitting #25328. Surfaces the AI settings section in the dashboard chrome and moves the existing AI Governance page out of `/deployment`. - `Navbar` / `NavbarView` / `DeploymentDropdown` gain a `canViewAISettings` prop sourced from the `viewAnyAIProvider` permission added in PR 2. The deployment dropdown gets a new AI entry that links to `/ai/settings`. - `DeploymentSidebarView` drops the AI-related entries that now live under `/ai/settings`. - `AISettingsSidebarView` expands to include AI Governance and a cross-section link to Manage Coder Agents. - `router.tsx` removes the `/deployment/ai-governance` route and mounts the matching `/ai/settings/governance` child route under the new AI settings layout. - `ChatsSidebar` settings panel repoints the Providers link from `/deployment/ai-providers` to `/ai/settings`. <details> <summary>Stack</summary> 1. #25579 jakehwll/DEVEX-355/01-primitives, primitives 2. #25580 jakehwll/DEVEX-355/02-api, API client and query layer 3. #25581 jakehwll/DEVEX-355/03-components, provider form components 4. #25583 jakehwll/DEVEX-355/04-pages, pages and routes 5. **jakehwll/DEVEX-355/05-section, section reshuffle (this PR)** Replaces #25328 once the stack lands. </details>
Previously we were only extracting the API when _not_ delegating auth; this is incorrect. We need to extract the key _always_ when BYOK is intended. --------- Signed-off-by: Danny Kopping <danny@coder.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `multi-select` form type description in the dynamic parameters docs incorrectly stated it renders checkboxes. The actual UI is a searchable dropdown combobox (`MultiSelectCombobox`) with selected items shown as removable chips. > This PR was authored by Coder Agents on behalf of @uzair-coder07.
|
/coder-agents-review |
There was a problem hiding this comment.
Round 4. CRF-4 fix verified: dead retryMessage UsageLimit case removed in 37a0500. All prior findings now resolved, deferred, or closed.
One new finding on the surviving test.
1 new P3. 0 open from prior rounds.
Quote from Mafuuu: "Removing dead code is better than testing dead code."
🤖 This review was automatically generated with Coder Agents.
Documentation CheckStatus: ✅ CompletePR #25676 has been updated with new commits. Documentation review shows: Current Documentation Changes:
Updates Included:
Previous vs. Current:
Assessment: Automated review via Coder Agents |
|
/coder-agents-review |
1 similar comment
|
/coder-agents-review |
There was a problem hiding this comment.
Round 5. All findings resolved. CRF-7 (vacuous test) verified removed by all reviewers. No new findings from Netero or the panel.
Final tally: 7 findings across 5 rounds. 4 fixed and verified (CRF-1, CRF-4, CRF-5, CRF-7). 1 deferred with ticket (CRF-2, CODAGT-495). 1 panel-closed (CRF-3, 5/7). 1 dropped (CRF-6).
Clean, well-scoped reclassification with a correct frontend guard. The real-world OpenAI error test and the ProviderQuotaExceeded story anchor the change.
🤖 This review was automatically generated with Coder Agents.
Fixes CODAGT-484.
Problem
The
authStrongPatternsinchaterror/signals.goincluded "quota", "billing", "insufficient_quota", and "payment required". This caused provider quota errors (e.g. OpenAI'sinsufficient_quota) to be classified asChatErrorKindAuth, showing users a misleading "Authentication failed" message.Additionally, the frontend unconditionally rendered a "View Usage" link to
/agents/analyticsfor allusage_limiterrors, which is incorrect for provider-originated quota errors (the issue is on the provider's billing page, not Coder's).Fix
Backend: Moved quota/billing patterns into a dedicated
usageLimitPatternsset and mapped them toChatErrorKindUsageLimit. The new rule fires before auth in the priority chain so quota text wins regardless of HTTP status code.Frontend: The "View Usage" button now only renders when
!usageLimitStatus.provider. Provider quota errors (which always haveproviderset from stream classification) fall through toChatStatusCalloutinstead.User-facing changes:
Implementation details
authStrongPatternsusageLimitPatternsslice with those patternsusageLimitMatchsignal and rule between overloaded and authStrong in priorityChatErrorKindUsageLimit!usageLimitStatus.providerguard on the "View Usage" AlertTestClassify_UsageLimitBeatsAuthwith 5 cases including real production OpenAI errorProviderQuotaExceededstory asserting no "View Usage" link and correctChatStatusCalloutrendering