improvement(credentials): keep valid client-credential tokens through refresh outages and pace token retries by failure kind - #9925

Summary

Makes client-credential token requests more resilient, following the patterns used by MSAL, Entra's resilience guidance and botocore. The changes apply to the per-instance mint cache, the shared coordinated resolver (Shopify, Linear, Atlassian, Vanta) and Vanta itself.

• Still-valid tokens keep working. Tokens are refreshed early, inside the 5-minute window before expiry. If that refresh fails with an outage or throttling, the cached or stored token is served until 30 s before it expires. That margin is checked again after the failed refresh, so a slow attempt cannot carry the token past it. A refusal of the credentials still fails at once.

• Retry timing depends on the kind of failure (shared policy in lib/credentials/token-request-resilience.ts):

  • Refusal (bad or revoked credentials): replayed for 30 s, keyed by the secret's fingerprint.
  • Throttled (429, or 503 with Retry-After): the provider's Retry-After is honoured (seconds or HTTP-date), clamped to 1 s to 5 min. Without one, wait 1 min, doubling up to 5 min.
  • Outage (other 5xx, timeout, network error, malformed or unreadable response): 5 s, doubling to 30 s while it lasts. Before the backoff, one retry inside the mint after about 1 s, only if the failed attempt was quick.
  • Each kind keeps its own streak. A 429 after a run of outages starts its backoff at 1 min and does not inherit the outage count.

• Retry-After is carried on errors from every client-credential minter. TokenServiceAccountValidationError gets an optional retryAfterMs, read by throttledRetryAfterMs (429/503 only). It is filled in by:

  • the shared helpers: throwForProviderResponse, exchangeClientCredentialsGrant, Shopify and Vanta
  • every bespoke minter: Box, Zoom, DocuSign, NetSuite, Help Scout, Ramp, Jama, Zendesk, Salesforce, Zoho Desk
  • the CrowdStrike, SailPoint and ZoomInfo client errors, through a new optional field

  A replayed failure carries an optional replayForMs, which the per-instance layer honours exactly.

• Coordinated resolver:

  • The failure is remembered once for every worker in the existing row. Remembered failures and the still-valid fallback are served without taking the application lock.
  • A refusal whose code the row cannot store (the previous build reads it as provider_unavailable) gets an additive refusal: true flag, so a replay never serves the stale token.
  • A throttled failure gets an additive throttled: true flag, so the throttling streak continues across workers and stays separate from an outage streak.
  • Vanta asks for a token at most every 15 s while exchanges fail, because Vanta allows 5 tokens a minute and each new token revokes the previous one. That includes a 200 whose body cannot be read.

• Single-flight is unchanged: coalesceLocally per instance, the advisory lock across workers.

• The new fallback warning leaves out the credential id in selector privacy mode, like every other log in credential-service.ts.

Type of Change

• Bug fix
• Other: resilience improvement

Testing

• Unit tests in credential-service-cancellation.test.ts, errors.test.ts and client-credentials-grant.test.ts:
  • fallback while the token is valid, error once it has run out (including after a slow failed refresh), and no fallback after a refusal
  • the single in-mint retry
  • each replay window, the Retry-After cap and the backoff floors
  • a 429 after two outages waits about a minute, not the outage streak's longer wait
• A Retry-After test for each bespoke minter, in the minter test files (new helpscout.test.ts).
• Real-Postgres integration tests in coordinated-client-token.integration.ts and vanta-token.integration.ts, run one file at a time:
  • fallback across workers
  • the retry while holding the lock, shared by waiting resolutions
  • serving without taking the lock
  • backoff growth and reset after recovery
  • a throttled answer after an outage backs off from 1 min, and throttling that continues backs off longer for every worker
  • sweep retention
  • a refusal the row cannot store stays a refusal on replay
  • Vanta's 15 s pacing, including an unreadable 200 body
• Rolling-deploy fixtures pin both earlier failure decoders. A refusal memo is still exactly {code,status}. An outage memo adds only consecutiveFailures, a throttle memo adds consecutiveFailures and throttled, and an unstorable refusal adds only refusal; the old decoders ignore all three. The replay length rides on the existing expiresAt.
• Each guard was reverted and its tests went red.
• For reviewers: check the 5-minute Retry-After cap, and that the retry while holding the lock stays inside the 15 s lock_timeout (2 s budget + about 1.2 s pause + one 10 s attempt).
• Follow-up, not in this PR: the Microsoft app (Entra client-credentials) token cache has its own LRU and still has no stale fallback or failure memo.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)