Description

• Crawlee does not currently handle 429 Too Many Requests responses correctly.
• When a target server starts returning 429s, Crawlee does not slow down.
• Instead, due to the current autoscaled pool logic, Crawlee may actually scale concurrency up when responses get slower (because of less CPU work).
• This creates a "death spiral" - the slower the server, the faster Crawlee increases concurrency, which can quickly overwhelm small websites.

Proposed solution

• Detect 429 responses and implement proper backoff logic (reducing concurrency of autoscaled pool, cooldown period, ...).
• Ensure the autoscaled pool does not interpret slow responses or 429s as a signal to increase concurrency.
• Consider respecting Retry-After headers if present.

References

• This was originally discussed on Slack https://apify.slack.com/archives/CD0SF6KD4/p1756993901117969.