Cap enqueueing at max_requests_per_crawl and return a result from enqueue_links and add_requests #2306

max_requests_per_crawl only stops the crawler once enough requests have been processed (_stop_if_max_requests_count_exceeded in _basic_crawler.py). enqueue_links() and add_requests() keep writing every link to the queue, and both return None, so a handler cannot tell what was added or dropped.

JS v4 caps enqueueing at the remaining budget and reports what went over it.

Proposed solution

• Feed requests to the request manager in chunks capped by the remaining budget (max_requests_per_crawl minus handled and pending). Count only requests that were actually new, so duplicates do not use up the budget. Stop once the budget is spent.
• Report requests over the limit through on_skipped_request, with a new 'limit' skip reason.
• enqueue_links() and add_requests() return a result with the added requests and the requests over the limit.
• Requests that the storage returns as unprocessed are not retried by the crawler or RequestQueue; the backend is expected to retry transient failures itself. Where retries should live long term is still open in JS (Push retry/backoff logic into concrete storage backend implementations crawlee#4220).

This will be a breaking change.

JS references

• Budget-capped enqueueing: Don't enqueue over maxRequestsPerCrawl crawlee#2728, fix: Prevent accidental request dropping with maxRequestsPerCrawl crawlee#3531
• Return value and add_requests on the context: feat: split enqueueLinks into extractLinks + addRequests crawlee#4010
• No frontend retries: refactor: Remove unnecessary retries when enqueueing new requests crawlee#3962, fix(core): cap retries for unprocessed requests in addRequestsBatched crawlee#3765

Related: #2122 (ordering of limit and transform_request_function).

✍️ Drafted by Claude Code