Skip to content

fix: Unify the request pipeline and retry policy across HTTP clients - #1006

Draft
vdusek wants to merge 3 commits into
masterfrom
unify-http-client-pipeline
Draft

fix: Unify the request pipeline and retry policy across HTTP clients#1006
vdusek wants to merge 3 commits into
masterfrom
unify-http-client-pipeline

Conversation

@vdusek

@vdusek vdusek commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Moves call, the retry loop, and the per-attempt request handling from ImpitHttpClient into HttpClient and HttpClientAsync, leaving Impit as a thin adapter over the transport hooks: send_request, is_retryable_transport_error, is_timeout_error, and close() / aclose() plus context managers. A custom client that overrides call keeps working unchanged and opts out of the shared pipeline.

The same pass fixes:

  • Transport errors a retry cannot fix (UnsupportedProtocol, LocalProtocolError, TooManyRedirects) fail on the first attempt instead of burning the whole backoff. Everything else in the impit.HTTPError tree stays retryable, including a bare HTTPError from a body that ends mid-chunk and a ProxyError from a proxy answering CONNECT with a transient status.
  • A failure while reading a streamed error response is classified and retried like a send failure, and the response is closed instead of being left open.
  • HttpResponse comes from typing_extensions, whose runtime protocol check looks attributes up statically, so isinstance(response, HttpResponse) no longer buffers or rejects an unread streaming body.
  • try_import only substitutes failed imports for the optional dependency it names, so an ImportError from the importing module or another dependency is no longer masked, and __all__ omits what a missing extra never provided.

Split out of #1004, which adds the built-in HTTPX client on top of this contract.

✍️ Drafted by Claude Code

Move `call`, the retry loop, and the per-attempt request handling from `ImpitHttpClient` into `HttpClient` and
`HttpClientAsync`, leaving Impit as a thin adapter over the `send_request`, error-classification, and lifecycle
hooks. A custom client that overrides `call` keeps working unchanged and opts out of the shared pipeline.

The same pass fixes:

- Transport errors a retry cannot fix (`UnsupportedProtocol`, `LocalProtocolError`, `TooManyRedirects`) fail on the
  first attempt instead of burning the whole backoff. Every other error in the `impit.HTTPError` tree stays
  retryable, including a bare `HTTPError` from a body that ends mid-chunk and a `ProxyError` from a proxy answering
  CONNECT with a transient status.
- A failure while reading a streamed error response is classified and retried like a send failure, and the response
  is closed instead of being left open.
- `HttpResponse` comes from `typing_extensions`, whose runtime protocol check looks attributes up statically, so
  `isinstance(response, HttpResponse)` no longer buffers or rejects an unread streaming body.
- `try_import` only substitutes failed imports for the optional dependency it names, so an `ImportError` from the
  importing module or another dependency is no longer masked, and `__all__` omits what a missing extra never
  provided.
@vdusek vdusek added adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. labels Aug 12, 2026
@vdusek vdusek self-assigned this Aug 12, 2026
@github-actions github-actions Bot added this to the 147th sprint - Tooling team milestone Aug 12, 2026
@github-actions github-actions Bot added the tested Temporary label used only programatically for some analytics. label Aug 12, 2026
@codecov

codecov Bot commented Aug 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.51244% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.91%. Comparing base (4281530) to head (3d4e037).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
src/apify_client/http_clients/_base.py 97.90% 3 Missing ⚠️
src/apify_client/_streamed_log.py 75.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1006      +/-   ##
==========================================
+ Coverage   94.86%   94.91%   +0.04%     
==========================================
  Files          58       58              
  Lines        5357     5406      +49     
==========================================
+ Hits         5082     5131      +49     
  Misses        275      275              
Flag Coverage Δ
integration 91.62% <68.65%> (-0.66%) ⬇️
unit 85.27% <97.51%> (+0.63%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants