Migrate from httpx to httpx2 #4427

httpx sees little maintenance, and the ecosystem is moving to httpx2, Pydantic's maintained fork of httpx 0.28.1. openai 3.x already requires httpx2, starlette's TestClient prefers it, and opentelemetry-instrumentation-httpx supports it. dstack already uses httpx2 for gateway auth, for its pool fixes (#4338), while the shared proxy code still uses httpx, so the gateway runs both libraries and their exception classes don't mix.

We currently use three HTTP clients:

• requests (+ requests-unixsocket): APIClient (CLI and Python API), backend API clients, runner/shim client, docker registry, REST plugin.
• httpx: server and gateway proxy, gateway client, probes, router worker sync, job replica client, fluentbit, vastai.
• httpx2: gateway auth and gateway server client.

Proposal

1. Replace httpx with httpx2 everywhere. This is mostly mechanical, since httpx2 keeps the httpx API. It includes switching the OTel instrumentor to HTTPX2ClientInstrumentor, which HTTPXClientInstrumentor doesn't cover:

2. Optionally, move our own requests code to httpx2 too, so we have one client for sync and async code, one TLS and proxy model, one exception hierarchy, one OTel instrumentor and one mocking approach. It would also drop requests-unixsocket, since httpx2 supports unix sockets natively. requests itself stays installed through third-party dependencies (gpuhunt, docker, kubernetes, google/azure SDKs, etc.).

TLS: certifi vs the system trust store

httpx2 no longer uses certifi. For verify=True (2.15.1+):

• SSL_CERT_FILE / SSL_CERT_DIR replace the default store on all OSes.
• macOS and Windows use the OS verifier via truststore (Keychain, Windows store).
• Linux loads the system CA files (OpenSSL's compiled-in paths, then the common distro bundle locations) into a stdlib SSLContext once.
• With no system CA bundle, all HTTPS fails with CERTIFICATE_VERIFY_FAILED. There is no certifi fallback.

User impact:

• Corporate CAs and TLS-inspecting proxies installed in the OS store start working without configuration.
• Minimal images without ca-certificates break. Our server image (python:3.11-slim) is fine, but custom images may not be.
• macOS verifies more strictly than OpenSSL, so some private or self-signed server certs may be rejected. See Getting SSLError(SSLCertVerificationError('“PRIVATE.CERT” certificate is not standards compliant')) after upgrading to pip 24.2 pypa/pip#12995.
• REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE stop applying to anything moved off requests. SSL_CERT_FILE is the replacement.

Options:

• A. Use the httpx2 default as is.
• B. Pass our own context that matches the httpx2 default but falls back to certifi when the system store is empty, similar to where pip is heading (Switch to using truststore only by default (currently mixed certifi + truststore) pypa/pip#14374). This keeps slim containers working, at the cost of trusting a bundle that only updates with the certifi package.

Python 3.10

httpx2 2.15+ requires Python 3.11, and on 3.10 the resolver stays on 2.14.0, which gets no further fixes. The migration should come with dropping Python 3.10, which depends on #4423.

Behaviour changes when moving off requests (optional part)

• Timeouts: httpx2 defaults to a 5s timeout, while requests has none. APIClient sets no timeout today, so long calls would start failing unless we set one explicitly.

• Redirects: httpx2 doesn't follow them by default. APIClient treats any status below 400 as success, so a 3xx needs explicit handling.
• Exceptions: requests.ConnectionError becomes httpx2.ConnectError / TransportError, and raise_for_status() also raises on 3xx. Python API users catching requests exceptions around dstack calls would break.
• TLS errors and retries: httpcore2 doesn't map every TLS error, so ssl.SSLError can escape the retry loop. See ssl.SSLError during a request is no longer retried since 3.14.1 openai/openai-python#3969.
• API differences: data=<str> becomes content=, .ok becomes .is_success, iter_content becomes iter_bytes.
• Unix sockets: http+unix:// URLs become HTTPTransport(uds=...) with a regular base URL.
• Tests: requests_mock (300+ usages) moves to httpx2.MockTransport or an httpx2 mocking plugin.

Plan

• Drop Python 3.10 ([Feature]: Support Python version upgrade for gateways and migrate from Python 3.10 #4423)
• Decide on TLS policy (A or B)
• httpx → httpx2 in server, proxy, gateway, fluentbit, vastai; switch OTel instrumentor
• (optional) APIClient and CLI → httpx2, with explicit timeouts, redirect handling, and release notes on REQUESTS_CA_BUNDLE → SSL_CERT_FILE
• (optional) Runner/shim client and backend API clients → httpx2
• Test the CLI on macOS with a private-CA server, on uv-managed Python, and in a minimal container