refactor(client): issue requests through a generated transport - #27
Draft
chandrasekharan-zipstack wants to merge 26 commits into
Draft
refactor(client): issue requests through a generated transport#27chandrasekharan-zipstack wants to merge 26 commits into
chandrasekharan-zipstack wants to merge 26 commits into
Conversation
The HTTP layer is now generated from the committed OpenAPI spec rather than hand-written, so URLs, query names and multipart encoding follow the spec instead of being restated here. tools/gen_sdk.sh regenerates it with a pinned generator; the tree is committed but never hand-edited. The public surface is unchanged on purpose: same constructor, same return dicts, same exceptions. What was deliberately kept rather than rewritten: - The retry policy, verbatim. Attempt counts, Retry-After on 429, exponential jitter, file rewinding, and the sync/async POST distinction are the contract, and nothing about the transport should restate them. - Transport failures are translated to their `requests` equivalents inside the retried call, not around it, so the retry policy still sees the exception types it is configured to retry. `requests` stays a dependency for those classes because callers catch them by name. - Response fields are read from the JSON body, never from a generated response model: a model exists only for the statuses the spec declares, and error bodies are typed too loosely to read. Only the parameters this client sets are sent. The generated builders write every declared default into a request, and sending a default is not the same as omitting it — it pins a value the server would otherwise choose, and the two diverge as soon as the server's default changes. No transport timeout is configured, as before: api_timeout selects a backend execution mode and is not a socket timeout.
The transport changed; the published behaviour must not. These tests compare against the 1.5.3 client vendored under tests/baseline: constructor and method signatures via AST, the request that goes out, the exceptions that come back, and the exact dict each method returns — the last by running both clients over the same responses. Also stop sending the generated fixed multipart boundary. An uploaded file containing those bytes would corrupt the encoding, so the header is dropped and the transport picks a random boundary, as the previous client did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
requests.ConnectTimeout is both a ConnectionError and a Timeout. Mapping httpx.ConnectTimeout to a plain Timeout — which is all httpx's own hierarchy implies — stops every caller that catches the connection family from catching a connect timeout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
httpx.ReadTimeout was landing in the TimeoutException catch-all and coming back out as requests.Timeout. Callers that catch requests.ReadTimeout by name stopped matching. The translation table test used pytest.raises, which is subclass-tolerant and passed either way; it now asserts the exact class. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
Contributor
Author
|
Backported a fix found by the sibling client's live run: The live round trip for this client is still outstanding — it needs staging credentials. |
The spec is now produced and committed by the backend that serves these endpoints, so this repo tracks that file instead of a copy maintained elsewhere. Regenerating picks up its root `tags` array; the generated tree is otherwise unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
…_file The deployment accepts twelve request parameters; the client could only send two, and only by way of the constructor. The rest had no argument to travel through, so callers that need a tag, an LLM profile or a HITL queue cannot reach them at all. They are added as keyword-only arguments named exactly as the API names them. Every one defaults to unset and an unset parameter is not sent, so the server still picks its own default and the request is byte-for-byte unchanged for every existing call shape. `timeout` and `include_metadata` fall back to the constructor values when not passed, and a `timeout` passed per request selects the execution mode for that request.
The status endpoint takes include_metadata, include_metrics and include_extracted_text; the client could send only the first, and only via the constructor, so a caller wanting metrics on one poll had nowhere to ask. They are added as keyword-only arguments named exactly as the API names them, each defaulting to unset. An unset parameter is not sent, so the query string is unchanged for every existing call shape and the server still picks its own default. execution_id stays out: it is read from the endpoint URL the server handed back.
The backend's spec now declares the deployment key as a bearer scheme on each operation, describes the error statuses a caller has to branch on, and no longer publishes the MCP endpoints or a request field the deployment does not accept. With no operation left outside the facade, the coverage check compares the declared set whole. Excusing an operation by name kept passing after the spec stopped declaring it, and a green run said nothing about whether the exception still described anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
A schema it cannot parse is downgraded to a warning: the endpoint or response it belongs to is dropped, the rest is written, and the run exits 0. Nothing downstream can tell that from a client that never had the operation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
httpx renders a bool as `true`; urlencoding a Python bool gives `True`, which is what went out before. The service reads both, so nothing breaks either way -- but a caller diffing traffic across the upgrade should see no change, and this is the only field that moved. The parity test could not see it: it stringified our parameters before comparing them with the published ones, which turned `True` into `True` on both sides. It now compares what the transport will actually send. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
Rebuilding the URL from the spec's path template dropped any prefix the deployment is served under -- an ingress route, an on-prem reverse proxy -- because no route template can carry one. The released client posted to the URL verbatim. The parity test could not see this: it compared against a deployment URL with no prefix, so both sides agreed. It now runs over a prefixed URL, a slash-less one and a mixed-case one, and compares against the released client's URL rather than a constant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
Three httpx failures reached callers as httpx classes, which nothing downstream catches: a redirect loop, an undecodable body, and any future RequestError that is not a TransportError. Two more were translated to a class the released client never raised for them -- requests had no write or pool timeout, and both surfaced as ConnectionError. The class chosen here also decides what gets retried, so an unsendable URL is now MissingSchema rather than a ConnectionError the retry loop would attempt four more times. The parametrised list of failures is replaced by a walk of httpx's own exception tree: a hand-written list is exactly as complete as the day it was written. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
The transport adds headers no client object holds, so the only place the two can be compared is a socket. Both clients now run against a loopback server and their request heads are diffed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
The generated tree is committed, so an edit inside it reviews like any other change and then vanishes on the next regeneration -- as does a spec change nobody ran the generator over. Regenerating in CI and diffing is what notices either one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
Nothing bounds a stalled connection: the transport is untimed, and api_timeout cannot serve as one because the backend reads it as an execution mode -- 0 selects async, and negative values are accepted. A run that stalled for roughly 985 seconds is what this is for. Keyword-only and unset by default, so no released call shape changes and the default behaviour stays exactly what it was. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
The released client concatenated its base URL with whatever the server handed back, so only a root-relative endpoint worked -- an absolute one became `https://hosthttps://host/...`. Reading the execution id out and rebuilding the route from the spec means all three spellings resolve to the same request, and this is what says so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
The baseline was pinned by a version string in its own header comment, which an edit to the file can rewrite as easily as the code below it. Every parity test compares against this file, so a weakened baseline weakens all of them silently.
The generated transport is written against one httpx minor series; an upgrade needs a regeneration and a test run, not a resolver decision taken at install time in someone else's environment.
…params A multipart form field carries no null, so a caller passing None got the literal string "None" sent as a tag, an LLM profile id or a queue name for the service to resolve. These are overrides the service defaults when absent, and absent is what None asks for.
Each of these stated what the line below it does, or described a prior state that is no longer there to check against. Keep the reason, drop the narration.
The status URL was built from scheme and host alone, so a deployment served under a path prefix could execute -- the execute call sends the caller's URL verbatim -- and then never poll, losing the result of a paid execution. Documented divergence: the previous release has the same gap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
The CLI that owns the name depends on this package, so the two always share an environment and the entry point collides on every install. `python -m unstract.clone` is unchanged, and the CLI offers the same command. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
A 3xx was returned as if it were the answer, which a poll loop reads as a finished execution with no status; the previous transport followed redirects on both verbs. A status endpoint carrying no execution id now fails instead of polling for a blank one. InvalidURL joins the translation table, and the docstring names the two httpx families that stay outside it. Adds the multi-file upload comparison the parity suite never had, and lets the drift gate see a file the generator newly creates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
The status endpoint is the service's instruction for reaching one execution. Only its execution_id was being read: any other parameter on it -- a region hint, a signature -- was dropped from every poll, and a deployment URL that does not carry the spec route was polled at a path rebuilt from that route rather than at the endpoint itself. Both end the same way, at a paid execution whose result is never collected. Remaining parameters are now forwarded, and where no path prefix can be derived the endpoint is used as it came. Also pins the exception classes a malformed api_url raises. They differ from the released client's for two inputs; the divergence is deliberate and the test says so.
MissingSchema is a ValueError, so the row that exists to record released parity could not tell the two apart; it asserts the exact class now. The README and a release-notes draft carry the differences a caller can observe, including the console script this branch removed.
It shells out to ruff for post-processing. Finding none, it warns and exits 0, and the warning gate reports that as a spec it could not parse -- a clean regeneration on a runner without a global ruff failed with a message pointing at the wrong thing entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
APIDeploymentsClientnow issues its requests through a transport generated from the API's OpenAPI spec, instead of hand-builtrequestscalls.specs/docstudio-oss.json(generated by the backend that serves these endpoints) +tools/gen_sdk.shregeneratesrc/unstract/api_deployments/sdk_docstudio/with a pinned generator. The generated tree is committed, markedlinguist-generated, stamped DO-NOT-EDIT, and excluded from lint — regeneration overwrites it wholesale, so fixes belong inclient.pyor in the spec.requestsequivalents inside the retried callable, so callers that catchConnectionError/Timeoutstill work and transport-error retry still counts.Behaviour changes
Two, both deliberate:
structure_fileare now closed after the request. The previous client leaked them.test_status_url_matches_the_released_clientpins that the resulting request is identical.Testing
tests/test_compat.pycompares this client against 1.5.3, vendored attests/baseline/client_1_5_3.pyand refreshed viatools/refresh_baseline.sh:api_timeout(a backend execution mode, not a socket timeout) never reaches the transportrequestsexception translation, asserted on the exact class (type(caught.value) is expected), not with a barepytest.raises— that accepts a superclass, and it hid two real mismatches: a connect timeout arriving as a plainTimeout, and a read timeout doing the same324 tests pass. The existing 745-line retry suite is unchanged apart from its patch target.
A live round trip against a deployment is still outstanding.
Added after review started
c291e36adds the deployment's remaining request parameters tostructure_file, as a separate commit so it reads as a delta rather than a rewrite.The deployment accepts twelve request parameters. The client could send two, and only through the constructor — the other ten had no argument to travel through. They are now keyword-only arguments named exactly as the API names them, each defaulting to unset, and an unset parameter is not sent. So the request is byte-for-byte unchanged for every existing call shape, and the server still picks its own defaults.
timeoutandinclude_metadatafall back to the constructor values; atimeoutpassed per request selects the execution mode for that request.The signature-parity test now exempts keyword-only parameters, since none of them is reachable from a released call. New tests cover the spec-exact names, keyword-only enforcement, that falsy values (
0,False,"") are sent rather than filtered out as absent, and that a per-requesttimeoutmoves the sync/async retry decision with it.ed89066does the same forcheck_execution_status, which could sendinclude_metadataand only from the constructor: it gainsinclude_metadata,include_metricsandinclude_extracted_textas keyword-only arguments, unset by default and therefore not sent, so the query string is unchanged for every existing call.execution_idstays out — it is read from the endpoint URL the server handed back.🤖 Generated with Claude Code
https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ