Repository navigation
Tags: Unstructured-IO/unstructured-platform-plugins
Tags
feat(etl-uvicorn): give plugins request-scoped settings and invocatio… …n context (#74) ## Summary - Resolve and bind `invocation_settings` plus `invocation_context` once per `/invoke` request for synchronous, asynchronous, and streaming plugins. - Generate local Python bindings from the ratified [`invocation-context/v1`](https://schemas.u10d.dev/invocation-context/v1.json) and [`errors/audience/v1`](https://schemas.u10d.dev/errors/audience/v1.json) contracts. - Advertise generic invocation transport capabilities and the format-specific `invoke_with_sealed_dag_node_settings_v2` capability by construction. - Map legacy `UserError` failures to a complete nested `plugin_error` envelope with `audience=user`; remove the redundant top-level `blame` spelling. - Preserve exact-format, default-deny sealed-settings dispatch during mixed-version rollout. ## Why Request limits, route registration, thread-context propagation, and HTTP responses are transport concerns shared by every FastAPI-wrapped plugin. The wrapper owns those mechanics while `utic-invocation-settings` remains the authority for document validation and field-envelope resolution. Cross-service wire contracts are referenced by schema `$id` and code-generated locally instead of being imported from another service package. The handwritten adapter retains only transport-specific error mapping and the equal-length batch invariant that JSON Schema cannot express. The generic transport capability never authorizes ciphertext. The controller dispatches on the document's own `format` and forwards sealed settings only when the plugin advertises the exact matching capability; v1 remains unchanged and v2 requires `invoke_with_sealed_dag_node_settings_v2`. ## Merge dependency Unstructured-IO/utic-public-libs#67 is merged and `utic-invocation-settings 0.5.0` is published to public PyPI. This branch now resolves that released wheel and contains no temporary source pin. - [x] Merge #67 and publish `utic-invocation-settings 0.5.0`. - [x] Remove the temporary source pin, relock against the published package, and verify a clean install. - [ ] Merge and publish this package before downstream PRs remove their remaining wrapper pin. ## Validation - `uv run --locked pytest -q` — 187 passed - `uv run --locked python scripts/generate_invocation_contracts.py --check` — generated bindings match the ratified schemas - `uv run --locked ruff check .` — clean - `git diff --check` — clean Part of DTPL-648, part of DTPL-707 --------- Co-authored-by: Nick Franck <nickfranck@unstructured.io>
chore(etl-uvicorn): prepare 0.0.46 release (#78) ## Summary - bump `unstructured-platform-plugins` from `0.0.45` to `0.0.46` - document the `failure_category` response plumbing merged in #77 ## Why `Unstructured-IO/platform-plugins#1780` depends on #77 and currently uses a temporary git source. Publishing `0.0.46` lets it return to a released package constraint. ## Validation - `make check` — passed - `PYTHONPATH=. uv run --group test pytest -q` — 93 passed - confirmed the changelog and package version both report `0.0.46` <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/Unstructured-IO/unstructured-platform-plugins/pull/78?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
fix(etl-uvicorn): do not require a body when every input field is opt… …ional (#73) ## Problem `wrap_in_fastapi` chooses its `/invoke` signature on parameter **presence**: ```python if input_schema_model.model_fields: async def run_job(request: input_schema_model) # no default => body REQUIRED else: async def run_job() # no body accepted ``` A pydantic body parameter with no default is mandatory **even when every field inside the model is optional**. So a plugin whose parameters are all optional gets a required body that no caller has a reason to populate — and before it grew those parameters, that same plugin accepted no body at all. Adding an optional parameter therefore looks backward-compatible while silently flipping the HTTP contract. ## Fix Default the body when the generated input model has no required fields: ```python body_is_optional = input_schema_model.model_fields and not any( field.is_required() for field in input_schema_model.model_fields.values() ) ``` - **all fields optional** → `request: Optional[input_schema_model] = None`; an absent body resolves each field to its own default, which is exactly what the function signature already promises. - **any field required** → unchanged, body mandatory. A downloader invoked without `file_data` still fails validation rather than receiving `None`. - **no parameters** → unchanged, no body accepted. The handler body is extracted into `run_job_with_body` so the two model-bearing branches cannot drift. ## Testing `make check-version`, `make check` clean; **75 tests pass** (5 new). | plugin shape | bodyless | `{}` | populated | |---|---|---|---| | no params | 200 | 200 | — | | all params optional | **200** (was 422) | 200 | 200 | | any param required | 422 | 422 | 200 | New tests are in `test/api/test_api.py`. Verified they catch the regression by reverting the fix: `test_all_optional_params_accept_absent_or_empty_body[None]` fails. The other four are guards against the fix over-reaching — notably `test_required_param_still_rejects_an_absent_body`, since silently accepting an absent `file_data` would be worse than the bug being fixed. Also verified against the **real** playground indexer served through the patched generator, rather than only synthetic functions: - bodyless POST → 200, indexes from the settings-file fallback - populated wire `invocation_settings` → 200, and **takes precedence** over the file fallback That second case is the point: this restores the bodyless contract without reverting the wire-settings capability. Both planes work. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/Unstructured-IO/unstructured-platform-plugins/pull/73?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
feat(etl-uvicorn): ignore SIGTERM in plugin uvicorn Server (#68) ## Summary uvicorn's default SIGTERM behavior load-sheds new requests with 504 while draining in-flight work. Plugin webservers in our platform are expected to **outlive their controller container** during pod shutdown so the controller can finish dispatching in-flight work before the pod is SIGKILLed — the default behavior races with that drain and orphans records. This PR monkey-patches `uvicorn.Server.install_signal_handlers` at module import time so plugin webservers using `etl-uvicorn` install only a SIGINT handler. The process now ignores SIGTERM and exits only on SIGKILL (or SIGINT for local-dev Ctrl-C). ## Why a monkey-patch instead of a Server subclass? Subclassing requires bypassing `uvicorn.run()` and manually constructing `Config + Server`, which loses uvicorn's built-in handling of workers / reload / multiproc. Plugin webservers don't use those today but the surface area isn't worth changing. The patch is 8 lines, well-scoped to module import, and works for any uvicorn.Server instance in this process. ## Test plan - [x] `make tidy` — clean - [x] `make check-ruff` — passes - [x] `pytest` — all tests pass (existing suite + 2 new in `test/test_signal_handlers.py` covering the patch is applied + the SIGINT-only handler set) - [ ] Smoke test: send SIGTERM to a plugin pod, confirm the process keeps serving until SIGKILL at end of pod grace - [ ] Verify plugin logs / traces show no 504s from `/invoke` during a rolling deploy after this version is rolled out ## Version Bumps `unstructured_platform_plugins` to `0.0.44`. Downstream consumers will need a separate change to pin the new version.
fix: use correct Python version in release workflow (#67) ## Problem The release workflow was using `${{ matrix.python-version }}` but there's no matrix in this job. `setup-uv` fell back to the latest Python (3.14), which caused `pydantic-core` to fail building from source — pyo3 0.24.1 only supports up to Python 3.13. The `PYTHON_VERSION: "3.10"` env var was already defined but never referenced. ## Fix - `python-version: ${{ matrix.python-version }}` → `${{ env.PYTHON_VERSION }}` - Remove `make install-dependencies` — not needed in the release job (only `uv build` + publish) ## Next step After merging, delete and re-create the `0.0.42` GitHub release to re-trigger this workflow.
chore: throw UnstructuredIngestError correctly in plugins (#62) ### Summary saw error for pinecone ``` 500: Error in uploader - [UnstructuredIngestError] http error: (400) .... Vector dimension 1536 does not match the dimension of the index 1024 ``` I believe this should be a 400 so I need to raise `UnstructuredIngestError` here unfortunately its defined twice: * https://github.com/Unstructured-IO/unstructured-ingest/blob/main/unstructured_ingest/errors_v2.py#L6 * https://github.com/Unstructured-IO/unstructured-ingest/blob/main/unstructured_ingest/error.py#L6
fix: pydantic error where status code could be None (#61) ### Summary Saw some ASGI errors from [plugins](https://platform-prod-stage-o11y.kb.us-central1.gcp.cloud.es.io:9243/app/r/s/stu2n) Pydantic error where dataclass is asking status code is a int but `e.status_code` is None
Nex-80: get rid of 512 UnrecoverableException (#59) ### Summary Remove UnrecoverableException and 512 ### Notes for the record I also want to remove function `wrap_error` entirely but we need them for now for all ingest error status code... this design is too much layers -> i would suggest ingest to have a parent error class like the client and then have error code defined in ingest -> not here wrapper
PreviousNext