Repository navigation
Tags: Unstructured-IO/unstructured-inference
Tags
ci(release): fix PyPI upload rejecting metadata 2.5 distributions (#528) The v1.6.14 release [failed at the PyPI upload](https://github.com/Unstructured-IO/unstructured-inference/actions/runs/39441662395), so 1.6.14 is tagged on GitHub but not on PyPI: ``` Checking dist/unstructured_inference-1.6.14-py3-none-any.whl: ERROR InvalidDistribution: Invalid distribution metadata: '2.5' is not a valid metadata version ``` The build backend (`hatchling`, unpinned) now writes `Metadata-Version: 2.5`. The publish action is pinned to a Sep 2025 commit whose bundled twine only accepts up to 2.4. v1.13 shipped in June because hatchling still wrote an older metadata version then. | | twine 6.2 (old action, Azure step) | twine 7.0 (this PR) | |---|---|---| | `twine check` on the 1.6.14 wheel and sdist | ERROR: '2.5' is not a valid metadata version | PASSED | **Change** - `pypa/gh-action-pypi-publish` → v1.14.2 (`dc37677`), which moved to twine 7 for metadata 2.5 uploads. - Release dependency group: `twine>=7.0.0` (lock: twine 6.2.0 → 7.0.0, packaging 26.0 → 26.3), so the best-effort Azure Artifacts upload accepts the same files. **After merge:** the published v1.6.14 release still points at the old workflow, so re-running its job won't help. Delete the v1.6.14 release and tag, then publish v1.6.14 again from `main` (package contents are unchanged: #522 plus #519). <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/Unstructured-IO/unstructured-inference/pull/528?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="View guided diff" src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
fix: render filled PDF form field values (init_forms) (#515) ## Summary Values typed into fillable PDF form fields live in **widget annotation appearance streams**, and pdfium only paints those appearances after the form-fill environment is initialized. `convert_pdf_to_image` opened the document but never called `init_forms()`, so filled form-field values were silently dropped from the rendered page image — and therefore from downstream OCR / hi_res partitioning. ## Change `unstructured_inference/inference/pdf_image.py` — call `pdf.init_forms()` immediately after opening the `PdfDocument`, inside the existing `_pdfium_lock`. The form environment is torn down automatically on `pdf.close()`. ```python with _pdfium_lock: pdf = pdfium.PdfDocument(filename or file, password=password) pdf.init_forms() n_pages = len(pdf) ``` ## Test New `test_pdf_image_forms.py` builds a synthetic 1-page PDF in-test (pypdf): an **empty content stream** plus one `/Tx` widget whose value is drawn only by its `/AP /N` appearance stream. - `test_convert_pdf_to_image_renders_acroform_field_value` — renders via `convert_pdf_to_image` and asserts the field value's pixels appear. - `test_convert_pdf_to_image_drops_form_field_without_init_forms` (control) — patches `init_forms` to a no-op, reproducing pre-fix behavior, and asserts the field region is blank. The empty content stream + control test prove the rendered value comes specifically from `init_forms()`, not from page content (verified: ~0 dark px without init, ~3000 with). ## Notes - Behavior change: every fillable PDF now renders previously-missing field text into the page image. - Pairs with the matching `unstructured` PR that recovers the same values in the pdfminer extracted-text layer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
fix: render rotated PDF pages consistently with extracted text (#514) ## Problem For PDF pages with a non-zero `/Rotate`, the rendered page image and pdfminer's extracted text ended up in different coordinate frames. `convert_pdf_to_image` re-applied the page `/Rotate` with PIL on top of pypdfium2's render — but pypdfium2 already renders the display frame, so the extra rotation effectively returned the image to the un-rotated MediaBox frame. pdfminer, meanwhile, honors `/Rotate` and reports coordinates in the display frame. The two were then off by the page rotation, which scrambled the merged layout downstream. ## Fix - `convert_pdf_to_image` no longer re-applies `/Rotate` on top of pypdfium2's render. The rendered image now shares pdfminer's display frame by default. - For `/Rotate != 0` pages, the dominant text orientation is estimated from pdfminer character matrices (the only signal that reliably matches the rendered image). When a single 90-degree orientation clearly dominates — controlled by the new `PDF_ROTATION_DOMINANT_ANGLE_THRESHOLD` (default `0.95`) — the rendered image is rotated so its text is upright. - The applied correction is exposed as `pdf_rotation_correction` in the page image metadata (PIL `.info` and PNG text), so downstream consumers can rotate extracted coordinates by the same angle and stay aligned. - Orientation estimation failures degrade gracefully to no correction (image stays in its native display frame — still consistent with pdfminer, just possibly not upright). ### Guards 1. Only pages with a non-zero `/Rotate` are considered for correction. 2. A correction is applied only when one 90-degree orientation accounts for at least the threshold share of characters; ambiguous pages are left untouched. The correction is applied to the image and reported for the coordinates together, so it can only affect uprightness, never the consistency between the layers. ## Tests - Existing rotation test still passes (now via the correct render-then-correct mechanism). - Added coverage for the no-correction path on unrotated pages and for the threshold guard skipping correction below the configured share. Paired with the corresponding change in `unstructured` that mirrors `pdf_rotation_correction` onto the extracted coordinates. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated description by cubic. --> --- ## Summary by cubic Fix rotated PDF rendering by removing duplicate rotation and applying a text-based correction. Images and extracted coordinates now align with `pdfminer`, pages become upright when a clear orientation is detected, and `pdfminer.six` is now declared so the correction always runs. - **Bug Fixes** - Stop re-applying page `/Rotate` in `convert_pdf_to_image`; `pypdfium2` already renders the display frame matching `pdfminer` coordinates. - For `/Rotate != 0` pages, estimate the dominant 90° text direction from `pdfminer` characters and rotate only if it meets `PDF_ROTATION_DOMINANT_ANGLE_THRESHOLD` (default 0.95); ambiguous pages are left as-is. - Store the applied angle as `pdf_rotation_correction` in PNG/PIL info and surface it in layout metadata so downstream consumers can rotate coordinates to match. - **Dependencies** - Add `pdfminer.six` (>=20231228) and move imports to module level to fail fast if missing; refresh lockfile. <sup>Written for commit b6c6c46. Summary will update on new commits.</sup> <a href="https://cubic.dev/pr/Unstructured-IO/unstructured-inference/pull/514?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat: Track the table extraction method (#513) <!-- CURSOR_SUMMARY --> > [!NOTE] > **Medium Risk** > Touches core `LayoutElement(s)` data structures and modifies cleanup logic that reorders/filters arrays, which could subtly affect downstream consumers if attribute alignment assumptions are wrong. > > **Overview** > Adds a new `table_extraction_method` attribute to `LayoutElement` and the vectorized `LayoutElements` container to record which table algorithm produced a given table (e.g., grid/tatr/vlm). > > Propagates this new field through `LayoutElements` operations (`__eq__`, `slice`, `concatenate`, `iter_elements`, `from_list`) and through layout cleanup routines so table-related metadata (`text_as_html`, `table_as_cells`, and the new method field) is retained and correctly aligned after sorting/filtering. > > Bumps the library version to `1.6.11` and documents the enhancement in `CHANGELOG.md`. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 70d9da7. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
Prepare 1.6.10 release (#512) ## Summary - Add a `1.6.10` changelog entry for Python 3.13 support - Bump `unstructured_inference.__version__` to `1.6.10` <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Low risk release metadata update only: bumps the library version and adds a changelog entry; no runtime logic changes. > > **Overview** > Prepares the `1.6.10` release by adding a `1.6.10` entry to `CHANGELOG.md` noting **Python 3.13 support**. > > Bumps `unstructured_inference.__version__` from `1.6.9` to `1.6.10`. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit fca3497. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
Update Python 3.11 support and release 1.6.9 (#510) ## Summary - Restore Python 3.11 support alongside Python 3.12 in package metadata, lockfile, and CI - Remove the hard `.python-version` pin so the project can be exercised against both supported runtimes - Bump the package version to `1.6.9` and record the change in the changelog ## Testing - `uv sync --locked --all-groups --python 3.11` - `uv sync --locked --all-groups --python 3.12` - `CI=true uv run --locked --no-sync pytest -q -m "not slow" test_unstructured_inference` - `uv build` - `ruff check .` and `ruff format --check .` <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Moderate risk because it expands the supported runtime to Python 3.11 and updates the `uv.lock` resolution, which can change dependency selection and CI behavior across environments. > > **Overview** > Restores Python 3.11 support alongside 3.12 by widening `requires-python` to `>=3.11, <3.13`, updating README messaging, and running `lint`/`test` jobs in CI against a Python 3.11/3.12 matrix. > > Removes the repo’s `.python-version` pin, makes the Docker base image Python version configurable via a `PYTHON_VERSION` build arg, and refreshes `uv.lock` to resolve dependencies for both runtimes (including adding conditional deps needed for 3.11). > > Bumps the package version to `1.6.9` and records the change in `CHANGELOG.md`. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 6a422a1. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
fix: fix rare occasion of initialization error by converting LazyDict… … into dict (#508) ## Summary Fix `"argument after ** must be a mapping, not LazyDict"` raised from `get_model()` in some production environments. `get_model()` was doing `model.initialize(**initialize_params)` where `initialize_params` is a `LazyDict`. `**` unpacking calls `.keys()` on the mapping, which `LazyDict` inherits from `collections.abc.Mapping` — but we've seen environments where that inherited method isn't resolved at call time, surfacing as the error above. The fix materializes `initialize_params` into a plain `dict` via `__iter__` + `__getitem__` (both defined directly on `LazyDict`) before the `**` unpack. `LazyEvaluateInfo` values are still resolved lazily via `__getitem__`, so download-on-first-use semantics are preserved. ## Changes - `unstructured_inference/models/base.py` — normalize `initialize_params` to a plain dict before `model.initialize(**...)`. - `test_unstructured_inference/models/test_model.py` — new regression test `test_get_model_with_lazydict_config` exercising the `LazyDict` path with a `LazyEvaluateInfo` value. - `CHANGELOG.md` + `__version__.py` — bump to `1.6.7`.
PreviousNext