Zipstack · chandrasekharan-zipstack · Aug 11, 2026 · Aug 12, 2026 · Aug 12, 2026 · Aug 12, 2026
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -0,0 +1,20 @@
+name: ci
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+jobs:
+  # Offline by design: no network, no credentials, sub-second. Live round trips
+  # are a manual pre-release step, not a per-PR gate.
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv venv --python 3.12
+      - run: uv pip install -e '.[dev]'
+      - run: uv run ruff check .
+      - run: uv run ruff format --check .
+      - run: uv run pytest -q
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,7 @@
+.venv/
+__pycache__/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+dist/
+build/
diff --git a/README.md b/README.md
@@ -0,0 +1,118 @@
+# unstract-cli
+
+`unstract` — one CLI for the Unstract suite: extract a document with
+LLMWhisperer, run it through a Document Studio API deployment, get structured
+JSON back. It also clones one organization's resources into another.
+
+```bash
+pipx install git+https://github.com/Zipstack/unstract-cli
+unstract config init
+unstract config doctor
+```
+
+## Output
+
+`unstract` prints a table by default — in a terminal and in a pipe alike, so
+what you see while trying something is what a script sees running it.
+
+**Parsing anything? Pass `-o json`.** stdout then carries exactly one envelope,
+on success and on failure alike:
+
+```json
+{"ok": true, "data": {...}, "error": null, "meta": {"contract_version": 1}}
+```
+
+`-o json` output depends on nothing but the command and its arguments — not the
+terminal, not the config, not the environment. `-o raw` prints one field
+unwrapped, for piping a document's text somewhere else. Diagnostics, warnings
+and progress always go to stderr.
+
+Consuming the JSON: ignore fields you do not recognise, and refuse a
+`meta.contract_version` above the one you were written against. `unstract
+--discover full` publishes the whole contract alongside every command and flag.
+
+If a coding agent is driving (detected from the environment it sets), the
+*default* becomes json. `--agent yes|no` forces that either way, and an explicit
+`-o` always wins over both.
+
+Failures exit non-zero with a stable code:
+
+| Code | Meaning |
+|------|---------|
+| 0 | success |
+| 1 | generic failure |
+| 2 | usage error |
+| 3 | authentication failed |
+| 4 | not found |
+| 5 | validation failed |
+| 6 | rate limited |
+| 7 | timed out (the job handle is in the error payload — resume, do not resubmit) |
+| 8 | server error |
+| 9 | result already consumed (one-shot read; use `--save` next time) |
+| 10 | the result was read but could not be saved — it is in `error.details` |
+| 130 | interrupted (128 + SIGINT) — the user stopped it, not a failure |
+
+## Configuration
+
+`~/.unstract/config.toml`, or a project-local `.unstract.toml` found by upward
+search, or `$UNSTRACT_CONFIG`, or `--config`. Every setting resolves
+**flag > env > profile > built-in default**, and the CLI is fully usable with no
+config file at all.
+
+```toml
+default_profile = "cloud-us"
+
+[profiles.cloud-us.llmwhisperer]
+base_url = "https://llmwhisperer-api.us-central.unstract.com/api/v2"
+api_key = "env:LLMWHISPERER_API_KEY"
+
+[profiles.cloud-us.docstudio]
+base_url = "https://us-central.unstract.com"
+org_id = "org_ABC123"
+api_key = "env:UNSTRACT_DEPLOYMENT_KEY"
+
+[profiles.cloud-us.deployments.invoices]
+api_name = "invoice-parser"
+```
+
+One `api_key` on the `docstudio` block covers every alias under it: a key minted
+under **Settings → API Key Manager** authenticates every API deployment in the
+organisation, so an alias normally carries only its `api_name`. Give an alias its
+own `api_key` when its deployment has a separate key of its own.
+
+Get an LLMWhisperer key from the LLMWhisperer console; a deployment key is shown
+on the API deployment's own page in the Unstract UI, and an organisation-wide one
+under Settings → API Key Manager. `config init` also writes an
+`onprem-example` profile as a shape to copy for a self-hosted install — its host
+is a placeholder, and only the *active* profile is ever resolved.
+
+Credentials use `env:VAR_NAME` indirection, so the file records where a secret
+lives rather than the secret itself. `unstract config doctor` reports where each
+setting resolved from — including whether an `env:` reference is actually set in
+the current process — without echoing any value. It exits non-zero when one of
+its own checks failed, so a setup script can branch on it.
+
+A project-local `.unstract.toml` **found by upward search** may not supply
+`api_key` or `base_url`. Those are ignored, with a warning; everything else in it
+— profile selection, `org_id`, deployment aliases — applies as usual. A checkout
+you did not write is not trusted to name the host your key is sent to. Name the
+file explicitly (`--config` or `$UNSTRACT_CONFIG`) and it is honoured in full.
+
+What that protects is the key and the host, not the routing: `org_id`,
+`api_name` and profile selection stay repo-controllable by design, so a
+project file can still decide *which* deployment a command runs against on a
+host you trust. Read one before you run inside a checkout you did not write.
+
+`clone` is the exception: it talks to two deployments at once, which no single
+profile describes, so it takes both endpoints as flags and both admin Platform
+keys from `UNSTRACT_SRC_PLATFORM_KEY` / `UNSTRACT_TGT_PLATFORM_KEY`. It exits 0
+when nothing failed, which is not the same as everything having moved: oversize
+and unsupported documents are skipped by design, and `data.skipped` counts them.
+
+## Development
+
+```bash
+uv venv && uv pip install -e '.[dev]'
+pytest        # offline; no network, no credentials
+ruff check .
+```
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
@@ -0,0 +1,48 @@
+# Release notes — draft
+
+Content for the first release. Not published yet.
+
+## What this is
+
+One CLI for the Unstract suite: extract a document with LLMWhisperer, run it
+through a Document Studio API deployment, clone one organization's resources
+into another. Install it with `pipx`, then `unstract config init`.
+
+## The `unstract` command name
+
+`unstract-client` released before this CLI installed a console script called
+`unstract` too, and that script has been removed there — its clone command is
+now `python -m unstract.clone`, and this CLI's `unstract clone` wraps the same
+code. An environment holding an older `unstract-client` alongside this package
+gives the name to whichever was installed last:
+
+```bash
+command -v unstract && unstract --version
+```
+
+`pipx` avoids the question by giving this CLI its own environment. A second
+console script, `unstract-cli`, always belongs to this package.
+
+## Behaviour worth knowing before you script against it
+
+- **A failure the service reports inside a successful HTTP response exits 5
+  (validation), not 8 (server error).** Exit 8 invites a retry, and on an API
+  that bills per execution a blind retry is a second charge for work that was
+  already done. The service's own report is in `error.details`.
+- **`clone` exits 0 when nothing failed, which is not the same as everything
+  having moved.** Oversize and unsupported documents are skipped by design;
+  `data.skipped` counts them.
+- **`config doctor` exits non-zero when one of its own checks failed**, so a
+  setup script can branch on it. A setting that is simply not configured is
+  reported, not failed.
+- **A custom `page_separator` needs LLMWhisperer v2.64.2 or later.** An older
+  service reads only the previous spelling of the parameter, falls back to the
+  default `<<<` separator, and reports no error.
+
+## Consuming the output
+
+Pass `-o json`: stdout is then exactly one `{ok, data, error, meta}` envelope on
+success and on failure alike. Ignore fields you do not recognise, refuse a
+`meta.contract_version` above the one you were written against, and branch on
+the exit code rather than on message text. `unstract --discover full` publishes
+the whole contract alongside every command and flag.
diff --git a/RUNBOOK.md b/RUNBOOK.md
@@ -0,0 +1,153 @@
+# Runbook
+
+Maintainer procedures. For what the CLI does and how to configure it, see the
+[README](README.md); this file covers the things that are done *to* the CLI —
+installing a build, moving the client pins, proving a build against real
+services, and cutting a release.
+
+## Install
+
+### From a published ref
+
+```bash
+pipx install git+https://github.com/Zipstack/unstract-cli
+unstract --version
+```
+
+Pin the ref when reproducing a report:
+
+```bash
+pipx install "git+https://github.com/Zipstack/unstract-cli@<tag-or-sha>"
+```
+
+`pipx` puts each install in its own virtualenv, which matters here: the two
+clients are pinned to exact commits, and a shared environment would let another
+package's resolver move them.
+
+### Other names for the same CLI
+
+- `unstract-cli` — a second console script this package always owns.
+- `python -m unstract_cli` — works from a source checkout with no install at all.
+
+`unstract-client` released before this CLI installed a console script called
+`unstract` too. An environment that still holds one of those versions gives the
+name to whichever package was installed last, so check what answers before
+filing a bug about a missing command:
+
+```bash
+command -v unstract && unstract --version
+```
+
+### From a checkout
+
+```bash
+uv venv && uv pip install -e '.[dev]'
+pytest          # offline: no network, no credentials
+ruff check .
+```
+
+## Moving the client pins
+
+The CLI derives its flags from the vendored specs intersected with the pinned
+clients' signatures, and takes flag help from those clients' docstrings. Moving
+a pin therefore changes the CLI's surface without a line of CLI code changing.
+That is the intent, so the check is that the change was the intended one:
+
+1. Update the `unstract-client` and/or `llmwhisperer-client` ref in
+   `pyproject.toml`.
+2. Refresh the vendored spec if the service's spec moved too — see
+   [`src/unstract_cli/specs/README.md`](src/unstract_cli/specs/README.md).
+   A spec and a client from different commits is exactly the state
+   `tests/test_contract.py` exists to catch.
+3. `uv pip install -e '.[dev]' && pytest`.
+4. Diff the surface before and after:
+
+   ```bash
+   python -m unstract_cli -o json --discover full > after.json
+   ```
+
+   Every added or removed flag should be one you can name a reason for.
+   `tests/test_contract.py` pins the spec parameters no command can reach; that
+   set should only ever shrink, and only on purpose.
+
+Both pins move to released versions before this ships publicly.
+
+## Live gate
+
+The offline suite proves the CLI is self-consistent. It cannot prove the
+services agree, and the defects worth catching here have all been of that kind:
+a payload shaped differently from the spec, a status code meaning something
+other than it appears to, geometry that divides by a value the service reports
+as zero. Run this against a real tenant before tagging a release.
+
+### Credentials
+
+Supply them through the environment, never on the command line and never in a
+file inside this repository:
+
+```bash
+export LLMWHISPERER_API_KEY=...
+export UNSTRACT_DEPLOYMENT_KEY=...
+export UNSTRACT_BASE_URL=https://<host>
+export UNSTRACT_ORG_ID=org_...
+```
+
+Use a staging tenant. Passing `--api-key` works and warns, because a key on the
+command line lands in shell history and in the process list.
+
+### Checklist
+
+Run against a document you can re-send; several of these submit real work.
+
+| # | Command | Pass |
+|---|---|---|
+| 1 | `config doctor --probe` | every setting reports where it resolved from; the LLMWhisperer probe answers live; exit 0 when nothing failed, and exit 1 with the same report under `error.details` when something did |
+| 2 | `whisper extract <pdf>` | polls to completion, returns text |
+| 3 | `whisper extract <pdf> --no-wait` then `whisper status <hash>` then `whisper retrieve <hash>` | the handle survives the round trip |
+| 4 | `whisper retrieve <hash>` a second time | refused, exit 9, and the error names the one-shot read |
+| 5 | `whisper highlights <hash> --target-width 800 --target-height 1000` | bounding boxes for the lines that carry geometry, and no traceback for the lines that do not |
+| 6 | `whisper usage` | quota returned |
+| 7 | `docstudio deployment run <alias-or-api-name> <pdf>` | polls to completion, returns structured JSON |
+| 8 | `docstudio deployment run <target> <pdf> --no-wait`, then `docstudio deployment status <target> <execution_id>` from the run envelope | the handle survives the round trip |
+| 9 | any command with `-o raw` | one field, not the envelope |
+| 10 | any command with `-o json` and a wrong key | exit 3, JSON envelope on stdout, no traceback |
+| 11 | any command with `-o json` and a path that does not exist | exit 2, JSON envelope on stdout |
+| 12 | any command with no `-o` | a table, in a terminal and through a pipe alike |
+| 13 | `clone --source-url ... --target-url ... --dry-run` | the plan is reported and nothing is written to the target |
+
+Two properties matter more than any single row, because they are what a caller
+depends on and what breaks quietly:
+
+- **With `-o json`, stdout is one envelope in every case above, including the
+  failures.**
+  A traceback on stderr with empty stdout is a bug even when the exit code is
+  right.
+- **A flag passed explicitly reaches the wire, including when its value is
+  falsy.** `--no-include-metadata` must produce a different payload than passing
+  nothing at all. A flag that is silently dropped looks identical to a flag that
+  worked.
+
+### Interpreting a failure
+
+A live failure is a finding about the CLI, the client, or the service, in that
+order of likelihood — check which layer the response actually came from before
+changing anything. Fixes go in the facade or the spec; never in a generated
+directory, whose contents are replaced wholesale on the next generation.
+
+## Release
+
+1. Live gate green against staging.
+2. `pytest` and `ruff check .` clean.
+3. Both client pins on released versions, not commits.
+4. Tag, then verify the tag installs clean in an environment that has nothing
+   else in it:
+
+   ```bash
+   pipx install --force "git+https://github.com/Zipstack/unstract-cli@<tag>"
+   unstract-cli --version
+   unstract-cli --discover groups
+   ```
+
+5. `--discover groups` on the fresh install should match the checkout's. It is
+   the cheapest proof that the built wheel carries the specs — they are package
+   data, and package data is what a build configuration silently drops.