feat: CLI scaffold with config, output envelope and poll engine - #2

Flags for an operation come from the spec the published client is generated
from, intersected with what that client's signature actually accepts: a spec
parameter the frozen client cannot name would raise TypeError at the call
rather than reach the API, so it is not offered.

Two rules keep the derivation honest. Every option defaults to None, meaning
absent, so an unpassed flag is not sent and the client or server default
applies rather than a value pinned here. And only None is treated as absent:
0, false and "" are choices a caller made and travel to the request.

Help text has three sources in order: the overlay, the spec, and the client
method's own docstring, which is the only one that describes the parameters
today. The overlay carries what a generated spec cannot express -- allowed
values, short flags, wording -- in TOML read with the stdlib.

Thirteen commands: whisper extract/status/retrieve/detail/highlights/usage
and its four webhook commands, plus deployment run and status. Each one holds
only what a spec cannot say -- which parameter is the argument, which the CLI
owns, and how a result is polled for.

The CLI runs the poll loop for both products rather than using the loop one
client ships, so --wait, --interval, --timeout and the handle-returned-on-
timeout behaviour are the same everywhere. Deployment runs are queued
(timeout=0) so a request does not hold a connection open for the length of the
job. Line-highlight scaling is arithmetic on a reply rather than a request, so
it is folded into the command that fetches the metadata.

Failures converge on one envelope: LLMWhisperer raises with a status code, the
deployment client returns one, and both become a CLIError with an exit code and
a hint. A result that can be read only once is written to disk before it is
printed.

--discover answers what --help answers, as JSON, in three tiers: groups names
the products, summary adds their commands, full adds every flag with its type,
choices and default plus the exit-code table -- enough to construct a call
without a second round trip. A caller starts cheap and drills down.

Every tier is read back from Click itself, so a described command cannot drift
from the one the parser accepts, and discovery reads no configuration: it is
how a caller learns what exists, so it has to work before anything is set up.

config doctor --probe adds the second diagnostic question -- does the resolved
key work -- to the one it already answered offline, where it resolves from.
LLMWhisperer is checked against its usage endpoint. A deployment has no
side-effect-free endpoint to call, so its entry reports that the settings
resolve and says plainly that nothing was verified.

The vendored specs and the pinned clients move independently, so a refreshed
spec can declare a parameter the published client has no argument for. Such a
parameter is dropped rather than offered and rejected at the call, and dropping
it silently is the failure this pins: the gap is written down per operation, so
widening it is a decision rather than an accident.

Two failures a live call found and no offline test could.

The metadata arrives as a named object carrying the coordinate list under
`raw`, while the client's geometry takes the bare list, so no line was ever
scaled. And a line the service has no geometry for is reported as all zeros,
whose page height is a divisor in that scaling: it raised ZeroDivisionError out
of the client, which the entry point does not catch, so the command printed a
traceback with an empty stdout. Such a line now gets no box.

Three follow-ups to the command surface.

Both client pins move forward, and the six extraction parameters and three
status parameters they gained appear as flags with no line written here --
which is what deriving from the specs was for. The contract test's unreachable
set shrinks to what the clients own rather than lack: the URL-in-body flag and
the execution id read from the endpoint URL.

--base-url, --api-key and (for deployments) --org-id sit on the product group
and fill the flag tier of flag > env > profile > default, which the loader
already supported but nothing populated. A key given on the command line warns:
it lands in shell history and in the process list.

The 406 hint is scoped to deployments. A whisper result read twice comes back
as a 400 whose body says so, and translating on that prose would break the
moment the wording changes -- the service's own message already says what
happened, and it is passed through verbatim.

`deployment status` derived --include-metadata, --include-metrics and
--include-extracted-text from the spec, collected them into **params, and never
passed them to the client. The command succeeded and the payload parsed, so a
dropped flag was indistinguishable from a working one. The poll loop behind
`deployment run --wait` had the same hole, which made a waited run return less
than the identical flags returned without --wait.

Both now forward what was asked for, and the parameters the status endpoint does
not accept are filtered out rather than sent. Tests cover each flag in both
polarities, since a flag silently dropped is exactly what the offline suite
missed.

Alongside:

- `config doctor` no longer reports an `org_id` setting for LLMWhisperer, which
  has none. It always read as unresolved and there was no way to resolve it.
- The deployment probe reports `ok: null`, not `ok: true`. Nothing is called, so
  there is no verdict; `true` beside `checked: false` reads as a live check that
  passed. `resolved` carries what is actually known.
- The 406 hint pointed at --save, which does not exist on the command that emits
  the hint. It now names the command that has it.
- A 400 carries a hint. The service can answer 400 with an empty error body, in
  which case the message was a synthesised fallback and there was nothing else
  to go on.

Adds RUNBOOK.md: install, moving the client pins, the live-gate checklist, and
the release steps.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ

Waiting returns the result and nothing else: the extracted text, or the
deployment's structured output. Neither names the job, so a caller who waited had
no handle to correlate against the service, quote in a bug report, or use for a
follow-up call. Without --wait the handle is the entire payload, so the identity
appeared and disappeared depending on a flag.

Both waited paths now carry it in `meta` -- the whisper hash and the execution
id -- leaving `data` exactly as it was.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ

`--save` exists to protect a read the service serves exactly once, and it
was the flag that lost the data: the write ran after the acknowledging
read, raised `OSError` through an entry point that does not catch it, and
left an empty stdout with the extraction gone. The target is now proven
writable before anything destructive runs, the write goes through a
temporary file so a full disk cannot truncate the previous copy, and a
write that fails anyway raises with the payload attached under its own
exit code -- by that point the envelope carries the only copy left.

Also on the one-shot path: a waited extract read the result with a bare
`.get("extraction")` where the sibling command falls back to the whole
payload, so a response shaped any other way printed `ok: true, data: null`
for a document that had been processed and billed. Both now read it the
same way, and a genuinely empty result is a failure rather than a silent
success.

Redaction was an opt-in keyword argument that only the success path
passed, so every error envelope and every stderr summary went out with
the key in it -- four times on stdout in the reproduced case. Credentials
are now registered where they resolve and scrubbed by every emitter, and
`CLIError.details` is redacted structurally rather than at each call site.

Three more places where a failure was reported as a success: the
standalone status commands ignored a finished-and-failed execution inside
an HTTP 200, the poll loop treated an unreadable body as progress and then
blamed the timeout on a job it never confirmed was running, and any status
outside 4xx/5xx mapped to exit 0 while printing `ok: false`.

Verified by mutation -- moving the save after the print, dropping the
registry, dropping the details redaction and dropping the status check
each fail the suite now, and none of them did before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ

A CLI whose output shape depends on whether a terminal is attached is a CLI
whose scripts break when they move from a shell to CI. This drops the
isatty question entirely: the default is a table, in a terminal and in a
pipe alike, and anything that parses the output asks for `-o json`.

An explicit `-o` is the last word. The environment picks the default and
nothing more, so the same `-o json` invocation renders the same bytes
wherever it runs -- which is the property a caller is actually relying on.
Coding agents are the exception worth making: they set a marker in the
environment, and there the default becomes json rather than making every
call carry a flag. `--agent yes|no` settles it either way.

Every envelope now carries `meta.contract_version`, and `--discover full`
publishes what a consumer has to do to hold up its end: ignore unknown
fields, refuse a version above the one it was written against.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ

Nothing bounded a stalled connection: the deployment client is untimed and
its api_timeout is an execution mode the backend reads, not a socket
timeout. --transport-timeout sets one. Unset by default, so a run that
would have hung still hangs rather than starting to fail in a way no
existing script expects.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ

Ctrl-C came back as exit 1 with nothing on stdout, which reads to a
supervisor as a failed command worth retrying -- the one thing that must not
happen to a run the user deliberately stopped. It now exits 130, the value
every shell already reads that way, and prints the same envelope as any
other failure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ

Overlay, spec and client docstring can each describe a flag. No spec
parameter carries a description today, so the order between them is
unexercised until one does, which is exactly when an inversion would ship
unnoticed.

The vendored LLMWhisperer spec was several revisions behind and now declares
enums the CLI was hand-listing. The two had already diverged: --mode rejected
three modes the service accepts and --output-mode two, and nothing would have
reported it. Read the enum off the spec, keep the overlay for narrowing one on
purpose, and drop the descriptions' own value lists for the same reason their
default sentences are dropped.

`highlights` gains a `mode` query parameter that the published client has no
argument for, so it joins the parameters the CLI cannot reach.

A sentence-shaped match ends at the first period, so "Defaults to 0.3." was
left in the help beside the default rendered from the signature. Strip each
restated sentence with its own end-anchored pass instead.

The pinned clients predated the fix that stops an omitted optional parameter
being sent as the string "None", so a CLI built on them sent it. The derived
surface is byte-identical across the move; neither signature changed.

Each of these restated the line below it, or described a prior state that is no
longer there to check against. Keep the reason, drop the narration.

Copies one organization's resources into another by calling the client's
orchestrator directly. Two endpoints with a key each, which no single profile
describes, so both are flags and both keys come from the environment.

Also moves the client pin forward to pick up the status path-prefix fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ

The vendored copy was several iterations behind the one the pinned client is
generated from, so the CLI's help, its parameter set and what --discover
publishes all described an older service contract.

The flag snapshot is the check that makes a resync safe: every other contract
assertion reads the spec on both sides of its comparison, so a spec that loses
a parameter loses the flag and the expectation with it.

…tatus

A transport error was translated into a CLIError outside the poll loop, where
the handle no longer exists, so the caller was left to resubmit a document the
service had already processed and billed. Translating at the call keeps the
loop's own context; the loop attaches the handle itself for anything the caller
did not translate.

`whisper status` reported a failed extraction as a success, its sibling in the
other product having already been fixed: both read the body, not the status
code.

Four failures the CLI reported as successes or as something vaguer than it knew:

- a server-reported error inside a 2xx got the catch-all exit code, which is
  the least informative one for the most interesting failure this API has;
- `config doctor` printed its own findings and exited 0, so a setup script
  branching on it read a broken configuration as a working one;
- a deployment alias pointing at an unset environment variable fell back to the
  profile's organisation and key, running against a tenant nobody named;
- a webhook's auth token was echoed verbatim.

The restated-default stripper was also greedy to the end of the string, so a
description whose value list came first lost every sentence after it.

The command that writes into a live organisation had none of its own
behaviour pinned. Its table output -- the one a person gets, and the only
output path that did not go through the emitter -- scrubbed by hand and was
run by no test, while the test that claimed a platform key never reaches
stdout passed with the registration deleted. Rendered output now goes out
through the same path as every envelope, and a key planted in a report is
asserted not to survive it.

Also: --on-name-conflict decides what is written into the target and is now
asserted to arrive; skipped documents are counted at the top of the payload,
because skipping is not fatal and a caller reading the exit code alone would
never learn a document did not move; `config doctor` resolves each deployment
alias the way a run does, instead of listing names its docstring implies it
checked; a failed retrieve is pinned to carry the handle; the restated-default
stripper ends at its own sentence rather than at the end of the text; and the
groups tier lists leaf commands apart from groups, which a consumer walks
differently.

The status endpoint's own query parameters are forwarded now, and a
deployment URL that carries no derivable prefix is polled where the service
said rather than at a rebuilt path.

The notes carry the console-script collision, the behaviours a script would
otherwise discover by being surprised, and the service version a custom page
separator needs. The pin moves to a documentation-only commit.

A .unstract.toml found by upward search comes from whatever checkout the
user happens to be standing in. It may still select a profile, set org_id
and define deployment aliases; api_key and base_url are withheld, with a
warning, and reported as withheld by config doctor. Named explicitly with
--config or $UNSTRACT_CONFIG, the same file is honoured in full.

Also point a first-time user at where keys are minted, from config init,
from doctor and from the README, and ship an on-prem profile shape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ

The discovered config path is written to as well as read from, so a
symlinked .unstract.toml let a repository redirect config set and
config init --force onto any file it named. The upward search now skips
a symlinked candidate, and the write opens with O_NOFOLLOW so a symlink
at the target is a clear error rather than a truncation.

Also: the config group reports the file's warnings instead of dropping
them, doctor answers for a withheld deployment-alias key the way it does
for a product one, trust is derived from the path rather than from how
the loader was called, and the README says plainly that routing stays
repo-controllable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ

Reverting either left the suite green: the discovered file classified as
project-local however its path is spelled, and the config group reporting
what the loader withheld.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ

A spec resync that narrows an enum, changes a type or moves a default
left the gate green while the CLI began rejecting a value it used to
take. The snapshot now carries the whole parameter surface, and the
failure names the flags that moved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014f9oEEYspPH4fmPULTnLkJ