Skip to content

feat(fp): Cloud CLI parity — evaluator authoring (code, LLM judge, Jev), dashboards, sentiment, agent, new look - #885

Draft
chhhee10 wants to merge 19 commits into
feat/tui-uxfrom
feat/fp-cli-parity
Draft

chhhee10 wants to merge 19 commits into
feat/tui-uxfrom
feat/fp-cli-parity

Conversation

@chhhee10

@chhhee10 chhhee10 commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Stacked on feat/tui-ux (#881); this draft targets feat/tui-ux, not main.
Companion agenteye skill-doc PR: https://github.com/FailproofAI/agenteye/pull/1073
(base feat/agent-skills). Merge the agenteye documentation PR first, then this client PR.

What changed

  • fp evaluators now authors code, LLM judge and Jev definitions through the existing API: typed inputs or API-body spec files, AI draft conversion, compile/session dry-run, immutable deployment (create first, disable predecessors second), labels/toggles, managed instances, schema, jobs, and code-only backfill. Adds session re-evaluation.
  • Dashboard/tile management, session overview/export/history, event latency, sentiment, SQL composition/export, alert recipients/issues and agent contracts.
  • Session-only policy author/backtest/candidacy, assistant artifacts/home blocks rendered through the permission-gated query runner, interactive write approvals (scripts/JSON decline unless --approve), and home briefs.
  • Organization creation, permission-set roles, member deletion/defaults, usage windows, offers/trial, model context-window overrides, Cloud OTEL controls and Slack status/disconnect.
  • Missing /v1 families and additive-capability 404 degradation (exit 6). Existing commands retain JSON contracts; new session-only commands refuse key mode before HTTP.
  • Shared Rich primitives adopt the mint/pink wordmark, grey labels, uppercase headings and open tables/cards. Full ids remain visible; narrow generic tables stack copyable values.
  • Bundled skill/reference, README, docs page and next-beta 0.0.2b1 changelog section. _version.py is unchanged. No release/publish workflow was dispatched.

Why

The Cloud dashboard can author/deploy evaluators and operate observability features that the CLI could only read. This keeps those workflows accessible to humans and coding agents, using the same commands as the parallel agenteye skill documentation, without adding server endpoints.

Tests

  • uv sync --locked --extra dev; FP_CLI_REQUIRE_CONTRACT=1 uv run --extra dev pytest: 1,544 passed, 1 skipped after round 4. The skipped case is an agent API-key route that does not exist; command-level key denial remains tested.
  • respx coverage for every new command, cookie/bearer routing, session-only refusal, 404 degradation, source validation, draft conversion, deploy ordering/failure recovery and interactive/default-decline approvals.
  • uv build, isolated clean wheel installation, installed fp --version.
  • python tests/fake_cloud_smoke.py --fp <isolated-wheel-env>/bin/fp --artifacts /job/artifacts: 20 real CLI invocations passed, including fake OTP login; draft → create-file → dry-run → deploy-v2 → list → disable; judge/Jev create; dashboards/tiles; sentiment; artifact query rendering; JSON decline, explicit approval, and a real PTY approval while SSE is paused.
  • Scoped Ruff (F,E9) and mypy on the new Python modules passed; root lint completed with 0 errors (existing warnings), root tsc --noEmit passed.
  • Root bun run test:e2e: 351 passed / 6 skipped after rebuilding the local public bundle. Initial failures used a stale generated dist. Rebuilt bun run test:run: 8,305 passed / 10 skipped / 2 failed, exclusively daemon tests that stage linux-x64 packages while this container is linux-arm64. An untouched origin/feat/tui-ux archive reproduces exactly those two failures (26 daemon tests passed); evidence in /job/artifacts/fp-base-daemon-tests.txt.
  • Mintlify validation passed; bun run validate:mdx: 1,061 MDX pages parsed without broken images. Skill frontmatter validation and changelog extraction passed.
  • Broad mintlify broken-links reports 266 links in 14 unchanged translated README files. The untouched base produces the identical 266/14 result; evidence in /job/artifacts/fp-base-broken-links.txt. No new broken links were reported for the changed Cloud CLI page.

Dependencies / open questions

  • New dashboard capabilities live on the stacked agenteye feature branches, not necessarily an older Cloud. New route 404s degrade cleanly; no server or main CLI/TUI files were changed.
  • Manager should run the final smoke against a real Cloud; this environment has no live Cloud.
  • Docker is not installed in this worker (docker: command not found), so the mandated Docker recipe could not run. Clean wheel installation + user smoke provide the Python-package substitute.
  • ci.yml excludes PRs targeting feat/tui-ux; its trigger fix is already owned by active feat/otel (feat: TUI redesign, config v2 and OpenTelemetry setup (one PR: #881 + #899 + OTel) #880), so it was left untouched. A disjoint fp-cli-parity.yml supplies Python 3.10–3.13 tests, clean-wheel user smoke, MDX/changelog checks for this stacked feature.
  • /v1/offers/* routes exist, but Cloud additionally requires actor identity. fp plan and fp plan claim now deny API-key mode before HTTP with the standard signed-session message.
  • Optional org slug is derived from its name because the existing server requires a slug. AI draft --kind is communicated in intent (the existing composer has no dedicated kind input); a mismatching draft fails clearly.
  • Output files are created exclusively to avoid silent overwrite. Deploy version defaults to the next numeric version; non-numeric catalogs require an explicit version.

CI

The original parity commit passed the stacked Cloud CLI matrix (all 1,236 tests plus
installed-wheel fake-Cloud smoke on Python 3.10–3.13), docs and external checks:
https://github.com/FailproofAI/failproofai/actions/runs/38484761581
Fix round 2 (54c19910) passed the Python 3.10–3.13 matrix, clean-wheel user smokes,
docs and external checks:
https://github.com/FailproofAI/failproofai/actions/runs/38494101566
No pending or failing checks. The PR remains a draft.

Fix round 3 final commit 79d0a16c passed Python 3.10–3.13 tests, the expanded
installed-wheel fake-Cloud smokes, docs and external checks:
https://github.com/FailproofAI/failproofai/actions/runs/38508895091
No pending or failing checks. The PR remains a draft.

Fix round 2 — review follow-up

  • Removed the unused colour-keyed PERM_RANK; action-based risk ordering stays unchanged and has a regression test.
  • Corrected full-id helper and chat-list docstrings/comments without changing id output.
  • Deploy --backfill uses the shared UTC formatter (Z, whole seconds), with exact request-body and ordering tests in session/key mode.
  • Failed /api/agent/answer delivery keeps its original exit code/request id, exits with one concise contextual message, and does not save/continue the chat. JSON errors stay structured. Real Rich live-display tests verify cursor restoration and no traceback.
  • Companion agenteye PR is linked at the top: https://github.com/FailproofAI/agenteye/pull/1073

Validation: 1,245 contract-mode Python tests passed; scoped Ruff/mypy and wheel build
passed. Installed CLI approval-failure smoke passed for 404 and network disconnects
in both PTY and JSON mode, followed by the existing 20-invocation user smoke.
Transcripts: /job/artifacts/fp-round2-approval-smoke.txt and
/job/artifacts/fp-round2/fp-user-smoke.txt. No version bump or release.

Fix round 3 — final QA/review follow-up

  • Judge validation preserves the server's six binary-score literals and adds normalized whitespace, equivalent directives (“give 0 when”, “return true if”), and pass/fail, yes/no, true/false pairs. Flags and spec-file inputs are covered, continuous grading remains accepted, and server HTTP 422 still surfaces with its status/message.
  • HTTP 404 classification prioritizes server route/record codes and messages, then request shape. An empty record-scoped 404 says the record was not found (exit 6); explicit unknown-route and collection-capability 404s use the older-Cloud availability message (exit 6). Saved-query ids, session ids, model ids and approval call ids supplied in bodies/queries are treated as record selectors. Existing non-additive commands retain their error contracts.
  • Standalone evaluator backfill normalizes offsets, fractional seconds and trimmed timestamps to the same whole-second UTC Z format as deploy backfill.
  • Top-level help advertises every registered subcommand, with a bidirectional coverage guard.
  • Artifact rendering puts the wordmark before the title, once per screen. agent ask key mode uses the same signed-session wording as other session-only commands.
  • README, bundled skill/reference, Cloud CLI page and changelog now distinguish unknown-route from missing-record errors. Mintlify and skill-creator guidance kept these edits focused on the changed contract; no agenteye code was edited.

Tests: final contract-mode pytest 1,360 passed / 1 skipped; scoped Ruff/mypy,
wheel build, Mintlify validation, 1,061 MDX pages and skill validation passed.
The installed-wheel fake Cloud smoke now runs 28 CLI invocations, including
QA judge phrasings, offset backfill normalization, missing-session/unknown-route
404 distinctions, help coverage, header ordering and key-mode denial. The real
PTY/JSON approval-failure smoke also passes with cursor restoration and no traceback.
Transcripts: /job/artifacts/fp-round3/fp-user-smoke.txt,
/job/artifacts/fp-round3-approval-smoke.txt. No version bump or release.

The first round-3 CI smokes exposed a test-only portability issue: forced ANSI
styling splits the wordmark into escape-decorated spans. This was reproduced locally
with FORCE_COLOR=1 and fixed by comparing visible text after stripping ANSI,
without weakening the header-before-title assertion. The forced-colour 28-command
smoke passes. Final head: 79d0a16c; CI is green on that exact commit.

Round 4 — manager's live Cloud output feedback

Read /job/manager/live-run.log and replaced the additive commands' generic
field/JSON renderer with human-only, resource-specific views. Evaluator create/show
summarize source/envelopes; lists remain real tables at 80 and 140 columns; deploy
names the key/new version and disabled predecessor versions. Roles display counts
and wrapped permission names; dashboards/tiles, sentiment, plan, OTEL, Slack,
context windows, queries, sessions, policies and briefs have short headings,
aligned grey labels, state glyphs and actionable empty states. JSON response
bodies and envelopes remain unchanged; raw exports remain available with --json
or --out, and CSV remains pipeable.

Pre-deploy dry-run results are now displayed per session before creation,
including status, score, reasoning and errors. Failed runs prevent POST/PATCH
unless explicit --force authorizes proceeding. Tests assert preview visibility
at the instant POST begins, blocked writes, force behavior and unchanged JSON.
Plan/claim need a signed-in person, deploy's positional key may fill a missing
spec-file eval_key, and an ambiguous older-Cloud settings/<key> 404 is treated
as route availability (explicit server missing-setting messages stay intact).

Validation: 1,544 tests passed / 1 skipped, including 154 human-view cases
with realistic server payloads and empty states at 80/140 columns, plus 30
dry-run/force/auth/spec/404 regressions. Scoped Ruff/mypy, wheel build, Mintlify
validation, 1,061 MDX pages, skill validation and changelog extraction passed.
The installed-wheel fake-Cloud smoke passed 33 CLI invocations at each width
under forced ANSI styling; PTY/JSON approval-error smoke passed too.
Artifacts: /job/artifacts/fp-round4-human/, /job/artifacts/fp-round4-80/,
/job/artifacts/fp-round4-140/, /job/artifacts/fp-round4-approval-smoke.txt.

Mintlify and skill-creator guidance kept the docs focused on human views, explicit
failed-preview overrides, export modes and session-only plan commands. No server,
agenteye, main CLI or version changes; no release.

Round-4 final commit 29872166 passed Python 3.10–3.13 tests, installed-wheel
fake-Cloud smokes, docs and external checks:
https://github.com/FailproofAI/failproofai/actions/runs/38523408423
No pending or failing checks. Mintlify deployment is intentionally skipped for
this draft; the PR remains a draft against feat/tui-ux.

@coderabbitai

coderabbitai Bot commented Oct 13, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Thanks @chhhee10 for your contribution to Failproof AI! 🙌

We'd love to discuss your PR and welcome you to our community.

Discord: https://discord.befailproof.ai/
Reddit: https://www.reddit.com/r/failproofai/

chhhee10 and others added 8 commits October 13, 2026 23:23
A new version deployed with flags dropped the predecessor's condition_source
(and labels), silently widening the evaluator to every session. Found in a live
run against a real Cloud: v1 scoped to session.agent_id == 'support-bot', v2
came out 'always'. Inherit both from the latest version unless --condition /
--label are given; a --file spec still states every field itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chhhee10 and others added 3 commits October 14, 2026 14:36
chhhee10 added a commit that referenced this pull request Oct 24, 2026
Applies the own delta of feat/fp-cli-parity (git diff feat/tui-ux...feat/fp-cli-parity)
on top of #928: evaluator authoring (code, LLM judge, Jev: draft, dry-run, deploy,
backfill), dashboards, sentiment, session export/history/re-evaluate, assistant
artifacts and approvals, account/integration controls, alert recipients/issues,
audit agent contracts and the mint/pink human output. JSON shapes unchanged.

Dropped the stacked-branch workflow (fp-cli-parity.yml, targeted feat/tui-ux) and its
test; ci.yml already runs the fp-cloud-cli suite. Root CHANGELOG keeps staging's text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NiveditJain pushed a commit that referenced this pull request Oct 25, 2026
…loud Jev policies (#933)

* [failproofai-config-account-login] Log in to FailproofAI Cloud from `failproofai config`, and give every agent the failproofai skill (#920)

* [failproofai-config-account-login] Log in to FailproofAI Cloud from `failproofai config`

The CONNECT step opens with a menu: log in through the browser (default),
paste an API key, or use open source. Logging in opens a link (or prints
it over SSH / with no display) where a person signs up or in, picks or
creates an org and authorizes the machine; the CLI polls with a secret
only it holds and shows who approved it, for which org. The key is
claimed at APPLY, after the daemon installs, so an abandoned run mints
nothing. A connected machine is offered "keep" first, or "log in again"
first when its saved key no longer works.

- src/hooks/cloud-login.ts: start / poll / claim against
  /api/auth/cli/*, injectable fetch, no redirects followed, only http(s)
  links ever handed to the opener; older clouds read as unsupported
- tui promptBrowserLogin: link printed above the live region (never cut),
  esc back, ctrl-c cancel, expiry/deny offer a new link, keeps polling
  through a dropped connection
- configure-wizard: Connect menu, claim at apply, re-login note, sudo
  re-asked at apply when the cache expired during the login
- open-browser: canOpenBrowser (false over SSH / no display), openUrl
  exported
- e2e harness __tests__/e2e/layout/cli-login.sh + pty driver
- docs: quickstart and CLI reference lead with the login

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Changelog: PR number for CLI login (#920)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [config-account-login] Match the simpler authorize page; re-ask sudo before a unit refresh

- The terminal now says "Not authorized in the browser." when the person picks
  [ don't authorize ] on the page (was "The request was cancelled in the
  browser."), matching the page's vocabulary; the e2e harness waits for it.
- The quickstart no longer asks people to check the key's name and
  permissions: the page stopped showing them, the server sets both.
- The e2e harness approves without a `jev` flag, exactly as the page does now,
  and still asserts the key gets jev:evaluate because the approver holds it.
- CodeRabbit: the sudo re-ask before apply also covers refreshing a stale
  service definition; failing it there costs the scheduled audit (caution),
  never the setup. The reference doc no longer says setup "never" prompts for
  a password: a terminal run does, and asks again if it expired.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [config-account-login] Start the login by itself; harden what the server sends

Connect to cloud on a machine with no connection (or a refused saved key)
now starts the browser login straight away — link generated, browser opened
or link printed, countdown running. Tab or Esc there opens the other
options; Ctrl-C cancels. A machine connected with a working key still gets
the menu, keep first, so re-running config never opens a browser. An older
cloud drops straight into the options without the login entry. Leaving the
waiting screen erases the link it printed (rows counted at the current
width), so no dead link sits above the menu.

Deep-scan fixes:
- Everything the server sends is printed as text only: org name and slug,
  approver, key name, errors (controls, bidi overrides and zero-width
  characters stripped — an org named with escape codes cleared the screen
  and retitled the terminal mid-setup), and the link as the URL parser
  serialises it. Same for the introspect path's "Connected as".
- The server's poll interval and expiry are clamped (an interval past 2^31
  ms made setTimeout fire every millisecond); slow_down never speeds up.
- A saved cloud address is validated like a typed one.
- A key that was created but could not be put to use is named, so it can
  be revoked; a "log in again" that created no key reports the previous
  connection, not open source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [config-account-login] A failed claim on a fresh machine is not "kept"

savedConnection() returns null, not undefined, so `saved !== undefined` was
always true: a fresh machine whose login claim failed printed "This machine
keeps its previous connection." and reported mode "cloud" with no credential
on disk. Truthiness, like every other check of `saved` here. The fresh-machine
claim-failure test now asserts mode "oss" and the absence of that line
(CodeRabbit, PR #920).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [config-account-login] Release as 1.0.12-beta.1

1.0.12-beta.0 is already on npm (published from this branch on `next`, built
from a42e75f), so publish preflight would refuse it from main. Move
package.json, the Cargo workspace (+ Cargo.lock) and the Hermes plugin
manifest to 1.0.12-beta.1, and give it its own changelog section with the one
change the published beta.0 lacks (b7aaddd).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [config-account-login] A refused saved key is not kept as if it worked

When the saved key was refused and the login that followed created no key,
the run ended with "This machine keeps its previous connection." The
connection is still kept (the CONNECT step never removes one, decision D4,
same as picking "keep" with a refused key), but the line now says its key no
longer works and to run config again. The verdict from the CONNECT step is
carried to the end as `savedRefused`. Regression test fails without the fix
(CodeRabbit, PR #920).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [config-account-login] Close the remaining CodeRabbit / Hermes findings

- checkCloudKey validates the address before sending a key to it: a saved
  http:// address to another host no longer receives the saved key when
  config checks it (CodeRabbit, outside-diff, b7aaddd). Five older tests
  used the placeholder address "u", which the guard now refuses; they use the
  valid https address their neighbours already use.
- configure_applied is sent after the cloud step with the final `connected`
  and mode, so a login whose claim failed is not counted as connected
  (CodeRabbit + Hermes F4).
- promptBrowserLogin: a retry erases the expired/refused link (and its
  heading) before printing the new one — `forgetAbove()`, shared with leaving
  on Tab/Esc (CodeRabbit + Hermes F3).
- startLogin: a 200 that is not JSON (the web app serving a page in the
  route's place) is `unsupported`, so the key prompt is offered (CodeRabbit).
- connectToCloud takes `org` from the caller when introspect gives none; the
  wizard passes the org from the login's claim, so the saved org is not left
  empty or stale (CodeRabbit).

Each new test fails without its fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [config-account-login] Release as 1.0.12 (stable)

1.0.12-beta.1 was published to npm on `next` from another branch
(feat/cloud-jev-on-main) before this PR took that number, so publishing it
from main would be refused by preflight. Cut the stable release instead:
1.0.12 across package.json, Cargo.toml/Cargo.lock and hermes-plugin/plugin.yaml,
and the top changelog section becomes `## 1.0.12`, which the stable
release-notes check and the Discord announcement read along with every
`1.0.12-beta.*` section below it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [config-account-login] Give every agent the failproofai skill

`config` now writes one small `failproofai/SKILL.md` into each agent it finds
(and any ticked), in that agent's own user-scope skills folder. The file is a
pointer: it sends the agent to the umbrella skill and its five siblings in the
public FailproofAI/skills repo, so the content is read live and never goes
stale on disk.

Every folder was verified against the real CLI in Docker (probe skills, the
CLI's own listing or a mock-model capture), then re-checked end to end with
this code for Claude Code, Codex, Copilot, OpenCode, OpenClaw, Devin and Goose:
- one folder per agent, following CLAUDE_CONFIG_DIR, CODEX_HOME, COPILOT_HOME,
  XDG_CONFIG_HOME, PI_CODING_AGENT_DIR, FACTORY_HOME_OVERRIDE and
  GOOSE_PATH_ROOT; every Hermes and OpenClaw profile gets its own copy;
- never the shared ~/.agents/skills (Claude ignores it, Codex lists dupes,
  Copilot drops it under COPILOT_HOME, OpenClaw lets it outrank its own);
- name = folder name, description under 200 chars and led by its subject
  (Copilot drops >1024, droid shows 200, Hermes 57).

A `.failproofai-managed` marker marks folders failproofai wrote: one without it
(e.g. `npx skills add FailproofAI/skills`) is never overwritten, and
`failproofai uninstall` removes only marked folders. A folder that can't be
written is logged and skipped; setup never fails on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [config-account-login] Add the failproofai skill silently

Drop the "Added the failproofai skill to N agents." line from config's
summary; the summary stays about enforcement and the connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [config-account-login] Don't give Codex the failproofai skill twice

Codex reads ~/.agents/skills as well as its own ~/.codex/skills and lists a
name found in both twice (0.164.0). `npx skills add FailproofAI/skills -g`
puts the full skill in ~/.agents/skills for Codex, so a machine with that and
`failproofai config` showed Codex two `failproofai` skills.

When ~/.agents/skills/failproofai exists, config now leaves Codex to it: it
writes no Codex copy, and removes one it wrote on an earlier run. Checked in
Docker against real Codex: npx then config lists it once; config then npx
lists it twice until config runs again, which takes ours back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [config-account-login] Never stop setup on removing an old Codex skill copy

Taking back our Codex copy (when ~/.agents/skills already serves Codex) called
rmSync outside any try, so a folder that could not be removed would throw out
of installAgentSkill and fail `config`, breaking its never-fails-setup
contract. Route it, and uninstall's removal, through one removeSkillDir that
logs and carries on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [config-account-login] Report the kept connection when a new login's key can't be used

Logging in again on a connected machine, with the key claimed but refused
on both streams, made connectToCloud write nothing, so the previous
connection stayed on disk and working. `keptOld` tested `!token`, though, so
the summary and configure_applied reported the machine as open source.
Test `!connected` instead. A refused saved key still counts as kept (mode
cloud, decision D4), with its own message. (CodeRabbit)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* ci: test pull requests targeting staging

* fix(deps): clear staging supply-chain findings

* fix(security): scope Python 3.10 filelock exception

* [failproofai-feat-fpai-collect-optional-sqlite] fpai-collect: SQLite pollers behind a default-on `sqlite` feature (#929)

* fpai-collect: SQLite pollers behind a default-on `sqlite` feature

AgentEye's server imports uploaded Claude Code and Codex transcripts with
fpai-collect's own converters and redaction, so an imported session is shaped
exactly like a captured one. It cannot depend on the crate as it stands:
rusqlite 0.40 links `sqlite3` through libsqlite3-sys 0.38, the server's sqlx
links it through 0.30, and Cargo allows one crate per native library.

- `rusqlite` is optional behind `sqlite` (default on). Gated: `sqlitepoll`,
  the goose/opencode/hermes/devin sources and `openclaw::sqlite`.
- `to_rfc3339_micros` moves from goose to `sources` (goose re-exports it), so
  the cursor file source keeps it without the feature.
- The seven integration tests that build real SQLite databases declare
  `required-features = ["sqlite"]`.
- CI lints the crate without the feature too; the workspace build always has
  it on, so nothing else would notice that shape breaking.

failproofaid builds with the feature and is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* CHANGELOG: PR number for the optional sqlite feature (#929)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix: address combined PR review findings

* [failproofai-feat-cloud-jev-on-main] Cloud-managed Jev and hybrid policies, scoped to agents (#928)

Squash of #928 (feat/cloud-jev-on-main @ 4e679a8) onto staging so main
gets one unified PR (#933).

Conflict resolutions:
- Versions stay at staging's 1.0.12 (package.json, Cargo.toml/lock,
  hermes-plugin/plugin.yaml). The unpublished 1.0.12-beta.2 fixes move
  into the 1.0.12 CHANGELOG section; the published 1.0.12-beta.1 section
  is kept as-is.
- cloud-connection.ts: keep #928's same-org reconnect short-circuit, and
  compare on staging's `org` (introspect, else the browser-login claim).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(jev): keep captured intent with the agent profile it was typed into

Review of #933 (High): Cloud Jev's tool-call review was already skipped for
a profile no Cloud Jev check targets, but intent capture still ran for every
profile and stored prompts under the session id alone. Session ids are the
harness's, so a non-targeted profile could submit a prompt under the same id
as a targeted one, and that prompt was then sent to Cloud Jev as the targeted
profile's stated intent, able to clear a reviewable `both` policy's regex
deny. It also kept prompt text from profiles the deployment did not target.

- The intent store is keyed by agent profile + session id:
  sessions/<integration>[.<instanceId>]/<sessionId>.json. The profile is the
  runtime AgentIdentity the handler resolves (the same one agent targeting
  matches on), or the bare integration when none is rostered. Capture
  (handler -> captureIntent profile) and review (startJevReview agent ->
  readIntent profile) use the same identity; jev-review's reader requires it.
- Legacy flat sessions/<id>.json entries cannot say which profile wrote them,
  so the hook path never reads them; they expire under the 6h sweep, which
  now also sweeps profile directories. The loose-directory check covers the
  profile directory too.
- Under a Cloud Jev mode, nothing is captured for a profile with no Cloud
  Jev assignment and no installed pack check (startTwoTier's own skip,
  minus the pause). A deployment without the assignment list captures as
  before. Local (non-Cloud) Jev captures as before, per profile.

Tests: handler-level reproduction of the attack (non-targeted profile, same
session id, cannot clear the targeted profile's deny; also when an installed
pack makes the other profile's prompt kept; also a legacy flat entry), the
capture gate, and store-level profile isolation, sweep and permissions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(fp): port Cloud CLI parity from #885 onto staging

Applies the own delta of feat/fp-cli-parity (git diff feat/tui-ux...feat/fp-cli-parity)
on top of #928: evaluator authoring (code, LLM judge, Jev: draft, dry-run, deploy,
backfill), dashboards, sentiment, session export/history/re-evaluate, assistant
artifacts and approvals, account/integration controls, alert recipients/issues,
audit agent contracts and the mint/pink human output. JSON shapes unchanged.

Dropped the stacked-branch workflow (fp-cli-parity.yml, targeted feat/tui-ux) and its
test; ci.yml already runs the fp-cloud-cli suite. Root CHANGELOG keeps staging's text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(fp): align ported commands with staging and fill dashboard gaps

- policies backtest: every-kind backtest for jev/both (published or --semantic),
  estimate by default, --run/--sample to spend Jev allowance; regex path unchanged
- policies author --kind regex|jev|both
- issues list/count: inbox filters; new issues facets
- sessions import FILE... (POST /api/events/import)
- list trigger_events / eval_envs / metric_keys
- evaluation backfill covers judge/Jev (#1183 on the server); budget warning
- usage windows: API key only (no dashboard session route)
- issues close help: exit 1, not 9; point at audits dismiss
- changelog: fold the parity entries into the open 0.0.2b0 section

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(fp): document every fp command and flag; bring in the skills-side edits

- skill: new description (the skills-repo wording; evaluator design routes to
  failproofai-eval-author; no pointer to the retired evaluator skill),
  openai.yaml nested policy key, an alerts section, globals
  --timeout/--quiet/--no-color, corrected --file and --fields lists,
  session-only/key-only lists (policies test works with a key), fleet deploy
  --target/--all-agents, issues filters/facets and close = exit 1, users create
  EMAIL required, users delete/defaults, sessions import, policies
  author/backtest kinds, backfill covers judge/Jev
- public cloud-cli reference and fp-cloud-cli README match
- sessions import --format failproofai (public name for the server's wire value)
- test fixtures use neutral session ids

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): note fp Cloud CLI parity under 1.0.12

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(config): the managed skill points at the eval, alert and audit authoring skills

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: chhhee10 <chetanraghuvanshi85@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant