Skip to content

fix(kanban): a crash streak the card has left is historical, not a current crash - #110463

Open
lucascmg-vx wants to merge 1 commit into
NousResearch:mainfrom
lucascmg-vx:fix/kanban-crash-streak-historical
Open

fix(kanban): a crash streak the card has left is historical, not a current crash#110463
lucascmg-vx wants to merge 1 commit into
NousResearch:mainfrom
lucascmg-vx:fix/kanban-crash-streak-historical

Conversation

@lucascmg-vx

@lucascmg-vx lucascmg-vx commented Sep 14, 2026

Copy link
Copy Markdown

What this fixes

_rule_repeated_crashes (hermes_cli/kanban_diagnostics.py) counts a trailing crashed streak in which any non-completed/reclaimed outcome "neither counts nor breaks" it — so a newer run that ended unreported, blocked, spawn_failed or timed_out leaves the old streak intact and the finding then describes that streak in the present tense and scales its severity as though the newest run were one of the crashes.

Measured on the live vx board, 2026-09-14 01:16Z, hermes kanban diagnostics --severity critical --json (pre-fix):

card finding said newest run (task_runs)
t_2d283056 Agent crashed 5x … / "The last 5 runs ended with outcome=crashed." run 10536 unreported (crashes: 09-12 17:12–17:29)
t_8afe5089 same run 10535 unreported (crashes: 09-11)
t_6db2248b same run 10524 blocked (crashes: 09-11 17:27–17:41)
t_7a11b2a2 Agent crashed 8x … run 9252 blocked (crashes: 09-13 07:00–07:11)

unreported is the deliberate outcome for a run that exited cleanly without a terminal kanban call — so the probe was reading the conversion working, and paging critical, about crash runs two days old.

What changed

The streak is a current incident only while the task's newest run is itself one of its crashes:

  • current (newest run crashed) — byte-identical to before: same title, same detail, same severity ladder (critical at 2× threshold, else error), so a real repeating crash and the pid <n> not alive mode page exactly as they did;
  • finished — the finding keeps the streak at warning (below the critical floor the board-health page pages on) and says so, dated: Agent crashed 5x (historical, last 2026-09-12 17:29Z): … with a detail that names the newest run id/timestamp/outcome and states "so this streak is historical — not a current crash";
  • basisdata now carries is_current, newest_outcome, newest_run_id, newest_run_ended_at, last_crash_run_id, last_crash_at and runs_read (the ordered list of runs the scan read, with their outcomes), so a reader can tell "currently crashing" from "crashed historically" without re-reading task_runs.

crash_threshold and the outcome vocabulary are untouched — this distinguishes, it does not suppress.

Scope mapping (the card's four jobs, and the review's criteria)

Item Disposition
1. The page's words match the board (either stop counting runs that are no longer crashed, or date the pattern as historical) both: a finished streak no longer pages critical (branch A) and the finding is dated and labelled historical when surfaced (branch B)
2. Keep the real incidents paging — pid … not alive included unchanged code path; test_repeated_crashes_current_streak_pages_exactly_as_before + test_repeated_crashes_pid_not_alive_mode_still_pages (fixture, both directions)
3. Say which state each claim derives from data.is_current + newest_outcome/newest_run_id/last_crash_at + runs_read
4. Do not silence by widening the threshold crash_threshold and _OUTCOME_LABELS unchanged; test_repeated_crashes_threshold_and_vocabulary_are_untouched
Review criterion 5 (a test that would pass pre-fix proves nothing) the three historical cases fail on the pre-fix rule ('critical' != 'warning'); the current-path case says in its own docstring which of its assertions pass pre-fix (they are the no-regression contract)

Evidence

RED — pre-fix source, new tests kept (git checkout <base> -- hermes_cli/kanban_diagnostics.py):

$ python -m pytest tests/hermes_cli/test_kanban_diagnostics.py -k repeated_crashes -q
FAILED …::test_repeated_crashes_finished_streak_is_historical_not_current
FAILED …::test_repeated_crashes_finished_streak_behind_a_blocked_run_is_historical
FAILED …::test_repeated_crashes_finished_streak_behind_a_live_run_is_historical
FAILED …::test_repeated_crashes_current_streak_pages_exactly_as_before
FAILED …::test_repeated_crashes_pid_not_alive_mode_still_pages
FAILED …::test_repeated_crashes_threshold_and_vocabulary_are_untouched
6 failed, 1 passed, 4 deselected
AssertionError: assert 'critical' == 'warning'

GREEN — fixed source: 11 passed (tests/hermes_cli/test_kanban_diagnostics.py).

Live board, same tree and same DB copy, pre-fix vs fixed (hermes kanban --board vx diagnostics --json, HERMES_KANBAN_DB pointed at a copy of ~/.hermes/kanban/boards/vx/kanban.db):

critical, pre-fix : t_8afe5089, t_6db2248b, t_2d283056, t_7a11b2a2  (4 repeated_crashes)
critical, fixed   : (none)
all severities, fixed:
  t_406e4b7e warning is_current=false newest=10573/unreported  Agent crashed 3x (historical, last 2026-09-11 03:53Z): pid 4079594 not alive
  t_8afe5089 warning is_current=false newest=10535/unreported  Agent crashed 5x (historical, last 2026-09-11 07:30Z): pid 364463 not alive
  t_6db2248b warning is_current=false newest=10524/blocked     Agent crashed 5x (historical, last 2026-09-11 17:41Z): …
  t_2d283056 warning is_current=false newest=10536/unreported  Agent crashed 5x (historical, last 2026-09-12 17:29Z): …
  t_7a11b2a2 warning is_current=false newest=9252/blocked      Agent crashed 8x (historical, last 2026-09-13 07:11Z): …

Rendered page — the board-health notifier (internal_processes/scripts/vx_board_health_notify.py --dry-run, VX_BOARD_HEALTH_DIAG_JSON fed the two diagnostics bases above) would send 4 repeated_crashes findings pre-fix and 0 fixed, on the identical board copy.

Regression suite (the 28 failures it reports are pre-existing — exonerated)

python -m pytest tests/hermes_cli -k kanban -q28 failed, 340 passed, 1 skipped, 11142 deselected in 878.47s. None of the 28 is in tests/hermes_cli/test_kanban_diagnostics.py (11/11 green above). Checked against a pristine worktree at this PR's base (ee4452991), splitting the 28 into two batches and running each batch on both trees:

14-ids batch: base 14 passed / fixed 14 passed      → diff of FAILED sets: identical (empty)
14-ids batch: base 14 failed / fixed 14 failed      → diff of FAILED sets: identical (empty)

So 14 reproduce on the untouched base tree, and the other 14 pass on both trees when run as that batch (order-dependent flakiness of this host's environment) — no failure is attributable to this change.

Preflight (implementer checks)

  1. Selection identity — N/A: pure rule function of (task, events, runs, cfg); the run list is re-read and re-ordered per call (unchanged).
  2. Coercion inventoryint(cfg.get("crash_threshold", 2)) unchanged. New _fmt_run_ts coerces ended_at or started_at with a phrase fallback instead of raising or printing None into an operator's page; a missing newest outcome reads "is still in flight (no outcome yet)". No write path.
  3. Authz write surface — N/A: read-only, no endpoints, no permissions.
  4. Success + visible errors — the new branch is explicit text plus an inspectable basis; the finding is not silently dropped (it stays in the CLI/dashboard at warning).
  5. Off-path — N/A (no flag). The rule's exemptions (done/archived/running, unified-rule suppression) are untouched.
  6. Tests assert something live — the new cases drive the real compute_task_diagnostics entry point with the board's own run shapes (ids, unreported/blocked, pid <n> not alive), and assert the rendered title/detail strings, not just severity.
  7. Cross-repo API contract — the only consumer of this diagnostic is internal_processes/scripts/vx_board_health_notify.py: it pages --severity critical and fingerprints data minus its volatile keys. Additive data keys mean a one-time re-page per currently-critical crash finding (a changed fingerprint is a changed finding); the warning severity removes exactly the false pages. No consumer needs a companion change to keep working; the host carry is carded (below).
  8. Backend staging & write safety — N/A: no external system is written; diagnostics are read-only and the live-board work above ran against a copy of the board DB.
  9. Redirect / open param — N/A.
  10. State + scope completenesscount == data["consecutive_crashes"], and the basis travels with the finding (runs_read); no hand-written duplicate of a derived value.
  11. SQL reserved words — N/A: no SQL added.
  12. Rate-limited endpoints / confirm-then-write races — N/A: no HTTP, no writes.

Adjacent observations (out of task scope)

  • hermes_cli/kanban_db.py's task_runs.outcome DDL comment still lists only completed | blocked | crashed | timed_out | spawn_failed | gave_up | reclaimed; unreported is absent there because it is introduced by the still-open fix(kanban): park a clean exit with no terminal call, carrying its last-known state #110322, not by this change. Left alone deliberately: this PR must not depend on that one merging.
  • The dispatcher host runs upstream + 18 local patches, so this PR alone does not change the page it renders. The carry is carded as kanban t_79c3fde9 (same route as t_ecf2866f / The-ValueExchange/internal_processes#492).

…rrent crash

`_rule_repeated_crashes` counted a trailing `crashed` streak in which any other
outcome (`unreported`, `blocked`, `spawn_failed`, `timed_out`) "neither counts
nor breaks" it, then described that streak in the present tense and scaled its
severity as if the newest run were one of the crashes.

On the 2026-09-14 board four cards whose newest run was `unreported` (the
deliberate clean-exit-without-a-terminal-call outcome) or `blocked` paged
`critical` with "The last 5 runs ended with outcome=crashed" about crash runs
two days old - the crash probe reading the conversion working as a live
incident: every qualifying run since the process swap was `unreported`, zero
`crashed`.

The streak is now reported as a current incident only while the task's newest
run is itself a crash:

* current (newest run `crashed`) - unchanged, byte-identical: same title, same
  detail, same severity ladder (critical at 2x threshold), so the `pid <n> not
  alive` mode and every real repeating crash page exactly as before;
* finished - the finding keeps the streak at `warning` (below the critical floor
  the board-health page pages on) and says so with the date of the last crash,
  the newest run's id/timestamp/outcome, and the runs the scan read in `data`,
  so a reader can tell "currently crashing" from "crashed historically".

`crash_threshold` and the outcome vocabulary are untouched: this distinguishes,
it does not suppress.
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cron Cron scheduler and job management labels Sep 14, 2026
@lucascmg-vx

Copy link
Copy Markdown
Author

VERDICT: APPROVE (reviewed_sha: 40914a9)

PR: #110463, base main, head fix/kanban-crash-streak-historical, exactly 2 files (+250/-13). Base check passes (hermes-agent targets main correctly; no unrelated files).

Scope diff (reviewed card t_5cf69692, 4 jobs): 1. words match board -> delivered (a finished streak leaves critical for a dated warning historical finding); 2. real incidents still page -> delivered (current-crash path byte-identical incl. severity ladder; pid not alive fixture proves the live mode); 3. inspectable basis -> delivered (data: is_current, newest_outcome/newest_run_id/newest_run_ended_at, last_crash_run_id/last_crash_at, runs_read with ordered outcomes); 4. no threshold-widening shortcut -> delivered (crash_threshold and outcome vocabulary untouched, pinned by test). Adjacent notes: ruled out of card scope (correctly left out; no gate required) -- the task_runs.outcome DDL comment belongs to still-open #110322, and the host carry is the carded delivery path, correctly not part of this upstream diff.

Lens table (all eight run):

  1. spec/acceptance fidelity -- pass. 4/4 card jobs delivered; current path byte-identical (title, detail, severity ladder verified against the diff); historical wording dates the last crash and names the newest run id/ts/outcome.
  2. regression/blast radius -- pass. Only consumer is internal_processes/scripts/vx_board_health_notify.py (--severity critical + fingerprint over data minus volatile keys): historical findings drop below the page floor (verified 4 -> 0 on the rendered page); a persisting current crash re-pages at most once on the additive keys, which the PR body discloses (preflight 7). No env, route, schema, or cross-VX surface touched.
  3. security -- none (read-only rule; no endpoints, secrets, or authz surface; new data keys carry run ids/outcomes/timestamps only).
  4. edge cases -- none. Empty runs return before ordered[0] is touched (rule exceptions are also swallowed by compute_task_diagnostics); completed/reclaimed still break the streak; any non-crashed newest (unreported/blocked/spawn_failed/timed_out/in-flight) reads historical, which is the card's rule; _fmt_run_ts never prints None (phrase fallback, in-flight wording asserted by test).
  5. tests that assert -- pass. 6 new cases drive the real compute_task_diagnostics entry point with the board's own run shapes and assert rendered title/detail strings. RED/GREEN verified by this reviewer (below). The current-path assertions that pass pre-fix are declared in-test as the no-regression contract; every historical assertion fails pre-fix.
  6. usability -- pass. Rendered page re-read (not only code): replay --dry-run on the same board copy goes 4 repeated_crashes findings -> 0; the CLI/dashboard keeps the streak at warning, dated, with its basis.
  7. VX invariants -- pass. No truncation of user data (runs_read capped at 50 is diagnostic basis, count stays exact); no invented defaults; no secrets; conventional commit; correct base.
  8. visual fidelity -- none (no design authority named in the task).

Findings: none.

Tests run by this reviewer (worktree /home/ubuntu/hw-crash-hist, head verified == reviewed sha):

  • python3 -m pytest tests/hermes_cli/test_kanban_diagnostics.py -q on the fixed tree: 11 passed.
  • Same file with hermes_cli/kanban_diagnostics.py checked out at base ee4452991 (new tests kept): 6 failed, 1 passed, 4 deselected -- the historical/current-basis assertions fail pre-fix as required (worktree restored to the reviewed sha afterwards, git status clean, HEAD == reviewed sha).
  • Rendered-page replay /home/ubuntu/t_5cf69692_scratch/render_page2.sh (same tree, same DB copy /tmp/vx_board_copy_5cf69692.db): PAGE(before) sends 4 repeated_crashes (t_8afe5089, t_6db2248b, t_2d283056, t_7a11b2a2); PAGE(after) sends 0.

Bugbot triage: did not run; vx-review substitutes for the triage record. At ~20min after PR creation (last update 2026-09-14T01:52:52Z): 0 commit check-runs, gh pr checks reports none, no reviews, no comments. Substitution basis is the independent verification above (own RED/GREEN run plus the rendered-page replay), not the implementer's report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants