fix(kanban): a crash streak the card has left is historical, not a current crash - #110463
fix(kanban): a crash streak the card has left is historical, not a current crash#110463lucascmg-vx wants to merge 1 commit into
Conversation
…rrent crash `_rule_repeated_crashes` counted a trailing `crashed` streak in which any other outcome (`unreported`, `blocked`, `spawn_failed`, `timed_out`) "neither counts nor breaks" it, then described that streak in the present tense and scaled its severity as if the newest run were one of the crashes. On the 2026-09-14 board four cards whose newest run was `unreported` (the deliberate clean-exit-without-a-terminal-call outcome) or `blocked` paged `critical` with "The last 5 runs ended with outcome=crashed" about crash runs two days old - the crash probe reading the conversion working as a live incident: every qualifying run since the process swap was `unreported`, zero `crashed`. The streak is now reported as a current incident only while the task's newest run is itself a crash: * current (newest run `crashed`) - unchanged, byte-identical: same title, same detail, same severity ladder (critical at 2x threshold), so the `pid <n> not alive` mode and every real repeating crash page exactly as before; * finished - the finding keeps the streak at `warning` (below the critical floor the board-health page pages on) and says so with the date of the last crash, the newest run's id/timestamp/outcome, and the runs the scan read in `data`, so a reader can tell "currently crashing" from "crashed historically". `crash_threshold` and the outcome vocabulary are untouched: this distinguishes, it does not suppress.
|
VERDICT: APPROVE (reviewed_sha: 40914a9) PR: #110463, base Scope diff (reviewed card t_5cf69692, 4 jobs): 1. words match board -> delivered (a finished streak leaves Lens table (all eight run):
Findings: none. Tests run by this reviewer (worktree /home/ubuntu/hw-crash-hist, head verified == reviewed sha):
Bugbot triage: did not run; vx-review substitutes for the triage record. At ~20min after PR creation (last update 2026-09-14T01:52:52Z): 0 commit check-runs, |
What this fixes
_rule_repeated_crashes(hermes_cli/kanban_diagnostics.py) counts a trailingcrashedstreak in which any non-completed/reclaimedoutcome "neither counts nor breaks" it — so a newer run that endedunreported,blocked,spawn_failedortimed_outleaves the old streak intact and the finding then describes that streak in the present tense and scales its severity as though the newest run were one of the crashes.Measured on the live
vxboard, 2026-09-14 01:16Z,hermes kanban diagnostics --severity critical --json(pre-fix):task_runs)t_2d283056Agent crashed 5x …/ "The last 5 runs ended with outcome=crashed."unreported(crashes: 09-12 17:12–17:29)t_8afe5089unreported(crashes: 09-11)t_6db2248bblocked(crashes: 09-11 17:27–17:41)t_7a11b2a2Agent crashed 8x …blocked(crashes: 09-13 07:00–07:11)unreportedis the deliberate outcome for a run that exited cleanly without a terminal kanban call — so the probe was reading the conversion working, and pagingcritical, about crash runs two days old.What changed
The streak is a current incident only while the task's newest run is itself one of its crashes:
crashed) — byte-identical to before: same title, same detail, same severity ladder (criticalat 2× threshold, elseerror), so a real repeating crash and thepid <n> not alivemode page exactly as they did;warning(below thecriticalfloor the board-health page pages on) and says so, dated:Agent crashed 5x (historical, last 2026-09-12 17:29Z): …with a detail that names the newest run id/timestamp/outcome and states "so this streak is historical — not a current crash";datanow carriesis_current,newest_outcome,newest_run_id,newest_run_ended_at,last_crash_run_id,last_crash_atandruns_read(the ordered list of runs the scan read, with their outcomes), so a reader can tell "currently crashing" from "crashed historically" without re-readingtask_runs.crash_thresholdand the outcome vocabulary are untouched — this distinguishes, it does not suppress.Scope mapping (the card's four jobs, and the review's criteria)
crashed, or date the pattern as historical)critical(branch A) and the finding is dated and labelled historical when surfaced (branch B)pid … not aliveincludedtest_repeated_crashes_current_streak_pages_exactly_as_before+test_repeated_crashes_pid_not_alive_mode_still_pages(fixture, both directions)data.is_current+newest_outcome/newest_run_id/last_crash_at+runs_readcrash_thresholdand_OUTCOME_LABELSunchanged;test_repeated_crashes_threshold_and_vocabulary_are_untouched'critical' != 'warning'); the current-path case says in its own docstring which of its assertions pass pre-fix (they are the no-regression contract)Evidence
RED — pre-fix source, new tests kept (
git checkout <base> -- hermes_cli/kanban_diagnostics.py):GREEN — fixed source:
11 passed(tests/hermes_cli/test_kanban_diagnostics.py).Live board, same tree and same DB copy, pre-fix vs fixed (
hermes kanban --board vx diagnostics --json,HERMES_KANBAN_DBpointed at a copy of~/.hermes/kanban/boards/vx/kanban.db):Rendered page — the board-health notifier (
internal_processes/scripts/vx_board_health_notify.py --dry-run,VX_BOARD_HEALTH_DIAG_JSONfed the two diagnostics bases above) would send 4repeated_crashesfindings pre-fix and 0 fixed, on the identical board copy.Regression suite (the 28 failures it reports are pre-existing — exonerated)
python -m pytest tests/hermes_cli -k kanban -q→28 failed, 340 passed, 1 skipped, 11142 deselected in 878.47s. None of the 28 is intests/hermes_cli/test_kanban_diagnostics.py(11/11 green above). Checked against a pristine worktree at this PR's base (ee4452991), splitting the 28 into two batches and running each batch on both trees:So 14 reproduce on the untouched base tree, and the other 14 pass on both trees when run as that batch (order-dependent flakiness of this host's environment) — no failure is attributable to this change.
Preflight (implementer checks)
(task, events, runs, cfg); the run list is re-read and re-ordered per call (unchanged).int(cfg.get("crash_threshold", 2))unchanged. New_fmt_run_tscoercesended_at or started_atwith a phrase fallback instead of raising or printingNoneinto an operator's page; a missing newest outcome reads "is still in flight (no outcome yet)". No write path.warning).done/archived/running, unified-rule suppression) are untouched.compute_task_diagnosticsentry point with the board's own run shapes (ids,unreported/blocked,pid <n> not alive), and assert the rendered title/detail strings, not just severity.internal_processes/scripts/vx_board_health_notify.py: it pages--severity criticaland fingerprintsdataminus its volatile keys. Additivedatakeys mean a one-time re-page per currently-critical crash finding (a changed fingerprint is a changed finding); thewarningseverity removes exactly the false pages. No consumer needs a companion change to keep working; the host carry is carded (below).count == data["consecutive_crashes"], and the basis travels with the finding (runs_read); no hand-written duplicate of a derived value.Adjacent observations (out of task scope)
hermes_cli/kanban_db.py'stask_runs.outcomeDDL comment still lists onlycompleted | blocked | crashed | timed_out | spawn_failed | gave_up | reclaimed;unreportedis absent there because it is introduced by the still-open fix(kanban): park a clean exit with no terminal call, carrying its last-known state #110322, not by this change. Left alone deliberately: this PR must not depend on that one merging.t_79c3fde9(same route ast_ecf2866f/ The-ValueExchange/internal_processes#492).