feat: report a skipped check as skipped, not as passed - #537
Conversation
A rule that never ran was reported as a pass. commit-check-action#258 is the visible cost: every check on it rendered as a green tick and the summary announced "All 5 checks passed", when in fact nothing had been validated -- the author is dependabot[bot], which the org config lists in ignore_authors. A bypassed policy was indistinguishable from an enforced one, in the JSON, in the Python API, and in anything rendering them. Measured on that exact case before the change: every rule came back "status": "pass" with "value": "", the empty value being the only trace that a skip had happened, and an incidental one at that. Adds ValidationResult.SKIP and returns it from the guards that already decide this -- _should_skip_commit_validation, _should_skip_branch_ validation, and the ignored-author branch of _validate_author. Those helpers are named for skipping; they were simply reporting it as PASS. validate_all_detailed maps SKIP to "skip" and forces the value empty, since a rule that did not run examined nothing. Overall status is "skip" only when every check skipped; one real verdict still yields "pass" or "fail". Only "fail" is an error, so the exit code is unchanged for existing callers and code branching on status == "fail" keeps working. The overall-status rule was duplicated between the CLI's --format json and the Python API, which is how the CLI kept printing "pass" for a fully skipped run after the API had been fixed. It now lives once, in engine.overall_status(), used by both. That also fixes a latent bug in the CLI's exit code: `0 if overall == "pass" else 1` would have turned a skipped run into a failure. Verified end to end in a repository shaped like #258 -- same repo, same config, only the author differing: dependabot[bot] -> overall "skip", every check "skip", exit 0 a human -> overall "pass", values reported, exit 0 a human, bad msg -> exit 1 The twelve existing tests that asserted PASS on these paths are all named for skipping (ignored_author, skips_validation, skip_conditions); they now assert SKIP. Four new API tests pin the behaviour, including a control that only the author differs so the skip test cannot pass by the rules having quietly stopped running for everyone. Reverting the skip reporting turns the first of them red. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn
|
Warning Review limit reached
Next review available in: 55 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughThe validation engine now distinguishes skipped checks from passing checks. API and JSON results preserve ChangesSkipped validation status
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant JSONCaller
participant commit_check_main
participant commit_check_engine
JSONCaller->>commit_check_main: run JSON validation
commit_check_main->>commit_check_engine: overall_status(outcomes)
commit_check_engine-->>commit_check_main: fail, skip, or pass
commit_check_main-->>JSONCaller: JSON result and exit code
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #537 +/- ##
==========================================
+ Coverage 97.72% 97.97% +0.25%
==========================================
Files 12 12
Lines 1272 1281 +9
==========================================
+ Hits 1243 1255 +12
+ Misses 29 26 -3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@commit_check/api.py`:
- Around line 64-69: Update the combined-result paths in validate_author and
validate_all to use the shared _build_result reducer instead of independently
defaulting to "pass", preserving "skip" when all nested checks skip. Add
coverage for all-skipped calls to both validate_author(name=..., email=...) and
validate_all(...).
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5c0625d5-1288-4280-828a-e1566490519c
📒 Files selected for processing (6)
README.mdcommit_check/api.pycommit_check/engine.pycommit_check/main.pytests/api_test.pytests/engine_test.py
Review caught a third and fourth copy of the reduce-to-overall rule that the first commit missed. validate_author(name=..., email=...) merges two separate runs, and validate_all() merges up to three; each combined its checks with a private `"fail" if any(...) else "pass"`, so a call in which every nested check skipped still reported "pass" -- the exact defect the skip status exists to prevent, surviving in the two entry points most likely to be called by automation. overall_status() now takes plain status strings rather than CheckOutcome objects, which is what lets every caller share it: the CLI, _build_result, and both combined paths, which hold already-serialised dicts. Four copies of this rule is how it drifted in the first place, so there is now one. Both new tests fail if the per-path rule is restored. Also covers the two skip branches codecov flagged, in BodyValidator and in CommitTypeValidator's non-ignore_authors path. Each comes with a control that changes only the author, so neither can pass by the rule having quietly stopped running for everyone. No line added by this PR is left uncovered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn
|
Merging this PR will not alter performance
Performance Changes
Comparing Footnotes
|
The pin was v2.13.0, whose requirements.txt installs commit-check 2.13.1. That engine cannot resolve a rebase target existing only as origin/main in a pull request checkout, so it reported "not rebased" for a branch that was -- the false CC202 this workflow has carried since its first run. v2.13.1 of the action installs commit-check 2.13.4, which carries the merge-base fix from #532. Confirmed by reading requirements.txt at the tag rather than assuming the action version tracks the engine version: v2.13.0 (124de73) -> commit-check==2.13.1 v2.13.1 (562a184) -> commit-check==2.13.4 The branch was already rebased before this change -- git merge-base --is-ancestor origin/main HEAD returned true while CC202 still failed -- so the failure was the engine, not the branch, and rebasing again could never have fixed it. Two things ride along on the newer engine. Skipped checks now report as skipped rather than as passes (#537), so a run bypassed by ignore_authors says so instead of showing green ticks over nothing. And the imperative whitelist goes from 396 verbs to 529, retiring a class of false CC003 -- 2.13.1 rejected "treat", which #527 had added three releases earlier. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn



The problem
A rule that never ran was reported as a pass.
commit-check-action#258 is the visible cost — every check rendered as a green tick under "All 5 checks passed", when nothing had been validated. That PR's author is
dependabot[bot], which the org config lists inignore_authors:Measured on that exact case before this change, every rule came back:
{ "rule_id": "CC001", "check": "message", "status": "pass", "value": "" }The empty
valuewas the only trace that a skip had happened — and an incidental one, not a signal anybody could rely on. A bypassed policy was indistinguishable from an enforced one in the JSON, in the Python API, and in anything rendering them.What changed
Adds
ValidationResult.SKIP, returned from the guards that already make this decision —_should_skip_commit_validation,_should_skip_branch_validation, and the ignored-author branch of_validate_author. Those helpers are named for skipping; they were simply reporting the outcome asPASS.validate_all_detailedmapsSKIPto"skip"and forcesvalueempty, since a rule that did not run examined nothing.Overall status is
"skip"only when every check skipped. One real verdict still yields"pass"or"fail". Only"fail"is an error, so exit codes are unchanged for existing callers and code branching onstatus == "fail"keeps working.A duplicated rule, and a latent exit-code bug
The overall-status rule existed in two places — the CLI's
--format jsonand the Python API. That is exactly how the CLI kept printing"pass"for a fully skipped run after the API had been fixed, so it now lives once inengine.overall_status()and both call it.That also fixes a bug that was waiting to happen. The CLI ended with:
which would have turned a skipped run into a failing exit code the moment a third status existed. It is now
1 if overall == "fail" else 0.Verification
End to end in a repository shaped like #258 — same repo, same config, only the author differing:
dependabot[bot]skipskippasspass, values reportedfail518 passed, ruff clean and formatted.PASSon these paths are all named for skipping (..._ignored_author,..._skips_validation,..._skip_conditions); they now assertSKIP.json.loads, matching the transcript standard set in docs: refresh README sample output to match what commit-check prints #535.One pre-existing failure is unrelated and untouched:
config_test.py::test_load_config_file_permission_errorfails on a clean checkout ofmainin this environment too, because the suite runs as root and a permission-denied path cannot be provoked.Follow-up
The action still renders
skipas a tick, since it only knowspass/fail. That side is a separate PR againstcommit-check-action.🤖 Generated with Claude Code
https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn
Generated by Claude Code
Summary by CodeRabbit
New Features
skipstatuses for overall validation results and individual checks.skip; failures take precedence when present.Documentation