Repository navigation
Report retry-confirmed failures as regressions, not flakiness - #78223
Open
TheRealAlexxx wants to merge 1 commit into
Open
TheRealAlexxx wants to merge 1 commit into
TheRealAlexxx wants to merge 1 commit into
Conversation
https://bugs.webkit.org/show_bug.cgi?id=321787 Reviewed by NOBODY (OOPS!). When a test fails in both the initial run and the retry run but with different result types, run-webkit-tests counts it as a regression in the summarized results (num_regressions, report REGRESSION) and exits non-zero, yet the buildbot-style report lists it under "Unexpected flakiness" and prints no Regressions section, because BuildBotPrinter.print_unexpected_results ignores the report field and treats any test with more than one actual result type as flaky. Bots then report, for example, num_regressions=1 alongside "reports: 2 FLAKY, 0 REGRESSION" for the same run. See https://bugs.webkit.org/show_bug.cgi?id=321787. BuildBotPrinter already documents that its groups must match summarize_results(), so use the report field it writes: tests marked REGRESSION are grouped with the regressions and tests marked FLAKY with the flaky results, keyed by the first actual result type in both cases. The existing pass, pixel-retry, multi-result, and single-result fallbacks are unchanged, as are the exit code and the summarized results themselves. A new test summarizes a run where a test times out in the initial run and crashes on retry, then checks that the buildbot report contains a Regressions section listing the test and no flakiness section, matching num_regressions=1. ``` PYTHONPATH=Tools/Scripts python3 -m unittest webkitpy.layout_tests.views.buildbot_results_unittest python3 Tools/Scripts/check-webkit-style Tools/Scripts/webkitpy/layout_tests/views/buildbot_results.py Tools/Scripts/webkitpy/layout_tests/views/buildbot_results_unittest.py ```
Collaborator
|
EWS run on current version of this PR (hash 14ba693) Details |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Report retry-confirmed failures as regressions, not flakiness
When a test fails in both the initial run and the retry run but with different result types, run-webkit-tests counts it as a regression in the summarized results (num_regressions, report REGRESSION) and exits non-zero, yet the buildbot-style report lists it under "Unexpected flakiness" and prints no Regressions section, because BuildBotPrinter.print_unexpected_results ignores the report field and treats any test with more than one actual result type as flaky. Bots then report, for example, num_regressions=1 alongside "reports: 2 FLAKY, 0 REGRESSION" for the same run. See https://bugs.webkit.org/show_bug.cgi?id=321787.
BuildBotPrinter already documents that its groups must match summarize_results(), so use the report field it writes: tests marked REGRESSION are grouped with the regressions and tests marked FLAKY with the flaky results, keyed by the first actual result type in both cases. The existing pass, pixel-retry, multi-result, and single-result fallbacks are unchanged, as are the exit code and the summarized results themselves.
A new test summarizes a run where a test times out in the initial run and crashes on retry, then checks that the buildbot report contains a Regressions section listing the test and no flakiness section, matching num_regressions=1.