Repository navigation
Browserstack and/or Sauce Labs should be required CI jobs #5407
Description
They are only optional today. but to get there, they need to be made reliable. We are currently running 19 browsers in SL and 15 in BS, and monitoring the output.
Thanks to #4417 which landed yesterday, the 2 are both green if everything goes well. But it doesn't happen often... We are experiencing 2 kind of errors:
- failure: browser does not start or disconnects or fails or is very slow, so the campaign is not fully run
- flake: a test that should pass fails because it reaches the timeout
To improve the situation, I've looked at about 60 builds in the last 10 days (all builds in master and presubmit-* after #4417). That's about 120 jobs or 2000 campaigns or 4M+ unit tests.
Most jobs failed (90%) due to 123 failures and 74 flakes. This is bad, but:
- 70% of the failures come from iOS9. Our campaign doesn't work well in this simulator, be it in SL,in BS or locally. In 80% of the campaigns, Safari mobile freezes silently, fails to load resources (XHR error), or is very slow.
- 80% of the flakes come from 5 unit tests which can "randomly" fail in any browser. They are:
PostMessageBusSink should send messages immediatly when run outside the zone
element probe should provide a global function to inspect elements
MdButton button[md-button] should disable the button
MdButton a[md-button] should remove disabled anchors from tab order
MdButton button[md-button] should handle a click on the button
It also interesting to note that there is a trend which shows that SL is better with Windows, while BS is better with Mac. This applies to both failures and flakes.
So an idea would be split browsers among the 2 providers to improve the overall reliability.
Next steps:
- remove iOS9 from CI until it is made reliable, if possible. There is no problem with the desktop version, so it shouldn't be an issue
- increase the timeout of the 5 identified flaky tests
- continue monitoring and apply the split if the trend is confirmed
Activity
JanStureNielsen commentedon Nov 20, 2015
@Mlaval Are Angular's BS&SL test output available somewhere? Do we know if the BS&SL failures for iOS 9 and Safai mobile are an accurate reflection of a real-user experience in these environments?
marclaval commentedon Nov 21, 2015
There are no tests failures in fact. When the campaign starts correctly, it is successful and fast.
As far as I know, there is no problem running an application in Safari Mobile 9.
marclaval commentedon Nov 30, 2015
Some updates:
Once all these PRs are integrated, a last round of monitoring should be done before making these jobs as required.
marclaval commentedon Dec 7, 2015
After a few days with the 3 PRs integrated, the situation is better but far from a level which would allow us to make these jobs as required in CI.
Looking at the last 130 builds, only 55% of them were successful in both SL and BS.
12% fail because of flakes and 33% because of failures.
marclaval commentedon Dec 8, 2015
More data on the last 188 builds: 52% of success.
Failures per browser:
Flakes per browser:
There is no obvious flake source. These 35 flakes have been generated by 28 different tests.
3 remaining items
DzmitryShylovich commentedon Dec 11, 2016
@Mlaval I believe they are already required.
DzmitryShylovich commentedon Dec 15, 2016
@pkozlowski-opensource close this one?
angular-automatic-lock-bot commentedon Sep 10, 2019
This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.
Read more about our automatic conversation locking policy.
This action has been performed automatically by a bot.