ci(docs-infra): test_aio/test_aio_local jobs occasionally fail due to low PWA score

Recently, the test_aio/test_aio_local jobs started flaking relatively often due to the PWA score being too low (71% vs 95% required).

Example failures: 284426, 284459, 284483, 284494, 284556, 284583.

Sample output:

Audit scores:
  - Performance:          42 / 100
  - Accessibility:        94 / 100
  - Best Practices:       93 / 100
  - SEO:                  82 / 100
  - Progressive Web App:  71 / 100

Lighthouse PWA score:
  - Expected:  95 / 100 (or higher)
  - Actual:    71 / 100

Error: PWA score is too low. (71 < 95)

For reference, here are two Lighthouse reports, one with PWA score 100% and one with 71%:

ci(docs-infra): temporarily lower the min required PWA score for loca…

By comparing the reports, the difference is that the SW is not detected in the failing runs (possibly due to it being registered after the app stabilizes).
Related Lighthouse issues: #2924, #5527, #6342

It turned out to be related to the old Chrome version used on CI (v70). Switching to puppeteer@1.14.0 (which brings in the equivalent of Chrome/75.0.3738.0), makes the tests pass reliably.

(Now we just need to figure out what is the best way to update Chrome across our CI jobs 😁)

EDIT:
Here is a PoC of running the PWA score tests against the latest Chrome version (but we should have a more generic way - potentially based on puppeteer).

EDIT (2):
Actually, that doesn't seem to be entirely true: #29921 (comment) 😞

Capturing some more info for future reference:

It seems that the failurs are somehow related to the particular job instance/VM instance. For example, ssh'ing into a failing job/VM (e.g. #29921's 285524), will consistently fail the PWA tests. (Consistently, as in 50/50 failures.)

In that case, Lighthouse fails to detect the SW (and I indeed see from the server logs that the SW script was not requested). At the same time, loading the target URL in the same puppeteer instance, I can see the SW script being loaded and executing correctly (based on the server logs). For example:

// Open a browser.
const browser = await puppeteer.launch({defaultViewport: null, headless: false});
const flags = {port: new URL(browser.wsEndpoint()).port};

// Open a page on the same browser to see if the SW is loaded.
const page = await browser.newPage();
await page.goto(url);
await new Promise(resolve => setTimeout(resolve, 10000));  // SW is loaded :)
await p.close();

// Run Lighthouse.
await lighthouse(url, flags, config);  // SW is not loaded :(

(FWIW, I even tried removing node_modules/ (both in root and in aio/), re-running yarn --cwd=aio build and nothing changed 😞)

For completeness:
Ssh'ing into a passing job/VM (e.g. #29921's 285531) and running the PWA tests failed 2/15 times (i.e. still failed but with a much lower fewquency) 🤷‍♂️

We have a work-around (thx to GoogleChrome/lighthouse#5527 (comment)) 🎉
PR: #29953

ci(docs-infra): increase wait for SW on localhost to avoid CI flakes

ci(docs-infra): temporarily lower the min required PWA score for loca…

ci(docs-infra): increase wait for SW on localhost to avoid CI flakes (#…

ci(docs-infra): increase wait for SW on localhost to avoid CI flakes (a…

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}

Navigation Menu

Repository navigation

ci(docs-infra): `test_aio`/`test_aio_local` jobs occasionally fail due to low PWA score #29910

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ci(docs-infra): test_aio/test_aio_local jobs occasionally fail due to low PWA score #29910

Description

Activity

gkalpak commented on Apr 16, 2019

gkalpak commented on Apr 16, 2019

gkalpak commented on Apr 16, 2019

gkalpak commented on Apr 16, 2019

gkalpak commented on Apr 17, 2019

angular-automatic-lock-bot commented on Sep 14, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

ci(docs-infra): `test_aio`/`test_aio_local` jobs occasionally fail due to low PWA score #29910