E2E Utils: Add retry mechanism to the REST API discovery by WunderBart · Pull Request #62331 · WordPress/gutenberg

WunderBart · 2024-06-05T11:29:43Z

Related thread: https://wordpress.slack.com/archives/C02QB2JS7/p1717143284055259

What?

Add a retry mechanism to the REST API discovery so that it doesn't fail when the env is not ready yet.

Testing Instructions

All tests should pass.
Perf tests on trunk have been failing, so I've triggered a workflow using params from the failing job with this branch as a test runner. It should pass.

github-actions · 2024-06-05T11:30:08Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: WunderBart <bartkalisz@git.wordpress.org>
Co-authored-by: kevin940726 <kevin940726@git.wordpress.org>
Co-authored-by: swissspidy <swissspidy@git.wordpress.org>
Co-authored-by: Mamaduka <mamaduka@git.wordpress.org>
Co-authored-by: GraemeF <graemef@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

kevin940726

Thank you! A few nitpicks but it's good to go IMO!

Co-authored-by: Kai Hao <kevin830726@gmail.com>

This reverts commit 47d01ba.

…est.ts" This reverts commit 111ae20.

This reverts commit 393c297.

kevin940726 · 2024-06-05T15:57:39Z

+			},
+			{
+				message: 'Failed to setup REST API.',
+				timeout: 60_000, // 1 minute.


Nit: Do we have a rough number when the REST API will be available? A minute seems like a long wait if it returns nothing. Is 5 seconds or 10 seconds enough?

Good question, and I have no idea TBH. Testing this thing takes ages, and it only fails in CI. Having said that, since this solution seems to be working across the board, I can lower the timeout value and do a couple more reruns. We can also address it in a follow-up PR since the failing perf tests keep blocking folks. What do you think?

This reverts commit efd2567.

WunderBart · 2024-06-05T16:23:16Z

-	reporter: process.env.CI
-		? './config/performance-reporter.ts'
-		: [ [ 'list' ], [ './config/performance-reporter.ts' ] ],
+	reporter: [ [ 'list' ], [ './config/performance-reporter.ts' ] ],


I think we should merge it with this change as it now prints the errors correctly, which would allow us to close #60366. What do you think @Mamaduka @kevin940726?

Sounds good. The performance test results are now attached to the summary so they won't be lost in the noise (#61450).

Ah, enabling the list reporter in this case actually means that only the error will be printed, not the tests themselves. It's because we're still using promisified exec and don't resolve with the stdout data. So, nothing in the output will change unless an error is thrown! 😄

An example error looks like this now:

▶ Running tests > Using: WordPress v6.5 > Suite: front-end-block-theme (round 1 of 1) > Branch: 9aec5fe38aa66962d1b0dc3c73bc22c7af08e3c5 > Starting environment > Running tests > gutenberg@18.5.0-rc.1 test:performance > wp-scripts test-playwright --config test/performance/playwright.config.ts front-end-block-theme Error: expect(received).not.toThrow() Matcher error: received value must be a function Received has value: undefined Call Log: - Timeout 60000ms exceeded while waiting on the predicate at ../../../packages/e2e-test-utils-playwright/src/request-utils/rest.ts:46 44 | let rootURL = ''; 45 | > 46 | await expect | ^ [47](https://github.com/WordPress/gutenberg/actions/runs/9385403378/job/25843486775#step:8:48) | .poll( 48 | async () => { 49 | [ nonce, rootURL ] = await Promise.all( [ at RequestUtils.setupRest (/tmp/wp-performance-tests/tests/packages/e2e-test-utils-playwright/src/request-utils/rest.ts:46:2) at globalSetup (/tmp/wp-performance-tests/tests/test/performance/config/global-setup.ts:26:2) Error: Command failed: npm run test:performance -- front-end-block-theme at genericNodeError (node:internal/errors:984:15) at wrappedFn (node:internal/errors:538:14) at ChildProcess.exithandler (node:child_process:422:12) at ChildProcess.emit (node:events:[51](https://github.com/WordPress/gutenberg/actions/runs/9385403378/job/25843486775#step:8:52)9:28) at maybeClose (node:internal/child_process:1105:16) at ChildProcess._handle.onexit (node:internal/child_process:305:5) { code: 1, killed: false, signal: null, cmd: 'npm run test:performance -- front-end-block-theme'

Before it was printing just the genericNodeError part, and now all the necessary info is there.

Good to know 😅

Pinging @jsnajdr, since it also means we don't need to refactor to spawn! 😄

github-actions · 2024-06-05T16:48:04Z

Flaky tests detected in 2b68944.
Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/9387734581
📝 Reported issues:

[Flaky Test] Redirection after template creation #60678 in /test/e2e/specs/site-editor/site-editor-url-navigation.spec.js

…2331) Co-authored-by: WunderBart <bartkalisz@git.wordpress.org> Co-authored-by: kevin940726 <kevin940726@git.wordpress.org> Co-authored-by: swissspidy <swissspidy@git.wordpress.org> Co-authored-by: Mamaduka <mamaduka@git.wordpress.org> Co-authored-by: GraemeF <graemef@git.wordpress.org>

Co-authored-by: WunderBart <bartkalisz@git.wordpress.org> Co-authored-by: kevin940726 <kevin940726@git.wordpress.org> Co-authored-by: swissspidy <swissspidy@git.wordpress.org> Co-authored-by: Mamaduka <mamaduka@git.wordpress.org> Co-authored-by: GraemeF <graemef@git.wordpress.org>

add retry mechanism to the REST API discovery

393c297

WunderBart marked this pull request as ready for review June 5, 2024 11:29

WunderBart requested a review from kevin940726 as a code owner June 5, 2024 11:29

WunderBart requested review from Mamaduka, oandregal and swissspidy June 5, 2024 11:30

WunderBart self-assigned this Jun 5, 2024

WunderBart requested a review from jsnajdr June 5, 2024 11:31

WunderBart mentioned this pull request Jun 5, 2024

Playwright e2e tests are configured to start wp-env but don't wait for REST API to be advertised #61627

Closed

WunderBart added [Tool] E2E Test Utils /packages/e2e-test-utils-playwright [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. labels Jun 5, 2024

kevin940726 approved these changes Jun 5, 2024

View reviewed changes

Comment thread packages/e2e-test-utils-playwright/src/request-utils/rest.ts Outdated

Comment thread packages/e2e-test-utils-playwright/src/request-utils/rest.ts Outdated

WunderBart and others added 2 commits June 5, 2024 14:19

Update packages/e2e-test-utils-playwright/src/request-utils/rest.ts

111ae20

Co-authored-by: Kai Hao <kevin830726@gmail.com>

Extend the discovery timeout to 2 minutes

47d01ba

oandregal mentioned this pull request Jun 5, 2024

Migrate theme.json based on origin #62305

Merged

swissspidy approved these changes Jun 5, 2024

View reviewed changes

WunderBart added 5 commits June 5, 2024 14:52

disable the webServer option

efd2567

tmp: print errors

7a24758

Revert "Extend the discovery timeout to 2 minutes"

702d7c1

This reverts commit 47d01ba.

Revert "Update packages/e2e-test-utils-playwright/src/request-utils/r…

4bee3ae

…est.ts" This reverts commit 111ae20.

Revert "add retry mechanism to the REST API discovery"

6d2b1c3

This reverts commit 393c297.

WunderBart force-pushed the fix/e2e-add-retry-mechanism-to-api-discovery branch from 66092fd to 5192d36 Compare June 5, 2024 14:22

retry API discovery along with login

60bf411

WunderBart force-pushed the fix/e2e-add-retry-mechanism-to-api-discovery branch from 5192d36 to 60bf411 Compare June 5, 2024 15:25

kevin940726 reviewed Jun 5, 2024

View reviewed changes

WunderBart added 2 commits June 5, 2024 18:19

add comment

94d0833

Revert "disable the webServer option"

2b68944

This reverts commit efd2567.

WunderBart commented Jun 5, 2024

View reviewed changes

WunderBart enabled auto-merge (squash) June 5, 2024 16:57

WunderBart disabled auto-merge June 5, 2024 16:57

WunderBart enabled auto-merge (squash) June 5, 2024 16:57

WunderBart merged commit 8d15c5f into trunk Jun 5, 2024

WunderBart deleted the fix/e2e-add-retry-mechanism-to-api-discovery branch June 5, 2024 16:58

github-actions Bot added this to the Gutenberg 18.6 milestone Jun 5, 2024

WunderBart mentioned this pull request Jun 5, 2024

Replace exec with spawn in performance runner script #62270

Closed

desrosj mentioned this pull request Sep 16, 2026

Repair wp/6.6 branch Actions: Fix performance-test infra crash and DataViews keyboard-nav flakiness desrosj/gutenberg#377

Closed

desrosj mentioned this pull request Sep 28, 2026

Repair wp/6.6 branch Actions: Retry REST API discovery in e2e test utils #83626

Closed

Navigation Menu

Repository navigation

E2E Utils: Add retry mechanism to the REST API discovery - #62331

E2E Utils: Add retry mechanism to the REST API discovery#62331
WunderBart merged 11 commits into
trunkfrom
fix/e2e-add-retry-mechanism-to-api-discovery

WunderBart commented Jun 5, 2024 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 5, 2024 •

edited

Loading

Uh oh!

kevin940726 left a comment

Uh oh!

Uh oh!

Uh oh!

kevin940726 Jun 5, 2024

Uh oh!

WunderBart Jun 5, 2024

Uh oh!

Uh oh!

WunderBart Jun 5, 2024

Uh oh!

Mamaduka Jun 5, 2024

Uh oh!

WunderBart Jun 5, 2024 •

edited

Loading

Uh oh!

Mamaduka Jun 5, 2024

Uh oh!

WunderBart Jun 5, 2024 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 5, 2024

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

WunderBart commented Jun 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Testing Instructions

Uh oh!

github-actions Bot commented Jun 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevin940726 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kevin940726 Jun 5, 2024

Choose a reason for hiding this comment

Uh oh!

WunderBart Jun 5, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

WunderBart Jun 5, 2024

Choose a reason for hiding this comment

Uh oh!

Mamaduka Jun 5, 2024

Choose a reason for hiding this comment

Uh oh!

WunderBart Jun 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mamaduka Jun 5, 2024

Choose a reason for hiding this comment

Uh oh!

WunderBart Jun 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 5, 2024

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

WunderBart commented Jun 5, 2024 •

edited

Loading

github-actions Bot commented Jun 5, 2024 •

edited

Loading

WunderBart Jun 5, 2024 •

edited

Loading

WunderBart Jun 5, 2024 •

edited

Loading