Blocks E2E: Fix db connection issues - #51658

Test using WordPress Playground

The changes in this pull request can be previewed and tested using a WordPress Playground instance.
WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Test this pull request with WordPress Playground.

Note that this URL is valid for 30 days from when this comment was last updated. You can update it by closing/reopening the PR or pushing a new commit.

Size Change: 0 B

Total Size: 2.38 MB

_{compressed-size-action}

This issue is keeping me awake. After a bunch of failed attempts (see commit history), I thought I had it by disposing of the request cache – I figured we're running out of resources because we're making a lot of requests. Unfortunately, it failed again after a couple of reruns (debugging this one takes forever). I obviously tried replicating the issue locally, without luck. It always passes, even when running entire shards. Anyway, it's time to ask for some help. Here's what I've learned so far:

• It breaks only in CI.

• It can break a single test and return to normal in subsequent tests.

• It can break halfway through the suite and stay broken until the next suite.

• It's not tied to a specific test.

• It's usually thrown within the product-collection.block_theme.spec.js suite.

  I can't see anything out of order with that suite, apart from the fact that it's the biggest one we have.

• It always breaks within the beforeEach hook.

• The error is always a 500 database disconnected one:

   {
     code: 'wp_die',
     message: '<h1>Error establishing a database connection</h1>',
     data: { status: 500 },
     additional_errors: []
   }

• Before the 500 response is thrown (e.g., after trying to activate the plugin or navigate), a bunch of plugin activation requests from an unrelated suite (!!!) are made.

  This seems to be what leads to the failure, but I haven't yet found the culprit and it still doesn't make any sense to me. How is a different suite bleeding into this one??

Example of a failed run.

It usually happens within the shard no. 4, which runs the following suites in the following order:

1. compatibility-layer.block_theme.spec.ts ✅ PASS
2. extensibility-events.block_theme.spec.ts ✅ PASS
3. inspector-controls.block_theme.spec.ts ✅ PASS
4. product-collection.block_theme.spec.ts ❌ FAIL

The (currently) failing test is this one:

product-collection/product-collection.block_theme.spec.ts:1014 › Testing "usesReference" argument in "registerProductCollection" › Collection "My Custom Collection - Cart Context" should not show preview label in Product Catalog template

From the failed test's trace.zip, the network requests are as follows:

The part that is completely out of order is the request to activate the product-collection-compatibility-layer plugin, which is only activated within the compatibility-layer.block_theme.spec.ts that has already passed three suites ago. Also, the register-product-collection-tester plugin activation should not be called multiple times if the first request was successful. Because of that, I thought it might be something with re-authorization where we repeat the request if the user is not yet authorized, but the requests are successful (until 500), so the user is authorized and yet they're still repeated.

As I write this, I’m in the process of cleaning up the numerous Product Collection E2E tests, particularly the product-collection.block_theme.spec.js file, and splitting them up further. These tests currently occupy the CI for the longest time, which is problematic. Many of them involve nested loops and are heavily parameterized, making them challenging to debug and follow. I’m working on simplifying things in the hope that I might either spot an issue or inadvertently fix it. If anyone has a better approach, I’m all ears! 🙏

The following issues might be related:

• Intermittent SQLite Database Disconnects WordPress/wordpress-playground#1596
• wp-env Error while running docker compose command WordPress/gutenberg#62242

cc: @gigitux @adimoldovan @adamziel @kevin940726 @Mamaduka @talldan

I'm pinging some core folks as it might be related to the @wordpress/e2e-test-utils-playwright package.

I've not seen a similar error in the Gutenberg repo, but I started noticing tests marked as flaky due to the apiRequestContext.fetch: socket hang up error. I wonder if these are related.

Thanks for working on this and for your great exploration, @WunderBart! 🙇

Did we try to use this patch and see if the issue disappears? In this way, we can ensure that the issue isn't related to WordPress.

Did we try to use this patch and see if the issue disappears? In this way, we can ensure that the issue isn't related to WordPress.

@gigitux yep, tried it in 8a59383. The DB kept disconnecting, unfortunately :(

I think we need to narrow down the issue. Is this a problem with wp-env and CIs or the Playwright issue?

Unfortunately, I don't have enough expertise with wp-env; I only use it to run the tests.

cc @noahtallen

It could be a dumb thought but I noticed that the last time that we updated @wordpress/e2e-test-utils-playwright and @wordpress/env was two months ago.

If the 8a59383 didn't work, it means that could be some regression related to WordPress/WooCommerce?

I think we need to narrow down the issue.

💯 I agree!

Hi @lanej0, @louwie17, @adimoldovan,

Apart from reviewing the code changes, please make sure to review the testing instructions and verify that relevant tests (E2E, Unit, Integration, etc.) have been added or updated as needed.

You can follow this guide to find out what good testing instructions should look like:
https://github.com/woocommerce/woocommerce/wiki/Writing-high-quality-testing-instructions

Thanks for the feedback so far, folks. I was trying to narrow it down but I just could not repro it locally yet. Will keep trying.

In the meantime, after the cleanup, I restarted the workflow several times, and the usual suspects didn't fail. Let's hope it stays like that. Marking this one as ready for review, let's see how it goes! 🙌