Add Playwright tests - #9

redirect to the correct URL

Wouldn't the plugin-issued redirect be the "correct" result? As in, the expected one for that plugin? Are we collecting any errors from that page?

Also the debug.log errors may be significant. And whether we see the error reporting modal or any errors in devtools during this entire process. And whether the plugin remains active after the installation. All of those together tell us about compatibility. Also – what other factors do you think should we consider here before we declare the plugin as operational in Playground?

Also – what other factors do you think should we consider here before we declare the plugin as operational in Playground?

This is a question I'm trying to answer right now in the new tester, and I'm not sure. Plugins can significantly change WordPress, so it's hard to determine what must work without it being plugin specific.
At a minimum, I think that we should be able to load wp-admin, open a post edit page, and save options.
All these operations shouldn't produce Playground crashes.
Ideally, there shouldn't be any PHP fatals, but some plugins have bugs, so these errors require manual review.

Ultimately, I think it's important to have the same tests run in an old and a new version of Playgoround to demonstrate that we reduced the error rate.

Wouldn't the plugin-issued redirect be the "correct" result? As in, the expected one for that plugin? Are we collecting any errors from that page?

This part of the test checks if the plugin is active by going to the /wp-admin/plugins.php page.

Sometimes a plugin like WooCommerce will redirect to their onboarding on the first /wp-admin/ requests after the plugin was installed.
This behavior breaks the test because it needs to be on the Plugins page for the test to work.

Ideally, this should be tested with unit tests by calling is_plugin_active from PHP, but we use e2e tests.

This part of the test checks if the plugin is active by going to the /wp-admin/plugins.php page.

This makes sense to me. We have no way of automatically going through the onboarding pages of different plugins. At the moment, we're focusing on testing the plugin activation, which doesn't necessarily mean that the plugin works, but it is a strong indication, because activation usually executes migrations, populates data, etc. Testing that the plugin successfully activates and doesn't break basic WordPress functionality seems like a reasonable approach to me.

Thanks for your feedback! I will run the tests as-is today, just to get 2025 stats.

After that, I can experiment with adding more tests and manually review the results.
The question about what it means for a plugin and theme to work is on top of my mind now as I'm working on the Playground Tester, and this is a good way of exploring different approaches, as it doesn't require me to rerun all tests for all WP.org plugins and themes.