Repository navigation
Performance testing next steps #1093
Description
From the below, top of mind are currently:
- Lack of care around the dashboard (no alerts, people not checking it)
- Outdated test suite (still testing TT3, not testing TT5, baseline is 6.1)
- Stability
- Project ownership
Improve core performance tests
- Core
- Unclear ownership / maintenance
- Measure interactivity metrics in performance tests
- Measure layout stability metrics in performance tests
- Add tests for Twenty Twenty-Five
- Test more combinations (e.g. Multisite, object cache enabled)
- Outdated baseline (6.1)
- Gutenberg
- Tests quality and stability
- Alignment between core and Gutenberg
- Core relies on GitHub Actions, while GB uses a sophisticated node script for running tests
- GB regressions are often only found after merging into trunk (too late)
- Improve demo content
- Consider covering specific realistic use cases (e.g., homepage, page with large LCP image, etc)
- Relevant for testing interactivity and layout stability
- Performance Metrics Stabilization #849
- Use WordPress Playground for more stability
- Just for context, this was something that @dmsnell brought up, most recently in this Slack message
- We could simulate different I/O latencies and intercept every DB and file operation
wp-nowhas some design problems that make it less favorable for testing (needs some refactoring first?)
- Use WordPress Playground for more stability
- Alignment between core and Gutenberg
- Dashboard
- Current custom dashboard is limited
- Maybe build a more powerful dashboard using Grafana or similar
- Lack of alerts for significant changes or broken workflow
Adoption
- Run performance tests at scale using Tide?
- More outreach (1:1, developer blog, etc.)
- Directly reach out to a few bigger plugins to help them set up performance testing, setting a precedent for others
- Consider moving GitHub Action to the WordPress GitHub org to make it more official
Typical reasons why plugin developers do not adopt similar performance tests:
- They use a different testing setup (Codeception, Cypress, Puppeteer, k6) or CI (Bitbucket, GitLab)
- Lack of tests setup in general (e.g. not even unit tests in place)
- Focus is more on backend performance rather than frontend performance
- Lack of resources for implementing additional kinds of testing
- No testing strategy, e.g. even if they had the whole setup, they wouldn't know where to start
- Metrics stability concerns
- Makes it especially difficult to tie metrics change to specific code change
Activity
joemcgill commentedon Mar 29, 2024
Thanks for kicking this discussion off, @swissspidy.
I appreciate the distinction between two main objectives. For now, I'm going to limit my thoughts to the first objective, "Improve Core/GB performance testing".
One of the things that I observed during the 6.5 release is that finding the source of a server timing regression was challenging when that regression was committed to the Core repo as part of a larger Gutenberg sync. I think we can improve this somewhat by updating the performance tests in the Gutenberg repo to include the same server timing metrics that we record for each commit to the WP Core repo (i.e., wpTotal, wpBeforeTemplate, and wpTemplate). While TTFB is a close proximity, the additional noise from network requests and calculating the metric in a headless browser makes pinpointing potential regressions more difficult.
I also strongly agree with your suggestion to improve our demo content, and think this applies as much to the Core tests as the Gutenberg tests. Currently in Core, we are only testing the default homepage for the Twenty Twenty-One and Twenty Twenty-Three themes after importing a Theme Test Data from this commit. While this keeps test content consistent over time, there are a number of limitations to this approach including the fact that some specific use cases that we care about (e.g., measuring the effect of image optimizations on LCP) are not covered by our current test content.
From a visualization point of view, our current dashboards at codevitals.run have become harder to use over time as we've added additional metrics. I'd love to investigate improving or replacing these dashboards with a system where we could more granularly filter results by metric, theme, template, object cache, etc. In doing so, we should also evaluate how we are normalizing and storing the raw data which currently gets normalized before it's saved to the dashboard's database meaning we don't have the ability to build new reports using the original unfiltered data.
Last idea for now, is that when we introduced these tests we used Twenty Twenty-One and Twenty Twenty-Three as representative themes for classic and block themes, but that has proven to be overly simplistic as some performance regressions are only visible based on characteristics of a theme (e.g., how many template part variations they register). At minimum, we should add Twenty Twenty-Four to our test matrix in both repos.
TL;DR:
swissspidy commentedon Apr 3, 2024
That all makes sense to me. I think most of those suggestions are already mentioned in some place or another. I also previously explored a Grafana-based dashboard that would be more user friendly and could be fed raw data.
Some more thoughts on adoption:
Let's start with directly reaching out to some top plugins that we think could benefit from performance testing but don't leverage that yet. Find out why, figure out what's missing, and help them get started.
This way, we can iterate on the tooling before publishing another blog post or improving the GitHub Action.
joemcgill commentedon May 6, 2024
swissspidy commentedon May 6, 2024
joemcgill commentedon May 6, 2024
swissspidy commentedon May 7, 2024
swissspidy commentedon Jul 1, 2024
Improve Core/GB performance testing
Mentioned above:
wp-nowhas some design problems that make it less favorable for testing (needs some refactoring first?)More thoughts:
Adoption
The main points I've heard so far in various discussions:
Mentioned above:
wp-nowappears to be unsuitable for testing like this :( (see above)15 remaining items
adamziel commentedon Sep 16, 2024
I would love that! :) I just replied in WordPress/wordpress-playground#1631
💡 interesting! I wonder what formats like that already exist specifically for E2E tests 🤔
swissspidy commentedon Sep 24, 2024
So far I only found https://www.npmjs.com/package/declarative-e2e-test
brandonpayton commentedon Sep 30, 2024
@swissspidy, is there anything more to do here from the perspective of WordPress Playground? We are tracking this issue on our project board and want to be sure you get what you need.
swissspidy commentedon Sep 30, 2024
@brandonpayton Thanks for staying on top of things, appreciate it!
I tried running multiple blueprints but couldn't get that to work, see WordPress/wordpress-playground#1631 (reply in thread)
I have a workaround, so it's not urgent, but it would be nice if that worked reliably.
Apart from that I don't think there's a concrete urgent missing piece.
[-]Performance testing next steps (adoption)[/-][+]Performance testing next steps[/+]swissspidy commentedon Dec 12, 2024
Split some quick wins into a new issue: #1735
swissspidy commentedon Apr 7, 2025
Actually, closing this in favor of #1735