Add category filtering to test selection by sadiksha-23 · Pull Request #65 · WordPress/wp-bench

sadiksha-23 · 2026-09-05T03:16:28Z

What

Added optional category filtering through RunConfig.categories.
Added a repeatable or comma-separated --category CLI option.
Applied category filtering before existing seeded, stratified limit selection.
Preserved explicit test_ids precedence over categories.
Added category information to dry-run output and result metadata.
Added focused selection, configuration, CLI, and metadata tests.
Updated the README and example configuration.

This is the category-only split requested in #56. It preserves the existing limit, seed, and stratified-selection behavior.

Why

This allows developers to run specific benchmark categories, such as hooks or rest-api, without manually listing every test ID or running the entire dataset.

Verification

ruff check python
mypy python
Focused category-selection tests: 24 passed
Full pytest python in CI

Local full-suite result: 203 passed. Four environment-dependent tests could not complete on Windows because PHP was unavailable and one test expects a POSIX path.

Score impact

None. This adds a test-selection filter without changing scoring algorithms or benchmark datasets.

github-actions · 2026-09-05T03:16:37Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Unlinked Accounts

The following contributors have not linked their GitHub and WordPress.org accounts: @Isadiksha0@gmail.com.

Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Unlinked contributors: Isadiksha0@gmail.com.

Co-authored-by: lezama <migueluy@git.wordpress.org>
Co-authored-by: sadiksha-23 <lsadiksha0@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

lezama

Works as described. ruff, mypy and pytest (207 passed) are green here. Two things that cross files:

This conflicts with #63, which removes difficulty. I merged the two branches locally: selection.py (module docstring) and test_dataset_filtering.py (the helper passes difficulty="basic") conflict, and after #63 the ExecutionTest constructor no longer accepts difficulty, so that helper would also fail at runtime. #63 lands first, then this branch rebases and drops the two spots flagged inline. The description line about preserving difficulty goes away with it.
A misspelled category is only an error when it is the only one. --category hooks,typo runs the 6 hooks tests and says nothing about typo. Flagged inline with a fix.

On the Windows note in the description: checked the diff, no skip was added, so CI still runs those four tests.

lezama · 2026-09-07T15:41:53Z

        return tests
+
+    if categories:
+        requested_categories = set(categories)


If one requested category is unknown and another is valid, the unknown one is silently dropped (--category hooks,typo runs 6 hooks tests and says nothing). Let's compute unknown = requested_categories - {t.category for t in tests} here and raise ValueError naming them before filtering. The zero-tests message in core.py then stays as the fallback.

lezama · 2026-09-07T15:41:53Z

+    assert selected_test_ids(selected) == [f"e-bbb-{i:03d}" for i in range(10)]
+
+
+def test_unknown_category_raises_value_error_naming_category() -> None:


Let's add the mixed case too: categories=["bbb", "not-a-category"] should raise naming not-a-category. Today it returns the 10 bbb tests.

lezama · 2026-09-07T15:41:53Z

        expected_behavior="expected",
-        category="general",
+        category=category,
        difficulty="basic",


This conflicts with #63, which removes difficulty from ExecutionTest. After rebasing on #63 drop this line, the constructor will reject it.

lezama · 2026-09-07T15:41:53Z

@@ -7,6 +7,9 @@
 limited runs deterministic (same seed = same subset), tunable (different
 seed = different subset), and representative (round-robin across
 category/difficulty groups).


Same conflict with #63, which rewrites this docstring to say categories only. Keep #63's wording when rebasing.

lezama · 2026-09-07T15:41:53Z

                "scoring_version": SCORING_VERSION,
                "seed": self.config.run.seed,
                "limit": self.config.run.limit,
+                "categories": self.config.run.categories,


This records the config value, not what ran. With test_ids set the categories are ignored but still land in metadata. Let's write [] when test_ids wins, so a results file never claims a category filter that had no effect.

sadiksha-23 · 2026-09-08T04:44:37Z

Thanks for the detailed review. I reb based the branch on trunk after #63, resolved the difficulty-related conflicts, added validation for mixed valid/unknown categories, and updated metadata to record an empty category list when explicit test IDs take precedence. All CI checks are now passing.

lezama

thanks for addressing everything! rebased locally on current trunk, merges clean and ruff/mypy/pytest are green. lgtm :)

sadiksha-23 mentioned this pull request Sep 5, 2026

simplify test selection and update dataset schemas #56

Closed

4 tasks

lezama mentioned this pull request Sep 7, 2026

feat: target WordPress 7.1, drop the difficulty field, and refresh the execution-test skill #63

Merged

lezama reviewed Sep 7, 2026

View reviewed changes

lezama mentioned this pull request Sep 7, 2026

Add category filtering to test selection #67

Closed

feat: add category filtering

70d1de4

sadiksha-23 force-pushed the feat/category-filter branch from b3fc516 to 70d1de4 Compare September 8, 2026 04:40

lezama approved these changes Oct 22, 2026

View reviewed changes

lezama merged commit fd40415 into WordPress:trunk Oct 22, 2026
6 checks passed

Navigation Menu

Repository navigation

Add category filtering to test selection - #65

Add category filtering to test selection#65
lezama merged 1 commit into
WordPress:trunkfrom
sadiksha-23:feat/category-filter

sadiksha-23 commented Sep 5, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 5, 2026 •

edited

Loading

Uh oh!

lezama left a comment

Uh oh!

lezama Sep 7, 2026

Uh oh!

lezama Sep 7, 2026

Uh oh!

lezama Sep 7, 2026

Uh oh!

lezama Sep 7, 2026

Uh oh!

lezama Sep 7, 2026

Uh oh!

sadiksha-23 commented Sep 8, 2026

Uh oh!

lezama left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		assert selected_test_ids(selected) == [f"e-bbb-{i:03d}" for i in range(10)]


		def test_unknown_category_raises_value_error_naming_category() -> None:

Conversation

sadiksha-23 commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Verification

Score impact

Uh oh!

github-actions Bot commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unlinked Accounts

Uh oh!

lezama left a comment

Choose a reason for hiding this comment

Uh oh!

lezama Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

lezama Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

lezama Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

lezama Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

lezama Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

sadiksha-23 commented Sep 8, 2026

Uh oh!

lezama left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sadiksha-23 commented Sep 5, 2026 •

edited

Loading

github-actions Bot commented Sep 5, 2026 •

edited

Loading