Objective

TestFlow handles the hour a session runs very well. This proposal is about the parts either side of that hour: choosing the tickets, matching them to the people who turn up, sending the messages, and remembering what happened last time.

I moderated a patch testing scrub using it and kept a note of every point where I had to leave the plugin and do something by hand. This is that list, turned into seven proposals, with screenshots of a working prototype for each one.

They are independent. Any one of them stands on its own and I would rather the team picked from them than took all of them.

The Problem

The plugin assumes a moderator who has already done the preparation. In practice that preparation is most of the work, and it is the part a first-time moderator has no help with at all.

• Every ticket is typed in by hand. The ticket pool is a plain textarea. A scrub needs fifteen to twenty tickets, each re-checked on Trac that morning. Nothing carries over from the shortlist you already built, and nothing about the ticket comes with it: no summary, no milestone, no component, no keywords.
• Nothing knows how hard a ticket is. The handbook asks moderators to match ticket complexity to a participant's experience. The plugin holds neither value, so the entire match lives in the moderator's head at the exact moment they are also running the room.
• Nothing is remembered between sessions. There is no record of who tested what. When somebody who came last month asks for a ticket, there is no way to know whether the one you are about to hand them is the one they already reported on.
• The Test Team is not core-only. The 4th Thursday session is a Gutenberg scrub. The pool treats every URL identically, so a Gutenberg issue and a core Trac ticket read the same and get the same columns, several of which only exist on Trac.
• Every message is copy, alt-tab, paste. Twenty or thirty times in an hour, while watching a thread and thinking about who to assign next.
• Participants are typed in by hand too, from a Slack thread that is open in another window.
• A first-time moderator starts from nothing. Before my first scrub I was asked for a written plan: the tickets by tier, what I dropped and why, how I would run the room. I wrote it by hand. Most of it was information the plugin already held.

Where all of this sits

Before the detail, this is the whole Patch Testing Scrub screen with everything in place, so the individual pieces below have somewhere to belong. The left column is unchanged from the plugin today. Everything proposed is in the right column, and all of it is collapsed by default.

1. Import tickets instead of typing them

The problem in detail

Building a pool of eighteen tickets means running a Trac query, opening each ticket, reading the summary, deciding whether it is workable, and typing or pasting the URL into a textarea. Then, because the plugin holds only the URL, all of that context is gone. The dropdown shows #65403 and nothing else. Half an hour of judgement is thrown away at the moment it is recorded.

Why the plugin cannot simply fetch them

Trac returns 403 to anything that is not a person in a browser. I checked this directly against ticket HTML, ?format=tab, CSV and RSS: all refused server-side. So a fetch-the-query feature is not available, and any proposal that assumes one is not buildable.

What is available is handing the moderator the query and accepting what comes back.

What the prototype does

Three ready-made queries, each with the columns already chosen. Click the name to see the tickets on Trac, or click download CSV to get the file, then drop the file in. Summary, status, milestone, priority, component and keywords all arrive with it.

The three queries are the three real cases: the usual needs-testing-with-a-patch pool, good-first-bugs for safe handovers, and tickets that already carry written test steps so the moderator writes nothing.

There is also a filter for the thing that wastes the most preparation time: "Only keep tickets a volunteer can actually work". It drops closed tickets, anything waiting on a design or maintainer decision, anything already heading for commit, and anything with neither a patch nor written test steps. Everything dropped is listed with its reason, so the decision stays visible rather than silent. That is the difference between a pool of eighteen and a pool of eighteen that are all actually workable.

Gutenberg needs no export step

GitHub, unlike Trac, answers a request from the plugin. So for the 4th Thursday session there is no CSV at all: pick a label, pick issues or pull requests, press the button.

The labels offered are the ones the Gutenberg repo actually uses:

And tickets can still be typed

Nothing is taken away. Numbers, #numbers and full URLs all work, mixed together, separated by commas, spaces or new lines.

The panel states its own limit plainly: tickets added this way start with no difficulty, because difficulty is read from keywords and keywords only arrive with a CSV. You can add one yourself after a pipe, or import the CSV later and the details fill in.

2. A difficulty level on tickets and on people

The problem in detail

The plugin's own guidance says "Match ticket complexity to the participant's experience level where possible." Nothing in the plugin holds either value. So the moderator holds eighteen ticket difficulties and every participant's experience in their head, and matches them live, while also watching a thread, sending messages and answering questions.

The failure mode is not dramatic. It is a first-time contributor being handed a REST API ticket, going quiet, and not coming back next month.

What the prototype does

Both sides carry a level. Tickets get one in the pool, after a pipe:

People get one in the tracker, editable at any time because the moderator learns things about someone mid-session:

Levels are read from keywords where keywords exist, so for a CSV import this is automatic:

• Trac good-first-bug gives Beginner
• Trac has-test-info gives Moderate
• GitHub "Good First Issue" gives Beginner
• A GitHub [Type] Bug that also asks for testing gives Moderate
• Anything unreadable gives Advanced, deliberately, because handing a hard ticket to a beginner costs more than holding an easy one back

3. Remember who tested what, across sessions

The problem in detail

Somebody who came last month says they are ready for a ticket. Did they already do #46017? The plugin cannot say. Neither can the moderator, reliably, three weeks later. So either a ticket gets tested twice while others get no coverage, or the moderator stops to scroll back through an old Slack thread mid-session.

What the prototype does

It records every assignment, and puts the answer in the dropdown, on the option, at the moment of the decision. No separate screen to check.

Reading that dropdown for @testahar, who is Beginner:

• Beginner tickets are sorted to the top, in black. Everything else is greyed but still selectable, because the moderator may know better than the keyword.
• #48954 (Beginner) - 1 had this before means somebody else in the history has had this ticket.
• #46017 (Beginner) - they had this before in italics means this person has had it. That is the one that saves a repeat.
• Nothing is blocked. Every option remains clickable.

The full record sits below, and it is honest about its limits: it knows what this plugin assigned, in this browser, and nothing else. It does not read Trac, so it cannot know about testing done independently.

Two things in that screenshot beyond the history itself:

• Export as CSV, so a moderator can hand a record to a team lead.
• Props built from people who were actually assigned or completed a ticket, not everyone who said hello. The props message at the end of a session is the one piece of credit a contributor takes away, and building it from the tracker rather than from memory means nobody who did work gets left out of it.

4. Treat the ticket source as a first-class thing

The problem in detail

Core Trac, Gutenberg, meta and themes are different workflows with different fields. A Gutenberg issue has no milestone, no component and no Trac keywords. A core ticket has all three. Presented in one undifferentiated list, either three columns are permanently blank or three columns are missing.

What the prototype does

Every ticket is classified by where it came from, with a count shown under the pool, and non-core tickets carry their source in the dropdown label so #81497 Gutenberg never gets mistaken for a Trac ticket.

A separate screen groups the pool by source and gives each group the columns that source actually has:

The Core table carries milestone, priority and component. A Gutenberg table drops all three and shows the GitHub state instead. Both carry the level, who has been handed the ticket across every session, and what state it is in now.

Note on this screenshot: this pool was typed in by hand rather than CSV-imported, which is why the detail columns are empty and everything is Core. After a CSV import and a Gutenberg sync the same screen fills in and splits into separate tables per source.

5. Optional Slack sending

The problem in detail

Every message in the plugin is a Copy button. The moderator copies, switches window, pastes, sends, switches back. Twenty to thirty times an hour. It works, and it is the single most repetitive thing in the session.

What the prototype does

Every message keeps its Copy button and gains a Send beside it. With nothing configured, Send is disabled and the plugin behaves exactly as it does today.

That screenshot also shows the announcements feed, covered in section 7.

The setup, and its two levels

This is deliberately staged, because the two halves are very different asks of a workspace admin.

Sending, the smaller ask. An incoming webhook is the least a workspace admin can grant: post only, one channel, no reading of anything. The URL is stored on the site and never sent back to the browser.

Reading, the bigger ask. A bot token can read the channel, so an admin has to weigh that. With one saved, the plugin can pull everyone who replied to the call for volunteers instead of the moderator typing their names, and it reads their wp.org handle from their Slack profile where they have set one. The pull table in that screenshot shows the names, their reply count, the wp.org handle and a level, with a checkbox per person, because the moderator decides who actually joins the session.

Note the panel says which window it reads: with a thread link it reads that thread, which is exactly the people who answered the call for volunteers. With only a channel URL it reads the last 12 hours, which works but picks up anyone who happened to be chatting.

Where each message goes

Session messages like Opening and Closing always go to the channel. Messages aimed at a person can go four ways, set per person, because people arrive differently:

Someone who replied in the scrub thread should be answered in that thread. Someone who spoke in the channel should be answered in the channel. The default follows where they came from, and every row can be changed on its own.

What this actually needs from a workspace admin

This is the part of the proposal with a real dependency on somebody else saying yes, so it is worth being precise rather than optimistic. There are two levels, and they are very different asks.

Level one: sending only, an incoming webhook

The smallest thing that can be granted. A webhook is a single URL that can post to one channel and can do nothing else. It cannot read a message, list a channel, see who is in it, or touch anything outside the channel it was made for.

This alone covers every Send button in the proposal. It does not cover pulling participants.

Level two: reading the channel, a bot token

Genuinely a bigger ask, because a token that can read a channel can read that channel. An admin should weigh that, and might reasonably decline.

The scopes needed, and what each is for:

The app also has to be invited to the channel with /invite, separately from being installed. Adding an app through Integrations adds the webhook, not the bot. That catches everyone once, so it is worth stating in whatever documentation this ends up with.

How I would suggest asking

The documented route is on make.wordpress.org/chat: "If you want to suggest an integration or tool for a channel, or report a bug or issue, join #slackhelp."

So the sequence I would propose, smallest ask first:

1. Agree here first, in this issue. There is no point asking for a token for a feature the team has not agreed it wants.
2. Ask for the webhook only, scoped to #core-test. Post-only, one channel, no read. That is a small enough ask to be a reasonable first request, and it delivers every Send button.
3. Only then discuss the bot token, with the participant pull as the specific justification, and with the team's answer to the "posts as the app" question already settled.
4. Whoever asks should probably be a team rep, not me. A request from the team for a team tool is a different conversation from a request from an individual contributor.

I am not asking anyone to go and do this now. I am flagging what the dependency is so it can be weighed with everything else.

Open questions I would want opinions on rather than decide myself

1. It posts as the app, not as the moderator, unless installed with a user token. Whether a bot posting into #core-test is acceptable is a judgement for the team.
2. Whether a shared workspace token is appropriate at all, versus each moderator using their own in their own test site.
3. Whether the participant pull is worth the ask. Typing three or four names is not that slow. The pull matters more in a busy session than a quiet one, and reasonable people could say the read scope is not worth it.

If the answer is no

Worth saying plainly, because it changes how risky this proposal is: six of the seven proposals need no Slack at all.

Ticket import, difficulty levels, cross-session history, ticket sources, the scrub plan and everything on the Test Chat screen except sending all work with nothing configured and no admin involved. They are pure client-side and server-side work inside the plugin.

If Slack integration is declined entirely, this is what remains for the messaging side:

• The Copy buttons stay exactly as they are today. Nothing is taken away. The proposal is additive throughout, and with no credentials saved the Send buttons are simply disabled.
• Open in Slack. A deep link that opens the right channel, or the right thread, in the Slack app or browser. It does not remove the paste, but it removes hunting for the correct tab, which is its own small tax when three windows are open.
• A webhook without a token. Sending works, reading does not. Participants get typed in as they are today.
• A moderator's own workspace. Anyone can create a free Slack workspace and grant themselves both in about a minute, which is how I tested all of this. That is enough for anyone who wants to try the feature without any decision being made centrally.

So the worst case is that section 5 does not happen and the other six are unaffected. That seems like an acceptable risk to me, but the team should know it is the shape of the risk.

6. A scrub plan the plugin can draft

The problem in detail

Before my first scrub, a team lead asked me to write a plan and send it over: what I intended to run, which tickets I picked, what I dropped and why, and how I would handle the room. That is the right thing to ask for. It is also several hours of assembling information that the plugin was already holding: the tickets, their levels, their summaries, and the exact messages it was going to send.

For anyone considering moderating for the first time, that document is a real barrier, and it is one nobody talks about.

What the prototype does

The left column asks only for what the plugin cannot know: the date and time, the channel, who is moderating, who to send it to, the release status line, how you chose the tickets, and what you dropped with reasons. The house rules come prefilled with the conventions the Test Team already runs on, fully editable, so a first-timer starts from something rather than a blank page.

The right column builds the document live. It quotes the actual messages the session will use, pulled from the same templates the plugin sends, so what a lead reviews is exactly what the channel will see. The ticket pool comes through as tables grouped by tier with summaries and milestones.

Four exports, because the destination varies:

• Copy for Google Docs puts real HTML on the clipboard, so headings and tables survive the paste
• Copy as Markdown for GitHub or a handbook page
• Download as a file
• New Google Doc opens an empty document to paste into. A plugin cannot create a document in your Drive, and the screen says so rather than implying it can

Underneath, it drafts the message that goes with the link when you send it to the lead.

7. The same treatment for Test Chat

The Test Chat screen has the same shape of problem, so it gets the same shape of answer.

Session Variables sit at the top, because the facilitator and note-taker are what everything else fills into.

Send on every message, exactly as on the scrub screen:

The Slack setup is shared with the scrub screen, so setting it in either place sets it in both, and the panel says so:

Attendance. Section 2 of the agenda is literally called Attendance Check and has no tooling. This reads who has posted since you pressed Start, so the roll call matches the meeting rather than the last twelve hours, excludes bots, and gives a tickable list you can copy or send:

Announcements. Section 7 is called Announcements and has no feed. This carries the latest posts from the Make WordPress blogs, so the section does not need a separate tab open:

Meeting notes. Builds the skeleton of the notes post from what the screen already knows, the agenda link, who ran it, who was there, plus decisions and action items typed as the meeting runs. Action items take an owner before a pipe:

The Overview screen, tying it together

One more piece, which is less a feature than the thing that makes the others findable. The landing screen currently has two cards. In the prototype it reads live state, carries a pre-flight checklist that ticks itself from what is actually configured, and includes a step-by-step walkthrough for somebody who has never moderated.

Worth calling out in that screenshot:

• The checklist marks the channel-invite step with a question mark, not a tick or a cross, because Slack only reports that when a call fails. Claiming either way would be a guess, and the screen says so.
• There is a copy-paste box for /invite @testflow_console, because adding the app through Integrations adds the webhook, not the bot, and that catches everyone once.
• "What this console cannot do" is a section on purpose. It cannot read Trac. It cannot know about testing done outside the plugin. It posts as the app unless installed with a user token. Session state lives in one browser. A tool that overstates itself in a testing workflow is worse than one that does less.

What this actually saves

I have not timed a controlled comparison, so I am not going to quote a number I cannot stand behind. What I can do is say precisely which work disappears, and let people judge it against their own sessions.

Before a session

During a session

The honest summary is that none of this makes a session shorter. A scrub is an hour because it is an hour. What changes is how much of the moderator's attention goes on clerical work while the session is running, and how much preparation stands between a willing contributor and their first time moderating.

The second one is the reason I care about this. The plugin already removes the fear of not knowing what to say. The preparation is the part that is still daunting.

Why now

Two things about the Test Team's own direction make tooling worth revisiting.

The scope is widening. Proposal #111 in this repo, to restructure the handbook for multiple testing domains, names Core, Gutenberg, Performance and AI. There is now a Make WordPress AI team shipping a canonical plugin on a weekly cycle, and the Abilities API landed in 6.9. All of that produces things that need testing, by people who need onboarding, in sessions somebody has to moderate.

The tooling has not widened with it. TestFlow today treats every ticket as a core Trac ticket. That was a reasonable assumption when it was written and it is becoming less true each release.

I want to be careful about one thing here, because it would be easy to overclaim: none of what I am proposing is AI, and none of it uses AI. It is CSV parsing, a GitHub API call, some sorting, and localStorage. What the AI team's existence changes is the volume and variety of testing work, which is an argument for better preparation tooling, not an argument for putting a model in a plugin. If anyone wants to discuss AI-assisted testing that is a genuinely interesting conversation and a completely separate proposal.

Benefits

1. Preparation stops being invisible work. The hour before the session is where a new moderator gives up, and it is currently unsupported.
2. The match between ticket and tester stops being guesswork, which is what the handbook already asks moderators to do.
3. Repeat testing stops happening by accident, and props go to the people who actually did the work.
4. The plugin covers the team's real scope, not just core.
5. Nothing above changes current behaviour unless it is switched on. Every panel is collapsed by default, every Send is disabled without a token, and every Copy button still does exactly what it does now.

What I have, and what I am actually asking

I built a working prototype of all of this while moderating, because it was quicker to test the ideas than to describe them. Every screenshot above is that prototype running against a real ticket pool and a real Slack workspace.

I am not proposing that anyone adopt that code. It exists so the ideas can be looked at rather than imagined, and I am happy to walk through it in a test chat or record a short video.

What I am asking is which of these the team thinks are worth having, and in what order. Anything that gets agreement I am glad to build properly as a focused pull request, one change at a time, against the plugin as it stands.

For context: I'm Azhar Ali, @softglaze on wordpress.org. I contribute to the Test team and moderate patch testing scrubs, so I use this plugin for what it was built for. I also maintain four plugins in the directory, so if the release side is ever a bottleneck I am happy to help there too.

Questions you might reasonably ask

I have tried to answer the ones I would ask if somebody brought me this.

Does any of this send anything without me pressing a button?

No. Every send is a button press. There is one optional checkbox that sends the assignment message when you click Assign, and it is off by default, because the alternative is a plugin that talks in a public channel on your behalf without you deciding to.

Where do the Slack credentials live?

In the site's options table, on the server. The webhook URL and the token are never sent back to the browser once saved. The screen shows a masked hint like xoxb-...P0N0 so you can tell which one is stored without it being readable.

What happens if a token leaks or needs revoking?

Revoke it in Slack and it stops working immediately. Nothing in the plugin caches a working copy. The same is true of a webhook.

Does the plugin store data about participants?

Names, and the assignment history, in the moderator's own browser via localStorage. Nothing is sent to any third party and nothing is written to the site's database. The wordpress.org handle, where it is used, comes from a field the person set on their own Slack profile.

That said, it is a fair thing to raise. If the team would rather no participant data persisted between sessions at all, the history feature is the one to challenge, and I would not argue hard for it over a legitimate privacy concern.

Why localStorage and not the database?

Because a scrub is one moderator's working session, not site content, and because it keeps the plugin free of new tables and new migrations. The honest cost is that state does not follow you to another machine or survive clearing your browser, which the Overview screen says out loud rather than hiding.

If the team would prefer this in the database, that is a reasonable position and a small change.

Does this break anything for people who use the plugin as it is now?

It should not. Every panel is collapsed by default, every Send button is disabled with nothing configured, and every Copy button behaves exactly as it does today. The intent throughout is additive.

Is this not a lot of new surface area to maintain?

Yes, and that is the strongest argument against taking all of it. Which is why I have listed them separately rather than as a package. If only ticket import and difficulty levels land, that is a good outcome and the rest can wait or never happen.

What if Trac changes its CSV format?

The import reads columns by name, not by position, and ignores columns it does not recognise. A new column is harmless. A renamed column would mean that field stops populating, not a crash. The pool would still import.

What about GitHub rate limits?

Unauthenticated requests are limited per IP, and a sync is one request. A moderator pressing the button a few times before a session is nowhere near the limit. If it ever became a problem, the endpoint could accept an optional token.

Does it work on multisite, or a shared install?

The credentials are per-site options, so on multisite each site has its own. The session state is per-browser, so two moderators on the same site do not see each other's sessions. I have not tested it on multisite though, and I would want to before claiming it works.

Who maintains this if you disappear?

A fair question to ask anyone proposing a feature. My answer is that everything here is plain PHP and vanilla JavaScript with no build step and no dependencies, which is a deliberate match for how the plugin is already written. Anyone comfortable in the existing codebase can maintain it. I am also not going anywhere, and I have four plugins in the directory that I have kept updated.

Why not just fork it and maintain your own version?

Because a second plugin doing the same job would split a small team's attention, and because the value is in everyone running the same thing. I would rather contribute six small pull requests to this plugin than maintain a competing one.

Can we see it before deciding?

Yes. I am happy to walk through it in a test chat, record a short video, or answer anything here. Whatever is most useful.