Skip to content

Commit 4736d6f

Browse files
JameswlepageJameswlepagedkottersethrubenstein
authored
Added > New Experiment: Comment Moderation (#155)
* Add Comment Moderation experiment Adds AI-powered comment moderation with: - Toxicity scoring and sentiment analysis badges in Comments list - Lazy analysis that processes pending comments on page load - Bulk action to queue multiple comments for analysis - AI Reply suggestions modal with tone selection - Comment meta storage for analysis results Includes shared run-ability.ts utility for Abilities API fallback and get_model_preferences() method for model selection. * Set a json schema instead of asking the LLM to return in json. Fix a bug with the column not rendering * Remove the comment reply Ability for now * Move our columns after the comment content * When a new comment is created, send it through our comment analysis ability and if it is above our threshold and is negative sentiment, automatically moderate it * Update text used for the comment analysis action. Add a filter around the response schema so that can be modified if someone wants to add additional analysis. Simplify docs. Bring back enable/disable all tests for the admin group * Add an inline link to trigger analysis. Fix a potential issue with re-renders * Add unit and E2E tests --------- Co-authored-by: Jameswlepage <isotropic@git.wordpress.org> Co-authored-by: dkotter <dkotter@git.wordpress.org> Co-authored-by: sethrubenstein <smrubenstein@git.wordpress.org>
1 parent f48f8c9 commit 4736d6f

17 files changed

Lines changed: 2284 additions & 6 deletions

File tree

‎.wordpress-org/screenshot-#.png‎

472 KB
Loading
Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
# Comment Moderation
2+
3+
## Summary
4+
5+
The Comment Moderation experiment adds sentiment and toxicity analysis to the classic Comments admin screen (`edit-comments.php`). It also performs automatic moderation when new comments are created: comments with **high toxicity** and **negative sentiment** are moved to moderation. The experiment exposes one WordPress Ability (`ai/comment-analysis`) that can be used from the UI or via REST API.
6+
7+
## Overview
8+
9+
When enabled, the Comments list table gets two additional columns: **Sentiment** and **Toxicity**. Each comment can show a badge with the current analysis state or result.
10+
11+
**Key Features:**
12+
13+
- Adds sentiment and toxicity badges to the Comments admin list table
14+
- Adds a bulk action (**Analyze Sentiment and Toxicity**) to queue selected comments for analysis
15+
- Processes queued comments in the browser, sequentially, to avoid server overload
16+
- Automatically moderates newly created comments when analysis indicates high risk
17+
- Uses one shared ability (`ai/comment-analysis`) for both automated and manual workflows
18+
19+
**Automatic moderation rule (default):**
20+
21+
- If `toxicity_score >= 0.7` **and** `sentiment === 'negative'`, the comment is set to pending moderation (`comment_approved = '0'`).
22+
23+
### Input Schema
24+
25+
The `ai/comment-analysis` ability accepts:
26+
27+
```php
28+
array(
29+
'type' => 'object',
30+
'properties' => array(
31+
'comment_id' => array(
32+
'type' => 'integer',
33+
'description' => 'The ID of the comment to analyze.',
34+
'required' => true,
35+
),
36+
),
37+
'required' => array( 'comment_id' ),
38+
)
39+
```
40+
41+
### Output Schema
42+
43+
The ability returns:
44+
45+
```php
46+
array(
47+
'type' => 'object',
48+
'properties' => array(
49+
'comment_id' => array( 'type' => 'integer' ),
50+
'toxicity_score' => array(
51+
'type' => 'number',
52+
'minimum' => 0,
53+
'maximum' => 1,
54+
),
55+
'sentiment' => array(
56+
'type' => 'string',
57+
'enum' => array( 'positive', 'negative', 'neutral' ),
58+
),
59+
),
60+
)
61+
```
62+
63+
### Permissions
64+
65+
- `ai/comment-analysis` requires `current_user_can( 'moderate_comments' )`
66+
- List table UI and bulk action are intended for comment moderators
67+
68+
## Using the Ability via REST API
69+
70+
### Endpoint
71+
72+
```text
73+
POST /wp-json/wp-abilities/v1/abilities/ai/comment-analysis/run
74+
```
75+
76+
### Authentication
77+
78+
You can authenticate using either:
79+
80+
1. **Application Password** (Recommended)
81+
2. **Cookie Authentication with Nonce**
82+
83+
See [TESTING_REST_API.md](../TESTING_REST_API.md) for detailed authentication instructions.
84+
85+
### Request Example
86+
87+
```bash
88+
curl -X POST "https://yoursite.com/wp-json/wp-abilities/v1/abilities/ai/comment-analysis/run" \
89+
-u "username:application-password" \
90+
-H "Content-Type: application/json" \
91+
-d '{
92+
"input": {
93+
"comment_id": 123
94+
}
95+
}'
96+
```
97+
98+
**Response:**
99+
100+
```json
101+
{
102+
"comment_id": 123,
103+
"toxicity_score": 0.81,
104+
"sentiment": "negative"
105+
}
106+
```
107+
108+
### Error Responses
109+
110+
The ability may return:
111+
112+
- `missing_comment_id`: `comment_id` was not provided
113+
- `comment_not_found`: no comment exists for the given ID
114+
- `already_processing`: comment is currently being analyzed
115+
- `parse_error`: AI output could not be parsed as expected JSON
116+
- `insufficient_capabilities`: current user lacks moderation permissions
117+
118+
## Extending the Experiment
119+
120+
### Customize moderation behavior
121+
122+
Use `wpai_comment_moderation_should_moderate` to override the default threshold-based moderation decision:
123+
124+
```php
125+
add_filter( 'wpai_comment_moderation_should_moderate', function( $should_moderate, $analysis, $comment_id ) {
126+
if ( $analysis['toxicity_score'] >= 0.6 ) {
127+
return true;
128+
}
129+
return $should_moderate;
130+
}, 10, 3 );
131+
```
132+
133+
## Testing
134+
135+
### Manual Testing
136+
137+
1. **Enable the experiment:**
138+
- Go to `Settings -> AI`
139+
- Enable global AI features and toggle **Comment Moderation**
140+
- Ensure valid AI connector credentials are configured
141+
142+
2. **List table badges and bulk analysis:**
143+
- Go to `Comments -> All Comments`
144+
- Select multiple comments and run **Analyze Sentiment and Toxicity**
145+
- Confirm a success notice reports queued count
146+
- Verify pending badges transition to analyzed badges (or failed state) as processing runs
147+
- Confirm `wpai_analysis_queued` is removed from the URL after processing, and refresh does not re-show the queue notice
148+
149+
3. **Automatic moderation on insert:**
150+
- Submit new comments with varied tone/content
151+
- Confirm high-toxicity negative comments are moved to moderation (`comment_approved = 0`)
152+
- Confirm lower-risk or non-negative comments are not auto-moderated by default
153+
154+
4. **REST API:**
155+
- Call `POST /wp-json/wp-abilities/v1/abilities/ai/comment-analysis/run` with a valid `comment_id`
156+
- Verify response shape and error handling for invalid IDs or insufficient permissions
157+
158+
## Notes & Considerations
159+
160+
### Requirements
161+
162+
- Requires valid AI credentials and text-generation-capable models
163+
- Requires users with comment moderation capabilities for ability access
164+
165+
### Performance & Behavior
166+
167+
- Frontend processing is sequential by design to avoid sending many concurrent requests
168+
- Ability-level status locking prevents duplicate simultaneous analysis on the same comment
169+
- Failed comments are marked `failed` and are retried on subsequent scans because failed badges remain queryable
170+
171+
### Limitations
172+
173+
- Works only on the classic comments list table (no block-based comments UI integration here)
174+
- Analysis is not batched server-side; one ability invocation per comment
175+
- Current retry behavior for failed badges is best-effort and may repeat on reload until resolved

0 commit comments

Comments
 (0)