Page MenuHomePhabricator
Create Task
Maniphest T400696

FY25-26 WE1.4.1 RecentChanges database performance improvements
Open, Needs TriagePublic
Actions

Description

Epic ticket to keep track of work done by many different teams and people. See T307328: Scalability issues of recentchanges table for more information

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT400696 FY25-26 WE1.4.1 RecentChanges database performance improvements
 OpenFeaturetstarlingT399458 Move category changes from RecentChanges/Watchlist to separate Special page
 ResolvedFeatureZabeT399455 Change default recentchanges query time on large wikis
 ResolvedZabeT36320 Remove obsolete recentchanges.rc_new database field
 ResolvedZabeT244080 Add index to recentchanges.rc_source
 ResolvedLadsgroupT400854 Add rc_source_name_timestamp index to recentchanges table in wmf production
 ResolvedLadsgroupT402763 Drop rc_new from recentchanges table in wmf production
 OpenNoneT400698 Reduce noisy wikidata injected rows to recentchanges
 OpenNoneT401284 [Epic] Reduce Change Propagation Noise from Wikidata to Client Wikis
 ResolvedNoneT401286 Turn off Wikidata injection for qualifiers and references
 In Progressseanleong-WMDET401288 Implement a more granular alias usage tracking
 Resolvedseanleong-WMDET403810 Hot fix for missing case for alias entity usage
 ResolvedNoneT401289 Add bot flag to change propagation parameter
 In ProgressNoneT401290 Implement new usage types for qualifiers and references
 OpenBUG REPORTNoneT407684 Lua's ipairs() function can no longer iterate over Wikidata references
 StalledNoneT401291 [Blocked] Bump collapsing usage limits
 OpenNoneT401315 Observe and metrics tracking
 OpenNoneT403038 Avoid row injection if the reparse is a no-op
 OpenZabeT74157 [Story] Use rc_source and drop rc_type
 ResolvedaudeT44289 [Task] Use rc_source for filtering Wikibase recent changes and watchlist entries in the client
 ResolvedZabeT401912 Duplicate rc_name_type_patrolled_timestamp index on rc_source
 ResolvedLadsgroupT402010 Add new rc_name_source_patrolled_timestamp index to recentchanges table in wmf production
 ResolvedtstarlingT403002 Unpurged renameuser rows in recentchanges causing isDenseTagFilter to return true inappropriately
 OpenNoneT403003 Index for RC filterable ORES scores
 ResolvedtstarlingT403142 recentchanges.rc_source registry of known values
 OpentstarlingT404416 ChangesListQuery class and related ChangesListSpecialPage refactoring
 ResolvedtstarlingT403222 Special:Newpages user filter performance
 ResolvedtstarlingT403798 Slow watchlist queries due to large and expensive temporary table construction
 ResolvedLadsgroupT404808 Clean up large bots watchlists in all wikis

Event Timeline

Ladsgroup created this task.Jul 29 2025, 1:12 PM
Restricted Application added a project: Moderator-Tools-Team. · View Herald TranscriptJul 29 2025, 1:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Ladsgroup added a subtask: T399458: Move category changes from RecentChanges/Watchlist to separate Special page.Jul 29 2025, 1:12 PM
Ladsgroup added a subtask: T399455: Change default recentchanges query time on large wikis.
Ladsgroup added a subtask: T36320: Remove obsolete recentchanges.rc_new database field.
Ladsgroup moved this task from Inbox to Epic - Database on the Data-Persistence board.Jul 29 2025, 1:35 PM
A_smart_kitten subscribed.Jul 29 2025, 1:35 PM
ldelench_wmf added a project: OKR-Work.Jul 30 2025, 7:49 PM
ldelench_wmf subscribed.
Jdforrester-WMF subscribed.Jul 30 2025, 8:07 PM
Ladsgroup mentioned this in T74157: [Story] Use rc_source and drop rc_type.Aug 1 2025, 7:24 PM
Ladsgroup added a subtask: T74157: [Story] Use rc_source and drop rc_type.
jsn.sherman moved this task from Inbox to Epics on the Moderator-Tools-Team board.Aug 6 2025, 6:19 PM
Ladsgroup added a comment.Aug 13 2025, 12:00 PM

Baseline. In June 2025 (30 days), we had:

Zabe changed the status of subtask T36320: Remove obsolete recentchanges.rc_new database field from Stalled to Open.Aug 14 2025, 3:51 PM
valerio.bozzolan subscribed.Aug 18 2025, 12:21 PM
Ladsgroup closed subtask T399455: Change default recentchanges query time on large wikis as Resolved.Aug 18 2025, 12:50 PM
Universal_Omega subscribed.Aug 18 2025, 6:36 PM
amastilovic subscribed.Aug 18 2025, 10:27 PM
TehKittyCat subscribed.Aug 19 2025, 2:50 AM
ldelench_wmf mentioned this in T400698: Reduce noisy wikidata injected rows to recentchanges.Aug 19 2025, 4:48 PM
Zabe closed subtask T36320: Remove obsolete recentchanges.rc_new database field as Resolved.Aug 25 2025, 8:13 AM
Stashbot added a comment.Aug 26 2025, 2:17 AM

Mentioned in SAL (#wikimedia-operations) [2025-08-26T02:17:12Z] <TimStarling> on db2202 creating copy of enwiki.recentchanges for performance analysis T400696

Nux subscribed.Aug 27 2025, 1:17 PM

I mentioned this already on the cloud mailing list, but please remeber to update the documentation if you are making database changes.

At the time of writing, there is no mention of either deprecation or removal of the rc_type and rc_new columns (which are now gone on the Toolserver).
https://www.mediawiki.org/wiki/Manual:Recentchanges_table#rc_type

Ladsgroup added a comment.Aug 28 2025, 10:42 AM

Mediawiki.org documentation is not the source of what is deprecated and what is not.

tstarling added a subtask: T403222: Special:Newpages user filter performance.Aug 28 2025, 11:37 PM
Ladsgroup closed subtask T403002: Unpurged renameuser rows in recentchanges causing isDenseTagFilter to return true inappropriately as Resolved.Aug 29 2025, 10:54 AM
MarcoAurelio mentioned this in T403710: Replace rc_type with rc_source.Sep 4 2025, 10:44 AM
tstarling closed subtask T403222: Special:Newpages user filter performance as Resolved.Sep 5 2025, 3:47 AM
• PEPE1234.13 closed subtask T403798: Slow watchlist queries due to large and expensive temporary table construction as Invalid.Sep 6 2025, 1:23 PM
Aklapper reopened subtask T403798: Slow watchlist queries due to large and expensive temporary table construction as Open.Sep 6 2025, 2:00 PM
tstarling subscribed.Sep 12 2025, 6:06 AM

Looking at the metrics, I think T399455 did the job already, at least for RC/RCL. It looks like a ~70% reduction in total slow query time for RC and ~79% for RCL.

CLSP performance 2025-09-12.png (422×843 px, 61 KB)

ldelench_wmf added a subtask: T404808: Clean up large bots watchlists in all wikis.Sep 19 2025, 12:42 PM
Stashbot added a comment.Thu, Oct 9, 9:25 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-09T21:25:18Z] <TimStarling> on db2202 cleaned up the tables I created for T400696

tstarling closed subtask T403142: recentchanges.rc_source registry of known values as Resolved.Thu, Oct 23, 6:33 AM
tstarling closed subtask T403798: Slow watchlist queries due to large and expensive temporary table construction as Resolved.Mon, Nov 3, 5:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits