Page MenuHomePhabricator

V1 release: Do not automatically enroll experienced editors into Mentorship when they visit the Homepage
Closed, ResolvedPublic

Description

User story & summary:

As an experienced editor who visits the Newcomer Homepage, I do not want to be assigned a Mentor, because I no longer need newcomer support.

As a Mentor, I do not want experienced editors assigned to me as Mentees, because it creates confusion and reduces my ability to focus on newcomers who need help.

Background & research:

This task is important because:

  1. The current logic is confusing both for Mentors and Experienced Editors who are auto-assigned a Mentor
  2. Mentors are busy, we should work towards reducing the number of mentees they support.
  3. Mentors are Asking for this:

The homepage tab could be tweaked to say after the six months something like "Congratulations, you are now experienced enough not to need a mentor! Use the Help Desk if you need advice from now on." instead of giving a name. I'd also suggest that anyone who activates the newcomer homepage when they already have over six month's experience should get that message. (I was amused that when I activated the tab to see what it did, I was auto-assigned someone less experienced than I am.)

Acceptance Criteria:

Given I'm an editor that meets the Mentorship criteria defined in Community Configuration.
When I visit the Homepage,
Then the Mentorship module displays the current "opted out from mentorship" state
(In other words, they can still "get a mentor" if they are interested in testing out the system, but we no longer automatically assign a mentor to every experienced editor who visits the Homepage).

image.png (1×750 px, 243 KB)

Configuration Option
A checkbox is added to Community Configuration that enables communities to define thresholds for automatically opting experienced editors out of mentorship.

Threshold Application
If the checkbox is selected, all users who meet both thresholds are automatically set to the “opted out from mentorship” state.
Users in this state must opt in if they want to receive a Mentor.

Retrospective Application
The thresholds apply retrospectively, meaning existing users who meet the criteria will also be opted out.
If a user who was opted out later opts back in, they are reassigned to the same mentor they had previously.

Consistency of User Experience
The mentorship module looks the same for users who are opted out automatically (via thresholds) as it does for those who opt out manually.

Release Approach
The feature is implemented as an opt-in configuration for communities. In other words the release will not change any behavior until an admin makes a change to the related Community Configuration.
Making this behavior the default may be considered later but will require further community discussion and consensus.

Related Objects

Event Timeline

Change #1184598 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/core@master] ConditionalDefaultsLookup: Add support for CUDCOND_MIN_EDITCOUNT

https://gerrit.wikimedia.org/r/1184598

Change #1184599 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/core@master] ConditionalDefaultsLookup: Allow extensions to modify the conditions dynamically

https://gerrit.wikimedia.org/r/1184599

Change #1184600 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] WIP: feat: Allow communities to decide experienced users start as opted out from mentorship

https://gerrit.wikimedia.org/r/1184600

Change #1184799 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/core@master] ConditionalDefaultsLookup: Add support for CUDCOND_REGISTERED_BEFORE

https://gerrit.wikimedia.org/r/1184799

This is what I am thinking about adding to Community Configuration as part of this task:

image.png (722×1 px, 116 KB)

If the checkbox is selected, the thresholds defined below are applied. All users who meet both thresholds will start in "opted out from mentorship" state, requiring an opt-in to get a mentor. This applies retrospectively (as that is the easiest to implement it seems), the same mentor should be assigned if you opt back again.

There is no difference between how the mentorship module looks like when you opted out manually and when you just meet the thresholds.

after an internal discussion we landed on this copy

[ ] Opt out experienced users from mentorship
    When enabled, mentees are automatically removed from mentorship after reaching both thresholds. They can still get a mentor from their Homepage.

Number of edits
[  500 ]

Account age (in days)
[ 30 ]

image.png (576×1 px, 54 KB)

Change #1186510 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] WIP: feat: Allow communities to opt out experienced users from mentorship

https://gerrit.wikimedia.org/r/1186510

Change #1184600 abandoned by Urbanecm:

[mediawiki/extensions/GrowthExperiments@master] feat: Allow communities to decide experienced users start as opted out from mentorship

Reason:

we used another another method

https://gerrit.wikimedia.org/r/1184600

Change #1184599 abandoned by Urbanecm:

[mediawiki/core@master] ConditionalDefaultsLookup: Allow extensions to modify the conditions dynamically

https://gerrit.wikimedia.org/r/1184599

Change #1184799 abandoned by Urbanecm:

[mediawiki/core@master] ConditionalDefaultsLookup: Add support for CUDCOND_REGISTERED_BEFORE

Reason:

no longer needed

https://gerrit.wikimedia.org/r/1184799

Change #1184598 abandoned by Urbanecm:

[mediawiki/core@master] ConditionalDefaultsLookup: Add support for CUDCOND_MIN_EDITCOUNT

Reason:

no longer needed

https://gerrit.wikimedia.org/r/1184598

Change #1187515 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] refactor: Add CreateMenteeHelpers

https://gerrit.wikimedia.org/r/1187515

Change #1187520 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] tests: Improve test coverage for MenteeGraduation

https://gerrit.wikimedia.org/r/1187520

Change #1187538 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] refactor: Use CreateMenteeHelpers in MentorManagerTest

https://gerrit.wikimedia.org/r/1187538

Change #1187515 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] refactor: Add CreateMenteeHelpers

https://gerrit.wikimedia.org/r/1187515

Change #1187538 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] refactor: Use CreateMenteeHelpers in MentorManagerTest

https://gerrit.wikimedia.org/r/1187538

Change #1187793 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] refactor(MentorManager): Introduce MENTORSHIP_ENABLED_EXPLICITLY

https://gerrit.wikimedia.org/r/1187793

Change #1187837 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/CommunityConfiguration@master] Add CommunityConfigurationEditCompleted hook

https://gerrit.wikimedia.org/r/1187837

Change #1187838 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] feat(Mentorship): Graduate all eligible mentees upon CC changes

https://gerrit.wikimedia.org/r/1187838

Change #1187793 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] refactor(MentorManager): Introduce MENTORSHIP_ENABLED_EXPLICITLY

https://gerrit.wikimedia.org/r/1187793

Change #1186510 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] feat: Allow communities to opt out experienced users from mentorship

https://gerrit.wikimedia.org/r/1186510

Change #1188767 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@wmf/1.45.0-wmf.18] feat: Allow communities to opt out experienced users from mentorship

https://gerrit.wikimedia.org/r/1188767

Change #1188767 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.45.0-wmf.18] feat: Allow communities to opt out experienced users from mentorship

https://gerrit.wikimedia.org/r/1188767

Mentioned in SAL (#wikimedia-operations) [2025-09-16T15:26:29Z] <urbanecm@deploy1003> Started scap sync-world: Backport for [[gerrit:1188767|feat: Allow communities to opt out experienced users from mentorship (T403563)]]

Mentioned in SAL (#wikimedia-operations) [2025-09-16T16:15:39Z] <urbanecm@deploy1003> Started scap sync-world: Backport for [[gerrit:1188767|feat: Allow communities to opt out experienced users from mentorship (T403563)]]

Mentioned in SAL (#wikimedia-operations) [2025-09-16T16:58:00Z] <urbanecm@deploy1003> urbanecm: Backport for [[gerrit:1188767|feat: Allow communities to opt out experienced users from mentorship (T403563)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-09-16T18:23:23Z] <urbanecm@deploy1003> Started scap sync-world: Backport for [[gerrit:1188767|feat: Allow communities to opt out experienced users from mentorship (T403563)]]

Mentioned in SAL (#wikimedia-operations) [2025-09-16T18:44:16Z] <urbanecm@deploy1003> Finished scap sync-world: Backport for [[gerrit:1188767|feat: Allow communities to opt out experienced users from mentorship (T403563)]] (duration: 20m 52s)

I deployed this feature to all wikis (available in Community Configuration). I also enabled the opting out via CC, as discussed with @IZapico-WMF during the Ambassadors meeting.

I tried dry-running the graduation script on eswiki:

[urbanecm@deploy1003 ~]$ mwscript-k8s -f GrowthExperiments:graduateEligibleMentees.php -- --wiki=eswiki --dry-run
⏳ Starting GrowthExperiments:graduateEligibleMentees.php on Kubernetes as job mw-script.eqiad.vlmj41vw ...
🚀 Job is running.
📜 Streaming logs:
Processing mentor Oscar ....    done! Would graduate 25 mentees so far.
Processing mentor Tsaorin...    done! Would graduate 35 mentees so far.
/usr/local/bin/mwscript: line 124:     8 Killed                  ${RUNNER} ${CMD} "${@}"
Processing mentor Hard...[urbanecm@deploy1003 ~]$

Something doesn't like the job. Figuring out what:

[urbanecm@deploy1003 ~]$ kubectl get job mw-script.eqiad.vlmj41vw
NAME                       COMPLETIONS   DURATION   AGE
mw-script.eqiad.vlmj41vw   0/1           18m        18m
[urbanecm@deploy1003 ~]$ kubectl get pods -l job-name=mw-script.eqiad.vlmj41vw
NAME                             READY   STATUS      RESTARTS   AGE
mw-script.eqiad.vlmj41vw-vgcbs   0/3     OOMKilled   0          18m
[urbanecm@deploy1003 ~]$

It seems to run out of memory. For reference, those are our current resource limits:

resources:
  limits:
    cpu: "1"
    memory: 1200Mi
  requests:
    cpu: "1"
    memory: 1000Mi

Not sure what is eating all that memory... Might be related to T395893: growthexperiments-deleteoldsurveys OOMKilled during June 1st run.

I believe this is merged and should be in QA, correct?
I see the feature available via Special:CommunityConfiguration/Mentorship, and I can see it has been enabled on Spanish Wikipedia.

I believe this is merged and should be in QA, correct?
I see the feature available via Special:CommunityConfiguration/Mentorship, and I can see it has been enabled on Spanish Wikipedia.

It depends. There are several issues here yet to solve:

  • config changes are not applied immediately (but only when the experienced user makes an edit),
    • this requires some changes in CC to make this possible (basically, CC needs to tell us "hey, the config has changed!")
    • once the CC updates are done, we can implement this
  • we did not run the maintenance script to opt-out all current experienced users on Spanish Wikipedia, because we run into memory issues (T404804)

So, from eswiki perspective, this is somehow half-way through. That being said, the main feature was indeed implemented and deployed.

I went ahead and created several new tasks tracking the remaining bits:

In addition to those tasks, I also filled T405963: Consider running GrowthExperiments:graduateEligibleMentees maintenance script periodically as an improvement we can consider in the future. I'll update the subtasks accordingly. Let me know if there is anything else unclear, @KStoller-WMF!

For the record, some of those tasks are actually started (within this one), but not yet finished. I'll move them into the appropriate column in the sprint (we can revisit that, of course).

Requirement

When a community enables the “opt out experienced editors” setting in Community Configuration (Mentorship), any user who satisfies the mentorship graduation thresholds (edit count + tenure) — including existing users — must be placed in an “opted out from mentorship” state rather than automatically assigned a mentor when visiting the Homepage. Those users should still see the mentorship module UI and be allowed to manually opt in (“Get a mentor”). Also, if an admin changes the graduation thresholds so a user now qualifies, that user should be silently moved to the opted-out state without requiring a new edit.

BDD

Feature: Mentorship opt-out for experienced editors  

Scenario: Legacy behavior when opt-out config is disabled  
  Given the “opt out experienced editors” checkbox is **disabled** in CommunityConfiguration  
  And I am a user who meets the mentorship graduation thresholds  
  When I visit the Homepage  
  Then I should be automatically assigned a mentor  

Scenario: Manual opt-in from opted out  
  Given I am in the opted out state  
  When I click “Get a mentor” in the mentorship UI  
  Then I should be assigned a mentor  
  And the UI should update to reflect that assignment  

Scenario: Threshold update triggers silent opt-out  
  Given I am a user who meets new thresholds due to a config change, but I have not made an edit since the change  
  When an admin updates the graduation thresholds so I now satisfy them  
  Then on next visit / refresh, I should already be in the opted out state without needing to perform a new edit  

Scenario: Mentees over the threshold are opted out on edit
  Given I am a mentee who meets the thresholds
  And I have never before manually opted in to mentorship (= `mw.user.options.get('growthexperiments-homepage-mentorship-enabled');` is not 50. It should be 1 or unset.)
  When I make an edit
  Then on next visit to the Homepage, I should be opted out of Mentorship (=`mw.user.options.get('growthexperiments-homepage-mentorship-enabled');` is 2)

Test Result

Status: ✅ PASS
Environment: beta
OS: macOS Sequoia 15.5
Browser: Chrome Canary / Safari / Firefox (latest as of test date)
Device: MS
Emulated Device: NA

Test Case 1: Legacy behavior when opt-out config is disabled

  1. As admin, open *Special:CommunityConfiguration → Mentorship*, verify the “opt out experienced editors” checkbox is unchecked
  2. Log in with a user account that meets mentorship graduation thresholds
  3. Visit the Homepage
  4. Verify a mentor is automatically assigned
  5. AC1: Mentor assignment occurs when opt-out is disabled

screenshot 246.png (2×1 px, 660 KB)

Test Case 2: Opt-out behavior when config is enabled
Not valid so omiitted

Test Case 3: Manual opt-in from opted-out

  1. While logged in as the opted-out user, click “Get a mentor”
  2. Wait/allow assignment process
  3. Confirm a mentor is assigned
  4. Confirm the UI now reflects the mentor relationship
  5. AC3: Manual opt-in functions correctly

screenshot 112.mov.gif (1×1 px, 2 MB)

Test Case 4: Threshold update triggers silent opt-out

  1. Use a user account that qualifies only after new thresholds change, and is not yet opted out
  2. As admin, change the graduation thresholds so this user now satisfies them
  3. Refresh the Homepage (or log out & log back in)
  4. Verify user is already in the opted out state without making a new edit
  5. AC4: Config change triggers silent opt-out

Not implemented yet

Test Case 5: Mentees over threshold opt out on edit

  1. Log in as a user who meets the mentorship thresholds, and ensure they have not manually opted in (i.e. mw.user.options.get('growthexperiments-homepage-mentorship-enabled') is 1 or unset)
  2. As that user, make any content edit
  3. After saving, navigate to or refresh the Homepage
  4. Use DevTools / console to inspect mw.user.options.get('growthexperiments-homepage-mentorship-enabled')
  5. Verify it equals 2 (indicating the user is opted out)
  6. AC5: After an edit, a mentee over the threshold is opted out on Homepage visit
Before EditAfter Edit
screenshot 248.png (1×1 px, 295 KB)
screenshot 249.png (1×1 px, 273 KB)
screenshot 250.png (1×1 px, 308 KB)

Thanks @Michael, I was able to add the 5th test scenario from your comment on slack. I understand AC4 isn't implemented, but what about AC2?

@KStoller-WMF, I removed the second AC since it was not a valid one. The only one remaining is AC4 (see Test Case 4) and that isn't implemented or working yet. But I believe you mentioned there are subtasks for that. If that is the case let me know and will Resolve this since I don't have the permissions to test this in prod.

KStoller-WMF renamed this task from Do not automatically enroll experienced editors into Mentorship when they visit the Homepage to V1 release: Do not automatically enroll experienced editors into Mentorship when they visit the Homepage.Wed, Oct 15, 9:01 PM