Change 866502 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Start mobile DiscussionTools A/B test

https://gerrit.wikimedia.org/r/866502

Change 866502 merged by jenkins-bot:

[operations/mediawiki-config@master] Start mobile DiscussionTools A/B test

https://gerrit.wikimedia.org/r/866502

Mentioned in SAL (#wikimedia-operations) [2022-12-08T22:17:48Z] <samtar@deploy1002> Started scap: Backport for [[gerrit:866502|Start mobile DiscussionTools A/B test (T321961)]]

Mentioned in SAL (#wikimedia-operations) [2022-12-08T22:19:33Z] <samtar@deploy1002> samtar and matmarex: Backport for [[gerrit:866502|Start mobile DiscussionTools A/B test (T321961)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-12-08T22:27:46Z] <samtar@deploy1002> Finished scap: Backport for [[gerrit:866502|Start mobile DiscussionTools A/B test (T321961)]] (duration: 09m 57s)

Now that the A/B test is live, next steps are as follows...

Next steps

• 1. Editing QA to verify people are being bucketed in production
• 2. @MNeisler to verify data is being logged server side as expected

To check bucketing, what you need to do varies slightly based on whether you're logged in or out.

In either case you can check your bucket by doing this in the JS console: mw.config.get( 'wgDiscussionToolsABTestBucket' ) which should say either control or test.

If you're logged in this should never change for a given user, as it's assigned based on your user ID.

If you're logged out this should stay the same unless you clear your cookies, at which point it should be randomly reassigned. (So clear your cookies a few times and make sure you see a vaguely-even distribution of control/test.)

Bucketing QA
@DLynch I'm reviewing the AB test data logged in EditAttemptStep and identified a few trends that seemed unexpected to me. See below and let me know if you need any additional details.

• Since the deployment of the test, we have only logged 2 edit attempts in the control group on a mobile talk page (as indicated by platform = phone). In addition, all edit attempts in either test group that occur on mobile are tagged as integration = 'discussiontools'. I know that most of the users in the control group will likely be editing using the talk page overlay, which will be tracked in MobileWebUIActionTracking, but I expected a few more VE or wikitext edit attempts from users that switch to the ReadAsWiki view.

• There is no anonymous_user_token assigned to edit attempts on platform = 'desktop' by anon users in either of the AB test groups. Is the anonymous user token only assigned when a user makes an edit attempt on a mobile talk page but the bucket is assigned to anyone accessing a mobile talk page?

90-day retention review
@ppelberg - I did a review to determine if we would lose any data needed for the analysis in T298062 after 90 days. I identified a few fields (listed below) that are needed and are removed after 90 days. As a result, I recommend we prioritize this analysis within 90 days.

• anonymous_user_token in EditAttemptStep: This is needed to identify distinct anon users in the group and will be needed to calculate the edit completion rate for anon users ( if defined as the percent of contributors that complete at least 1 edit).
• page_token field in EditAttemptStep: This is needed to join data with MobileWebUIActionTracking schema which will be used to track talk page views. This is needed for the following metric "Percentage of unique Junior Contributors who visit a talk page and engage with it in some way."
• token in MobileWebUIActionTracking: This is needed to identify unique sessions in MobileWebUIActions which will be used to calculate edit completion rate and other interactions that occur on the MobileFrontend talk page overlay for the control group.

Since the deployment of the test, we have only logged 2 edit attempts in the control group on a mobile talk page (as indicated by platform = phone).

It looks like this is being stepped on by the mobile preferred editor test code, so I'll get a patch up that makes it work properly. (You're almost certainly not going to get many VE edits on 'phone', though, as talk pages don't allow switching to VE.)

There is no anonymous_user_token assigned to edit attempts on platform = 'desktop' by anon users in either of the AB test groups. Is the anonymous user token only assigned when a user makes an edit attempt on a mobile talk page but the bucket is assigned to anyone accessing a mobile talk page?

Turns out that the desktop editors need to be adjusted slightly -- they're still set up for the new topic version of this, which didn't need to have the anonymous tokens logged.

Patches incoming shortly.

Change 868742 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/WikiEditor@master] Log token for the DiscussionTools mobile a/b test

https://gerrit.wikimedia.org/r/868742

Change 868743 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/MobileFrontend@master] Log bucket/token for the DiscussionTools mobile a/b test

https://gerrit.wikimedia.org/r/868743

Change 868744 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/DiscussionTools@master] a/b test anonymous ID was being reset because of cookie prefixes

https://gerrit.wikimedia.org/r/868744

Change 868745 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@master] Log bucket/token for the DiscussionTools mobile a/b test

https://gerrit.wikimedia.org/r/868745

Change 868544 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[operations/mediawiki-config@master] Revert "Start mobile DiscussionTools A/B test"

https://gerrit.wikimedia.org/r/868544

Change 868544 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "Start mobile DiscussionTools A/B test"

https://gerrit.wikimedia.org/r/868544

Mentioned in SAL (#wikimedia-operations) [2022-12-19T03:08:51Z] <tstarling@deploy1002> Synchronized wmf-config/InitialiseSettings.php: disable wgDiscussionToolsABTest T325477 T321961 (duration: 15m 23s)

Notes from 19 December conversation we had offline:

• The A/B test was paused because of T325477
• We think the issue that caused T325477 will be resolved by way of 869169
• We plan to re-enable the A/B test, along with the following patches in first available backport window when we return in January 2023:
  • 869169
  • 868742
  • 868743
  • 868744
  • 868745

cc @DLynch

Change 868745 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Log bucket/token for the DiscussionTools mobile a/b test

https://gerrit.wikimedia.org/r/868745

Change 868744 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] a/b test anonymous ID was being reset because of cookie prefixes

https://gerrit.wikimedia.org/r/868744

Change 868743 merged by jenkins-bot:

[mediawiki/extensions/MobileFrontend@master] Log bucket/token for the DiscussionTools mobile a/b test

https://gerrit.wikimedia.org/r/868743

Change 868742 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@master] Log token for the DiscussionTools mobile a/b test

https://gerrit.wikimedia.org/r/868742

Change 874868 had a related patch set uploaded (by Bartosz Dziewoński; author: DLynch):

[mediawiki/extensions/DiscussionTools@wmf/1.40.0-wmf.14] a/b test anonymous ID was being reset because of cookie prefixes

https://gerrit.wikimedia.org/r/874868

Change 874869 had a related patch set uploaded (by Bartosz Dziewoński; author: DLynch):

[mediawiki/extensions/VisualEditor@wmf/1.40.0-wmf.14] Log bucket/token for the DiscussionTools mobile a/b test

https://gerrit.wikimedia.org/r/874869

Change 874866 had a related patch set uploaded (by Bartosz Dziewoński; author: DLynch):

[mediawiki/extensions/WikiEditor@wmf/1.40.0-wmf.14] Log token for the DiscussionTools mobile a/b test

https://gerrit.wikimedia.org/r/874866

Change 874867 had a related patch set uploaded (by Bartosz Dziewoński; author: DLynch):

[mediawiki/extensions/MobileFrontend@wmf/1.40.0-wmf.14] Log bucket/token for the DiscussionTools mobile a/b test

https://gerrit.wikimedia.org/r/874867

Change 874870 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Revert "Revert "Start mobile DiscussionTools A/B test""

https://gerrit.wikimedia.org/r/874870

Change 874866 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@wmf/1.40.0-wmf.14] Log token for the DiscussionTools mobile a/b test

https://gerrit.wikimedia.org/r/874866

Change 874867 merged by jenkins-bot:

[mediawiki/extensions/MobileFrontend@wmf/1.40.0-wmf.14] Log bucket/token for the DiscussionTools mobile a/b test

https://gerrit.wikimedia.org/r/874867

Change 874868 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@wmf/1.40.0-wmf.14] a/b test anonymous ID was being reset because of cookie prefixes

https://gerrit.wikimedia.org/r/874868

Change 874869 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@wmf/1.40.0-wmf.14] Log bucket/token for the DiscussionTools mobile a/b test

https://gerrit.wikimedia.org/r/874869

Mentioned in SAL (#wikimedia-operations) [2023-01-03T14:33:02Z] <taavi@deploy1002> Started scap: Backport for [[gerrit:874866|Log token for the DiscussionTools mobile a/b test (T321961)]], [[gerrit:874867|Log bucket/token for the DiscussionTools mobile a/b test (T321961)]], [[gerrit:874868|a/b test anonymous ID was being reset because of cookie prefixes (T321961)]], [[gerrit:874869|Log bucket/token for the DiscussionTools mobile a/b test (T321961)]]

Mentioned in SAL (#wikimedia-operations) [2023-01-03T14:34:48Z] <taavi@deploy1002> taavi and matmarex: Backport for [[gerrit:874866|Log token for the DiscussionTools mobile a/b test (T321961)]], [[gerrit:874867|Log bucket/token for the DiscussionTools mobile a/b test (T321961)]], [[gerrit:874868|a/b test anonymous ID was being reset because of cookie prefixes (T321961)]], [[gerrit:874869|Log bucket/token for the DiscussionTools mobile a/b test (T321961)]] synced to the testservers:

Mentioned in SAL (#wikimedia-operations) [2023-01-03T14:41:33Z] <taavi@deploy1002> Finished scap: Backport for [[gerrit:874866|Log token for the DiscussionTools mobile a/b test (T321961)]], [[gerrit:874867|Log bucket/token for the DiscussionTools mobile a/b test (T321961)]], [[gerrit:874868|a/b test anonymous ID was being reset because of cookie prefixes (T321961)]], [[gerrit:874869|Log bucket/token for the DiscussionTools mobile a/b test (T321961)]] (duration: 08m 31s)

Change 874870 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "Revert "Start mobile DiscussionTools A/B test""

https://gerrit.wikimedia.org/r/874870

Mentioned in SAL (#wikimedia-operations) [2023-01-03T14:43:58Z] <taavi@deploy1002> Started scap: Backport for [[gerrit:874870|Revert "Revert "Start mobile DiscussionTools A/B test"" (T321961)]]

Mentioned in SAL (#wikimedia-operations) [2023-01-03T14:45:46Z] <taavi@deploy1002> taavi and matmarex: Backport for [[gerrit:874870|Revert "Revert "Start mobile DiscussionTools A/B test"" (T321961)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-01-03T14:53:12Z] <taavi@deploy1002> Finished scap: Backport for [[gerrit:874870|Revert "Revert "Start mobile DiscussionTools A/B test"" (T321961)]] (duration: 09m 13s)

I went ahead and rechecked the AB test data logged since the AB test was redeployed on 3 January 2022. The data now appears to be logging as expected and the issues identified in T321961#8474391 have been resolved.

See summary of checks below:

• Number of overall and per wiki edit attempts and users per bucket appears balanced based on a 50/50 split.

Overall Edit Attempts in AB Test Logged to Date

• We are now correctly logged edit attempts on mobile talk pages in both the control and test groups.
• An anonymous_user_token is correctly assigned to all edit attempts in the AB test.
• Edit attempts where event.integration = 'page' or where event.integration = 'discussiontools' are both being logged in the AB test. The control group does not have any mobile talk page edit attempts where event.integration = 'discussiontools' as expected.
• Attempts by editor interface appear as expected.
• Confirmed I can join editing data with MobileWebUIActionsTracking using the page token field to determine mobile page views.

@ppelberg - Following up on my comment in T321961#8474391, recommend we complete this analysis within 90 days or obtain permission from legal to retain data for a longer period of time. The test started on 3 January 2022 so we will start to lose some event data on 3 April 2022.

QA Notebook

@Ryasmeen pointed out an issue with bucket assignment for logged out users that made me dig into it:

• bucket assignment winds up being per-page because of caching for logged out users
• the anonymous_user_token is reliably stored per-user, though

This would mean that our logged out data is sort of a mess for analysis (the bucket will just refer to the initial presentation of the page, and users will be getting inconsistent experiences between pages), but in an amusing twist it turns out that none of the wikis we wound up turning the a/b test on for in the end have the talk page tab exposed to logged out users. So, logged out users on mobile should only be seeing talk pages if they're following links from outside the mobile site to those pages, which I suspect will be happening at very low rates...

Hi, could you help filling up the incident report at https://wikitech.wikimedia.org/wiki/Incidents/2022-12-18_World_Cup ? While I was able to state the operational bits, I am mostly ignorant about the code details/deployments/fixes leading to the outage and its resolution.

In T321961#8527733, @jcrespo wrote:

Hi, could you help filling up the incident report at https://wikitech.wikimedia.org/wiki/Incidents/2022-12-18_World_Cup ? While I was able to state the operational bits, I am mostly ignorant about the code details/deployments/fixes leading to the outage and its resolution.

hi @jcrespo: do you have all that you need from the Editing Team as it relates to the "...code details/deployments/fixes leading to the outage and its resolution." ?

In T321961#8545576, @ppelberg wrote:

In T321961#8527733, @jcrespo wrote:

Hi, could you help filling up the incident report at https://wikitech.wikimedia.org/wiki/Incidents/2022-12-18_World_Cup ? While I was able to state the operational bits, I am mostly ignorant about the code details/deployments/fixes leading to the outage and its resolution.

hi @jcrespo: do you have all that you need from the Editing Team as it relates to the "...code details/deployments/fixes leading to the outage and its resolution." ?

There is still some yellow gaps at https://wikitech.wikimedia.org/wiki/Incidents/2022-12-18_World_Cup I wonder if you could help filling them, specially the section about mitigation timeline (linking to patches deployed to mitigate the issue). Thank you!

Done

So, logged out users on mobile should only be seeing talk pages if they're following links from outside the mobile site to those pages, which I suspect will be happening at very low rates...

@DLynch - When reviewing the AB test data, I've noticed that we have recorded a decent number of edit attempts from logged-out users on mobile talk pages.

Here are the mobile talk page edit attempts recorded to date by logged-in and logged-out users:

The number of logged-out user edit attempts is much lower than we typically see but still seems a little high to me. Does these numbers likely reflect users following links from outside the mobile site to those pages or is something else going on?

I think that following direct links would be the only way to make that happen, yes.