This needs to be blocked on T419021: LDAP groups for tiered access to GrowthBook

The short term fix is to pause the mobile TOC experiment on zhwiki.

zhwiki was removed from the experiment's target wikis as of an hour ago – the experiment is off zhwiki for the remainder of the experiment

@BTullis shared:

What seems to have been happening in this case (T418412), I believe, is that we were querying against a very large number of partitions in the event_sanitized.visualeditorfeatureuse table.
This caused the presto workers to have to transfer and chew over a very large number of files from HDFS to the presto-workers (an-presto 10[06-20]) - These queries took too long, so they timed out in the UI, but kept running in Presto in the background.
https://grafana.wikimedia.org/goto/H_qJAxODg?orgId=1

From investigation in Slack (thread):
@HMonroy was developing some heavy queries that use MediaWiki History and stopped them in the UI / they timed out, but continued in the background. We've seen the ghost query issue more and more frequently lately.

Or maybe I should have created a new task instead?

I'm still frequently getting

QueuePool limit of size 10 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/14/3o7r)

and intermittent 500 errors, including just now while trying to open https://superset.wikimedia.org/users/userinfo/

From https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/1242300/comments/9335d643_1309cf56:

I'm not opposed to Michael's proposal but want to give it a thought. In the past GrowthExperiments had added hidden fields to the account creation form including information such as a variant assigned. I think the TestKitchen JS SDK should be able to add the subjectId information along with the variant and all be sent within the authentication request and received on the server side PHP hook that ensures account has been created. I will give it a thought and get back asap.

Thank you!!

In T416100#11644444, @phuedx wrote:

This will work but delivery simply won't be as reliable, as @Sgs noted in T416100#11620735.

Wouldn't client side be unreliable for capturing account_created events as they are prone to be cut-off by add-blockers, private browsing settings and others while the server side account creation is reliable

AMAZING WORK. I did some benchmarking (using the synth A/A/A test with 808K events) and the speed gains are EXQUISITE:

If you generate the version string by hashing the appropriate properties (e.g. include machine-readable name, exclude friendly name), then you don't need to worry about friendly database tinkerers keeping the version field up to date.

The scenario I mentioned is possible because GB are limited in what configs/dbs they regularly test on (ref. https://github.com/growthbook/growthbook/pull/5298) and we know we're basically the only ones using kerberos auth, so if anything emerges there we're the canary.

So, at first I was thinking production data lake because one check we need to be able to do with growthbook-next is presto connector and kerberos authentication. If GB releases a version that accidentally breaks either of those, we should detect that in gb-next rather than in main/prod instance.

Thanks so much for thinking through it and thank you for including potential code snippets to inform the implementation! I love the 2-tier record/ledger idea.

In T416100#11611292, @Sgs wrote:

I'm trying to implement the instrumentation re-using as much existing instruments and metrics as we can and I have some trade-off questions:

1. We seem to want to capture 4 CTRs, one for each of Sign up, Log in, Edit without logging in and Temporary accounts / learn more links. I was thinking of using the generic Clickthrough per user but the documentation for CTPU states It is generic because it does not differentiate between impressions/clicks from multiple sources (instruments) of impression/click events that might be present in the data collection for an experiment, so I guess for 4 different CTRs we'd need custom queries similar to the Getting started notification clickthrough which had one CTR for each notification type and primary/secondary link.
   • Does option (a) ensure a greater outreach given the .1% traffic limitation is per-experiment?
   • What's the recommendation for this recurring setup that has less overhead in order to streamline automated analysis: (a) 4 experiments with a generic CTR (CTPU) which would result in 4 superset automated analysis dashboards. Or (b) a single experiment with custom queries per-CTR and a condensed dashboard, same as we did for the Getting notification setup?
   • If this setup (or another) becomes a pattern, would it be worth on working on some template queries?

@Sfaci @KReid-WMF: @phuedx and I were talking about the future where metric specifications specify which contextual attributes should be collected per event, rather than per instrument/experiment. We've seen a need for this already, both with the exposure logging work and Editing & Growth's experiments where only edit_saved events need page_namespasce_id and page_revision_id but those aren't needed for any other events.

Santi and I confirmed that the contributors experiments events are now decorated with the correct contextual attributes.

@Sfaci: Sorry for last minute scope change, I just realized we also need performer_is_bot as one of the contextual attributes.

Via the background queue (implemented in DefaultEventSubmitter) and via a new method that calls navigator.sendBeacon() directly, e.g. Instrument#sendImmediately()

@cjming: By the way, the mechanism here doesn't have to be super complicated or "perfect". We just need a way of avoiding unnecessarily logging exposure if/when we can.

(Sounds good!)

1. Notable observations

1. v0.1 (initial release)

@MoritzMuehlenhoff: Oh that's a great point. Yes, the outcome is that @AKhatun_WMF would have admin rights on the analytics_product Airflow instance. If getting that outcome requires the membership you mentioned, please make the necessary change.

@MNeisler: The new instrumentation is live. I was able to test some of the new queries from the MR, but will have to wait until later today when there's more data collected & processed.

Thanks! I copied that into the acceptance criteria and updated exp_platform to experiment_platform

About https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/1237223/3/includes/Specials/SpecialHomepage.php#137…

SuggestedEdits::isActivated( $this->getContext()->getUser()

seems more accurate for action: experiment_exposure event than

SuggestedEdits::isEnabledForAnyone( $this->wikiConfig )

We want the exposure event to only fire if the suggested edits module has been activated and contains either the control or the treatment experience. We don't want exposure event on a homepage visit where the module hasn't been activated and so there is no actual exposure.

As the approving party for both groups (and the person requesting this access), I approve @AKhatun_WMF's membership.

There is still some nuance here about what exactly counts as an exposure. But based on Kirsten's general support above, I'll go for the users visiting the homepage when SuggestedEdits are enabled for the wiki. Please let me know if that is not sufficiently precise enough!

@JVanderhoop-WMF's thoughts:

I think the use case here is really specific: only some folks in the treatment group created reading lists, and they want to ensure that those folks are told that it's now a beta feature, that their list lives on, etc.

This seems rare, and I am concerned about a slippery slope of providing more "individual" information rather than the aggregates that matter in our A/B testing context. (Though I'm not clear that I've articulated that feeling all too clearly here)

I think exposure logging guidance probably warrants its own guide/page which is then linked to from the conduct an experiment guide.

@MNeisler and I discussed this and arrived at the following proposal:

Thank you!

Some additional data to assist with investigation: https://docs.google.com/spreadsheets/d/1P3_8tGbg3Suvfa1q8TrgAXv_jk29_z7cfkOenXDuQYE/edit?gid=1285859969#gid=1285859969 (WMF internal only)

@KReid-WMF @phuedx: I think Reader Growth has a pretty good pattern/practice that we can recommend in the docs as part of T414735: Update documentation with guidance on exposure logging.