Add-a-link Experiment on Enwiki

Activation, Retention, Revert Rate

WE1.2.11, FY24-25
Author
Affiliation

Irene Florez

Product Analytics, Wikimedia Foundation

Published

06/29/2025

This page in a nutshell

This analysis finds that on the mobile web platform, the Add-a-link results are positive: a 33.7% increase in the constructive activation rate, a 3.7% increase in the constructive retention rate, and a 19.6% decrease in revert rate. Users assigned to the treatment group with the Add-a-link feature enabled saw improvement across our three experiment metrics compared to the control group.

Introduction and Basics

In the second half of 2024, the Wikimedia Foundation began gradually introducing the “Add-a-link” structured task to newcomers at English Wikipedia T386029. Structured tasks are suggested edits that can be broken down into step-by-step workflows with simple steps that make sense to newcomers, are easy to do on mobile devices, and may be assisted by machine learning. Structured Task “Add-a-link” recommendations exist on articles that need links and are underlinked; some of these may be lower view count, shorter, and/or less developed articles.

From “turned off” in August 2024 the feature was released to 2% on November 25th and then progressively further rolled out to 20% in March 2025. This study evaluates the Add-a-link impact on constructive activation, constructive retention, and revert rates in English Wikipedia T382603, utilizing the gradual roll out as an A/B test opportunity.

We build on the previous 2021 study involving ten Wikipedias (Arabic, Bengali, Czech, Vietnamese, Russian, French, Polish, Romanian, Persian, Hungarian) which found that newcomers who received the Add-a-Link structured task were 11.7% more likely to make a first article edit compared to the baseline in the control group.

Here we conducted a two-arm between-subjects randomized controlled trial using a parallel-group design, with a phased variant allocation ramp from 2% to 20%. Newcomers on English Wikipedia were assigned to one of two groups: treatment or control. Outcome measures were assessed at the end of the experiment via edit data.

Constructive activation is our primary metric. Revert rate and constructive retention are secondary metrics in this experiment. For constructive activation, we sampled constructive activation data, analyzed the full dataset and modeled the relationships. For revert rate we analyzed revert rate findings and also pulled general revert rate data for newcomers on English Wikipedia during the experiment period to contextualize our findings. For constructive retention we analyzed the full dataset and report on the findings.

Finally, as mobile is the focus of the 1.2 KR we focus on mobile data results and provide further breakdowns where possible.

Findings Summary

Results reveal that the Add-a-link structured task improves outcomes for newcomers over a control group that did not have access to the “add-a-link” tasks.

Newcomers who get the Add-a-link structured task are more likely to be activated (i.e. make a constructive, non-reverted, first article edit). When analyzing the full data set we see a 17.2% constructive activation rate for those in the mobile treatment group compared to those in the mobile control group which saw a 12.9% constructive activation rate. These differences are statistically significant. Add-a-link increases constructive activation by 33.7% relative to the control on mobile.

We see a 3.7% increase in Constructive Retention (article) for mobile web editors in the treatment group (3.1%) relative to the control (3.0%). The findings confirm the original hypothesis.

Mobile web treatment group editors experienced a 19.6% lower revert rate (32.0%) than mobile control group editors (39.8%). The findings confirm the original hypothesis.

We can conclude that there is enough evidence to roll out this treatment widely at this time on English Wikipedia.

Figure 1: Add-a-Link: Screenshot from “Add a link” feature on Simple English Wikipedia

Constructive Activation (Article) A newcomer making at least one edit to an article in the main namespace on a mobile device within 24 hours of registration, with that edit not being reverted within 48 hours of publication. In this notebook we use the variable is_const_activated_article. is_const_activated_article = (num_article_edits_24hrs - num_article_reverts_24hrs) > 0

Constructive Retention (Article) If we increase constructive activation, but that doesn’t flow into retained users, then the impact of this work will be limited. We ensure mobile web newcomer retention remains stable or improves. In this notebook we use the variable is_const_retained_article is_const_retained_article = is_const_activated_article & ((num_article_edits_2w - num_article_reverts_2w) > 0)

Revert rate The proportion of namespace = 0 edits that were reverted within 48 hours out of all such edits made, for users that edited on mobile. This is by definition 0% for users who made no edits, and we exclude these users from the revert rate analysis. We use edit tags to identify edits and reverts, and reverts have to be done within 48 hours of the edit. In this notebook we use the variable prop_rev_article_edits. prop_rev_article_edits = num_article_reverts_24hrs + num_article_reverts_2w) / num_total_article_edits if num_total_article_edits > 0; otherwise it is set to 0. Essentially we take the average revert rate across users in the specified group.

Detailed Findings: Activation

Hypothesis: For new logged in account holders (account <24hrs) on English Wikipedia, if we introduce the “Add-a-link” Structured Task in Wikipedia articles, then we expect to increase the percentage of new account holders who constructively activate on mobile web by 10% compared to the control group.

Constructive Activation (Article) is this experiment’s primary metric.

Takeaway

The findings from analyzing the full dataset confirm the original hypothesis and surpass initial expectations.

When analyzing the full data set we see a 17.2% constructive activation rate for those in the mobile treatment group compared to those in the mobile control group which saw a 12.9% constructive activation rate. These differences are statistically significant.

Add-a-link increases constructive activation by 33.7% relative to the control on mobile.

Show the code 
display_html(as.character(t9))
Summary - Constructive Activation (Article) Aggregations by Platform
platform Group Count Percent
desktop Control 10152 15.4
desktop treatment 1057 16.9
mobile Control 4155 12.9
mobile treatment 508 17.2
Show the code 
display_html(as.character(t12))
Summary Comparison: Constr. Mobile Web Activation (Article)
Control<->Treatment %Changes
Metric Value
Mobile Control Percent 12.9
Mobile Treatment Percent 17.2
AbsoluteChange 4.3
PercentagePointChange 4.3
PercentChange 33.7
Show the code 
constructive_activation_article_namespace_mobile

Detailed Findings: Retention

Hypothesis: Mobile web users who receive the Add-a-link structured task will have a 3% or higher retention rate than mobile web users who do not.

If we increase constructive activation, but that doesn’t flow into retained users, then the impact of this work will be limited. Thus we look to ensure newcomer Constructive Retention (article) remains stable or improves.

Retention is a secondary metric we track in this experiment.

Takeaway

We see a 3.7% increase in Constructive Retention (article) for mobile web editors in the treatment group relative to the control. The findings confirm the original hypothesis.

Show the code 
display_html(as.character(aggr_tbl_retention_platform_render))
Summary - Constructive Retention (Article) by Platform
platform Group Count Percent
desktop Control 2608 4.0
desktop treatment 261 4.2
mobile Control 972 3.0
mobile treatment 92 3.1
Show the code 
display_html(as.character(aggr_tbl_retention_platform_render_comp_render))
Summary Comparison: Constructive Mobile Web Retention (Article)
Control<->Treatment %Changes
Metric Value
Mobile Control Percent 3.0
Mobile Treatment Percent 3.1
AbsoluteChange 0.1
PercentagePointChange 0.1
PercentChange 3.7
Show the code 
constructive_retention_article_namespace_mobile

Detailed Findings: Revert Rate

Hypothesis: “Add-a-link” Structured Task mobile web participants will not experience a higher edit revert rate than mobile web editors in the control group.
Revert Rate is a secondary metric we track in this experiment.

Takeaway

Mobile web treatment group editors experienced a 19.6% lower revert rate (32.0%) than mobile control group editors (39.8%). The findings confirm the original hypothesis.

These figures reflect the revert rates for all article edits made by users in each group on English Wikipedia rather than only edits made through the “Add-a-link” structured task. Edits made via “Add-a-link” specifically have a lower revert rate than the overall rates presented here.

For context, during the experiment period, English Wikipedia new editors saw a 33.6% revert rate on desktop and 42.9% mobile.

When it comes to reverts, we again focus on the Article namespaces because that is where Add-a-link asks newcomers to edit. Secondly, it does not make sense to measure reverts for users who make no edits, so this analysis is limited to users who made at least one edit in those namespaces in the first two weeks after registration.

While revert rate is our best measurement of the quality of edits, it is important to note that conversations with communities have indicated that revert rate may not be a fully accurate proxy for the quality of Add-a-link edits:

  • Add-a-link edits frequently add multiple links to an article. In some cases when most of the links are an improvement and some are not, patrollers may not go through the process of reverting the whole edit, and rather let a partially good edit stay.
  • In those same situations, patrollers may manually remove some of the links while keeping others. These “partially reverted” edits are not detected as reverts by our analysis.
Show the code 
display_html(as.character(avg_rr_by_platform_render))
Revert Rate per Group - mean, by Platform
platform Group n Prop_Rev_Article_Edits Percent
desktop Control 16729 0.32 32.2
desktop treatment 1667 0.29 29.3
mobile Control 7057 0.40 39.8
mobile treatment 753 0.32 32.0
Show the code 
display_html(as.character(avg_rr_platform_lift_render))
Summary Comparison: Revert Rate Mobile Web
Control<->Treatment %Changes
Metric Value
Mobile Control Percent 39.8
Mobile Treatment Percent 32.0
AbsoluteChange -7.8
PercentagePointChange -7.8
PercentChange -19.6
Show the code 
rr_mobile

Methodology

Here we conducted a two-arm between-subjects randomized controlled trial using a parallel-group design, with a phased variant allocation ramp from 2% to 20%, resulting in a 11:1 split between control and treatment groups. Outcome measures were assessed at the end of the approximately four month experiment period via edit data.

Because constructive activation is the primary experiment metric, we sampled constructive activation data, analyzed the full dataset, and modeled the relationships. Leveraging generalized linear models (GLMs) with a binomial link function and logistic regression–based inference, we reviewed the outcomes of those that received the feature (treatment) and those that did not (control) and modeled the data to understand how treatment/control and desktop/mobile factors influenced the odds of constructive activation. Given the 11:1 split between control and treatment groups, random sampling informed by prior power analysis served as a confirmatory analysis alongside modeling to validate observed differences in constructive activation rates.

Revert rate and constructive retention were secondary metrics in this experiment. For revert rate we analyzed revert rate findings and also pulled general revert rate data for newcomers on English Wikipedia during the experiment period to contextualize our findings.

Finally, as mobile is the focus of the 1.2 KR we focused on mobile data results and provided further breakdowns where possible.

  • Control-to-Treatment Allocation Ratio: The phased variant allocation ramp from 2% to 20% resulted in a 11:1 split between control and treatment groups. This ration may reduce precision in the smaller treatment arm and limit generalizability of the findings.
  • Allocation Point vs Interaction Point: Analysis was based on intent-to-treat and included all users assigned to the treatment arm, regardless of whether they actually saw or interacted with the feature, biasing results toward the null.
  • One-sided noncompliance (users assigned to treatment choosing not to engage) dilutes the estimated effect.
  • Target audience: Welcome Survey data suggests that many newly registered users create accounts with a specific edit or article creation in mind. They may be less likely to respond to suggestions not matching their initial intention.

  • Control-to-Treatment Allocation Ratio: Based on prior power calculations, we randomly downsampled the control group during exploratory analysis of our primary metric (independent of our main modeling analysis) to balance group sizes, ensure fair comparisons, and yield more precise treatment-effect estimates—confirming that our primary metric findings remained consistent.

  • Allocation vs. Interaction Point: We applied the same exposure-aware analytical framework used in previous experiments to ensure consistency and enable direct comparisons across studies.

In this experiment, the “Add-a-link” structured task is being introduced to newcomers via a controlled, gradual roll-out at English Wikipedia. Newcomers are randomly assigned into one of two groups: those who receive the existing interface experience (the control group) and those who receive the interface with the “Add-a-link” structured task enabled (the treatment group). Known users to exclude are explicitly excluded so that experimental results aren’t skewed by test traffic/engagement.

Exposure to the variant increased from 2% to 20% during the experiment period.

Analysis in this experiment was carried out using edit and revert data from English Wikipedia for those in the experiment, focusing on edits to articles in the main namespace made on any device. Mobile and desktop platform splits in this analysis are based on the platform where the user first registered an account (based on the reg_on_mobile field in the user dataset). This method follows previous practices for similar analysis. While we don’t expect that there are many users that switch platforms and so didn’t search for edits where array_contains(revision_tags, 'mobile web edit') here, in future Growth analysis we will search for mobile web edit edits specifically to match queries now utilized by the editing team and others.

The data sources are:

  • event.homepagemodule: Get user treatment/control assignments
  • wmf.mediawiki_user_history: Get user registrations on the given wikis for the given dates, ignoring users on the exclude list and excluding bots
  • user: Grab the user IDs of known test accounts so they can be added to the exclusion list
  • event_sanitized.serversideaccountcreation: Identify all self-created, non-app created, non-bot registrations using ServerSideAccountCreation.
  • event_sanitized.mediawiki_revision_create, event_sanitized.mediawiki_revision_tags_change: Gather data to answer questions about our high level metrics: activation, retention, productivity, and revert proportions for those user ids in the experiment.
  • The EPIC phab task for the larger area of work is T304110
  • Deployment task information is in ticket T395524
  • The analysis ticket is T382603

Experiment Dates and Group Counts

Show the code 
cat("Experiment enrollment began on:",format(exp_start_ts, "%Y-%m-%d %H:%M:%S"), "\n") 
cat("Experiment enrollment ended on:",format(exp_end_ts, "%Y-%m-%d %H:%M:%S"), "\n")
cat("We collected edit data from experiment start up to:", format(end_date_plus15, "%Y-%m-%d %H:%M:%S"), "\n")
cat("We collected revert data up to:", format(edits_plus_two, "%Y-%m-%d %H:%M:%S"), "\n")
Experiment enrollment began on: 2024-11-25 00:00:00 
Experiment enrollment ended on: 2025-03-15 23:59:59 
We collected edit data from experiment start up to: 2025-03-30 23:59:59 
We collected revert data up to: 2025-04-01 23:59:59
Show the code 
#Convert to a character string containing the HTML code & display
display_html(as.character(tbl_html))
Summary: Group Sizes
platform Group Group_count Experiment_total_count Percent
desktop Control 66102 72366 91.34
desktop treatment 6264 72366 8.66
mobile Control 32242 35190 91.62
mobile treatment 2948 35190 8.38

Appendix

Constructive Activation Overall (Control vs Treatment)

Show the code 
display_html(as.character(aggr_tbl_const_activation_article_render))
display_html(as.character(aggr_tbl_const_activation_article_comp_render))
constructive_activation_article_namespace
Summary - Constructive Activation (Article) Aggregations
Group Count Percent
Control 14307 14.6
treatment 1565 17.0
Summary Comparison: Constructive Activation (Article) Control<->Treatment % changes
Metric Value
Control Percent 14.6
Treatment Percent 17.0
AbsoluteChange 2.4
PercentagePointChange 2.4
PercentChange 16.8

Show the code 
display_html(as.character(f))
Summary - Mobile Constructive Activation (Article) by Wiki
Group wiki_db Percent
Control enwiki 12.9
treatment enwiki 17.2

Constructive Retention Overall (Control vs Treatment)

Show the code 
display_html(as.character(aggr_tbl_const_retained_article_render))
display_html(as.character(aggr_tbl_const_retained_article_comp_render))
constructive_retention_article_namespace_overall
Summary - Constructive Retention (Article)
Group Count Percent
Control 3580 3.6
treatment 353 3.8
Summary Comparison: Constructive Retention (Article) - Control<->Treatment %Changes
Metric Value
Control Percent 3.6
Treatment Percent 3.8
AbsoluteChange 0.2
PercentagePointChange 0.2
PercentChange 5.2

Revert Rate Overall (Control vs Treatment)

Show the code 
display_html(as.character(avg_rr_render))
display_html(as.character(avg_rr_lift_render))
rr_overall
Revert Rate per Group - Mean
Group n Prop_Rev_article_Edits Percent
Control 23786 0.34 34.4
treatment 2420 0.30 30.1
Summary Comparison: Revert Rate Control<->Treatment %Changes
Metric Value
Control Percent 34.4
Treatment Percent 30.1
AbsoluteChange -4.3
PercentagePointChange -4.3
PercentChange -12.5