Parsoid/Round-trip testing

The Parsoid code includes a round-trip testing system that tests code changes, composed of a server that gives out tasks and presents results and clients that do the testing and report back to the server. The code is in the testreduce repo which is an automatic mirror of the repo in gerrit. The roundtrip testing code has been fully puppetized.

There's an instance of the server on ctt-rt-testing-01.wikitextexp.eqiad1.wikimedia.cloud that currently tests a representative (~160000) set of pages from different wikipedia languages. You can access the web service at https://parsoid-rt-tests.wmcloud.org/

Private setup

The instructions to set up a private instance of the round-trip test server can be found here. A MySQL database is needed to keep the set of pages and the testing results.

Overview

Before we dive into the details here is an overview of how the current setup looks like.

• mw-parsoid is a dedicate mediawiki instance on k8s that runs the exact same setup like production but mounts an overridden copy of parsoid
• mw-experimental is the host where devs can change parsoid codebase for dev/debugging purposes
• mw-parsoid mounts a read-only version of /srv/parsoid-testing and on the k8s pod itself devs can only write at /tmp
• mw-parsoid pod runs the production wikimedia image, as a result the env is very stripped down
• mw-parsoid mediawiki instance can be accessed via HTTP using the x-wikimedia-debug header from regular traffic
• testreduce server is hosted in cloudvps (since mw-parsoid can be reached by public traffic could be any VM, not a private network like it used to be)

Parsoid testing MW instance (mw-parsoid)

The docs for mw-parsoid can be found here: https://wikitech.wikimedia.org/wiki/Mw-parsoid

We have a dedicated mediawiki instance on k8s for parsoid testing. It enables overriding the parsoid codebase under /srv/parsoid-testing by providing devs write access (both for pulling specific commits and for manual intervention). The setup is similar to mw-debug which lets requests to be routed to a specific backend via an http header.

More specifically in our case this will GET the page using the overridden dev parsoid version, and more specifically the one on eqiad:

curl https://en.wikipedia.org/wiki/Earth -H "X-Wikimedia-Debug: k8s-mw-parsoid-eqiad"

There are two envs: k8s-mw-parsoid-eqiad, k8s-mw-parsoid-codfw. The parsoid files can be accessed via the experimental nodes.

# ssh mw-experimental.{codfw,eqiad}.wmnet
# cd /srv/parsoid-testing

mw-experimental instance is just used for mounting the overridden version of parsoid. This is not the server that mediawiki/php runs.

To check out to a specific version of parsoid:

# ssh mw-experimental.{codfw,eqiad}.wmnet (depending on which env you are working on)
# cd /srv/parsoid-testing
# git checkout <COMMIT>

Wikidevs have right access, so editing /srv/parsoid-testing manually is also possible

Deploying mw-parsoid

mw-parsoid is not deployed with the rest of the train. We need to manually deploy to use the current mediawiki images. Our test scripts do this as part of the orchestration steps. For manual deployments:

> ssh deployment.eqiad.wmnet
> cd /srv/deployment-charts/helmfile.d/services/mw-parsoid
> helmfile -e eqiad apply
> helmfile -e codfw apply

RT-testing setup

Coordinator runs on ctt-rt-testing-01.wikitextexp.eqiad1.wikimedia.cloud. RT-testing clients run on the same node. The node is hosted on cloudvps and you need to make sure you have access as described in the docs. These clients access Parsoid REST API that runs on mw-parsoid a dedicated parsoid testing instance of mediawiki that is hosted in kubernetes.

The clients are managed/restarted by systemd and the config is in /etc/systemd/system/parsoid-rt-client.service.

To {stop,restart,start} all clients on a VM (not normally needed):

# On ctt-rt-testing-01
sudo service parsoid-rt-client stop
sudo service parsoid-rt-client restart
sudo service parsoid-rt-client start

Client logs are in systemd journals and can be accessed as:

### Logs for the parsoid-rt-client service on ctt-rt-testing-01
# equivalent to tail -f <log-file>
sudo journalctl -f -u parsoid-rt-client
# equivalent to tail -n 1000
sudo journalctl -n 1000 -u parsoid-rt-client

### Logs of the parsoid-rt testreduce server
sudo journalctl -f -u parsoid-rt

### Logs for the parsoid service
sudo journalctl -f -u parsoid

In the current setup, the testreduce clients talk to a global parsoid service that runs on mw-parsoid. There is a dedicated logstash dashboard for this instance.

Starting a test run

It's probably best to check that we're not currently running tests on a parsoid commit. Use the sudo journalctl -f -u parsoid-rt-client command on ctt-rt-testing-01 to verify that it says "The server does not have any work for us right now".

To start rt-testing a particular parsoid commit, run the following command on your local computer from your checked-out copy of Parsoid:

# To test a commit which is already merged on Parsoid's main/master branch:
bin/start-rt-test.sh <sha-of-parsoid-code-to-rt-test>

# Add -u if your .ssh/config does not automatically use the correct userid:
# Example: bin/start-rt-test.sh -u ssastry 645beed2

# You can also test a commit which has not yet been merged:
bin/start-rt-test.sh --gerrit <changeid>

This updates the parsoid checkout on mw-parsoid and ctt-rt-testing-01, deploys the latest mediawiki image on mw-parsoid and parsoid-rt-client services.

(Note: if it complains "detected dubious ownership in repository at /srv/parsoid-testing" go ahead and ssh into both ctt-rt-testing-01 and mw-experimental and run the command that it suggests: git config --global --add safe.directory /srv/parsoid-testing.)

Updating the round-trip server code

The rt-server code lives in /srv/testreduce.You can run npm install on the testreduce serve if you need to update node modules.

cd /srv/testreduce
git pull
sudo service parsoid-rt restart

Running the regression script

After an rt run, we compare diffs with previous runs to determine if we've introduced some new semantic differences. However, since the runs happen on different dates and the production data is used, there's going to be some natural churn to account for. The regression script automates the process of rerunning the rt script on a handful of pages to determine if there are any true positives

# Use https://parsoid-rt-tests.wmcloud.org/commits to select a commit pair to
# check, then click "Regressions" at the top, and pass the resulting URL to this
# command run on yourlocal machine
# Make sure that an rt run isn't currently in progress
php tools/RegressionTesting.php --url https://parsoid-rt-tests.wmcloud.org/regressions/between/<oracle>/<commit>

# Alternatively, you can manually create a list of titles to check, and use:
php tools/RegressionTesting.php -t <title-file> <oracle> <commit>

# Again, `-u <bastion-uid>` can be added if it's not already in your `.ssh/config`

Note that the script will checkout the specified commits while running but that it doesn't do anything for dependencies -- Parsoid will be running in integrated mode with the latest production mediawiki version(s) and their corresponding mediawiki-vendor packages (depending on the Host field used in the request). So, at present, it isn't appropriate when bumping dependency versions in between commits. The <oracle> ("known good") and <commit> ("to be tested") can be anything that git recognizes, including tag names -- they don't necessarily have to correspond to the hashes you provided to the rt server, although usually that's what you'll use.

Crashers prevent the script from running and may need to be pruned. They can then be tested individually as follows:

# On mw-experimental, checkout the commit you want test
ssh mw-experimental.eqiad.wmnet
cd /srv/parsoid-testing
git checkout <somecommit>

# On ctt-rt-testing-01, run the roundtrip script
node bin/roundtrip-test.js --parsoidURL https://DOMAIN/w/rest.php --domain en.wikipedia.org "Sometitle" --headers '{"X-Wikimedia-Debug":"k8s-mw-parsoid-eqiad"}'

Finally: be sure to check on the parsoid-tests dashboard for notices and errors; that will be our first warning of logspam created by our tested release.

Ad hoc roundtrip

To run an ad-hoc run of round trip testing for a specific page:

> ssh ctt-rt-testing-01.wikitextexp.eqiad1.wikimedia.cloud
> cd /srv/parsoid-testing
> node bin/roundtrip-test.js \ 
    --headers '{"X-Wikimedia-Debug":"k8s-mw-parsoid-eqiad"}' \
    --parsoidURL https://DOMAIN/w/rest.php \
    --domain en.wikipedia.org \
    Dog

With the current setup, mw-parsoid is exposed to public traffic so the same command can also run from your local env

Running parsoid tools on mw-parsoid

Mw-parsoid is a k8s dedicated deployment of mediawiki, using the exact same setup of the production kubernetes. The difference is that it runs with limited resources and mounts a checked out version of the parsoid repository so devs and rt-testing can override the parsoid version.

Mw-parsoid is read only (devs have write access only on /tmp). The parsoid code is hosted in mw-experimental. All the code changes for debugging purposes or git operations should happen there.

Getting shell access

Currently this is a bit hacky but there is a script that devs can run from the home dir in deployment node. To get a bash shell to the running mw-parsoid container:

> ssh deployment.eqiad.wmnet
> # Download https://phabricator.wikimedia.org/P84273 as mw-experimental-shell
> chmod +x mw-experimental-shell
> ./mw-experimental-shell mw-parsoid eqiad
> # To run a maintenance command
> php /srv/mediawiki/multiversion/MWScript.php maintenance/getText.php --wiki enwiki Earth

To fetch files from the running container to the deployment node (eg. debug artifacts):

> ssh deployment.eqiad.wmnet
> # Download https://phabricator.wikimedia.org/P92440 as mw-experimental-fetch
> chmod +x mw-experimental-fetch
> ./mw-experimental-fetch mw-parsoid eqiad /tmp/foo ~/localtmp

To check out to a different version (or edit files) of /srv/parsoid-testing:

> ssh mw-experimental.eqiad.wmnet
> cd /srv/parsoid-testing
> git checkout <commit>

Important notes

• The container only mounts the parsoid codebase as read only.
• Getting a shell to the running container doesn't allow you to edit parsoid.
• All the code editing or git operations should happen in mw-experimental.{eqiad,codfw}.wmnet
• Devs have write access only on /tmp. Make sure that ad-hoc scripts write to a folder like /tmp/out-$(date +%s).
• The running container is very limited to what binaries it can run because is a very stripped down image.

Run maintenance scripts

Parsoid will run in integrated mode on mw-parsoid from /srv/parsoid-testing but it requires use of the MWScript.php wrapper in order to configure mediawiki-core properly. More information on mwscript is at Extension:MediaWikiFarm/Scripts.

A sample mediawiki maintenance command would look like:

$ ssh deployment.eqiad.wmnet
... fetch if not available the mw-experimental-shell script as described above ...
$ ./mw-experimental-shell mw-parsoid eqiad
$ echo '==Foo==' | php /srv/mediawiki/multiversion/MWScript.php /srv/parsoid-testing/bin/parse.php --wiki=hiwiki --integrated

For a more parsoid specific example:

$ ssh deployment.eqiad.wmnet
... fetch if not available the mw-experimental-shell script as described above ...
$ ./mw-experimental-shell mw-parsoid eqiad
$ cd /srv/parsoid-testing
$ echo "foo" | php /srv/mediawiki/multiversion/MWScript.php /srv/parsoid-testing/bin/parse.php --wiki enwiki --integrated
<p data-parsoid='{"dsr":[0,3,0,0]}'>foo</p>

Parts of that command are often abbreviated as an helper alias mwscript in your shell to make invocations easier.

The maintenance/parse.php script in mediawiki-core also has a --parsoid option that may be useful.

Tracing / dumping with MWScript.php wrapper

If you need to trace / dump via the --trace and --dump CLI options with the mwscript wrapper, you will need to use the --logFile option in parse.php to get the logs.

Requests to mw-parsoid

Mw-parsoid can't be accessed from public traffic using a specific header (same way mwdebug does).

$DEBUG_HEADER = "x-wikimedia-debug: k8s-mw-parsoid-eqiad"
$URL = "https://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/html/Dog"
curl -H $DEBUG_HEADER $URL

Overriding mediawiki codebase

By default mediawiki codebase served in the mw-parsoid instance is the mediawiki bundled with the container image. There are some cases were devs need to edit mediawiki codebase for debugging purposes (eg. adding a logger for an edge case to extract the value of a var). In order to do that the current setup allows mounting /srv/mediawiki from the experimental node to the pod.

We have 2 mw-parsoid envs

• eqiad
• codfw

By default rt-testing is using eqiad so codfw can be a good place for ad-hoc codebase changes.

In order to prepare the env for ad hoc edits these are the steps from a high level point of view

• Flip the mw-experimental flag on deployment charts
• Run a deployment on mw-parsoid
• Edit code in /srv/mediawiki on experimental node
• Flip the mw-experimental flag on deployment charts back to the default value
• Run a deployment on mw-parsoid

For more information on mw-experimental: https://wikitech.wikimedia.org/wiki/Mw-experimental

Notes

• Code changes are live for 1 hour
• Every hour the /srv/parsoid-testing is refreshed with the latest codebase
• Users can lock the environment for more, avoiding refreshing /srv/parsoid-testing (see mw-experimental docs)
• Since changing mediawiki code can have side effects its suggested that the flag is flipped back to false after debugging is done
• By default rt-testing is using eqiad so it might be a good idea to avoid any change on eqiad and use codfw for ad-hoc testing

Example

To enable ad hoc code editing in codfw

• Change the mw-experimental value in the deployment-charts (via a gerrit patch)
• Deploy mw-parsoid
  • ssh deployment.eqiad.wmnet
  • cd /srv/deployment-charts/helmfile.d/services/mw-parsoid/
  • helmfile -e codfw -i apply
• Change the codebase in /srv/mediawiki
  • ssh mw-experimental.codfw.wmnet
  • cd /srv/mediawiki
  • Make an edit under the mediawiki version that testwiki currently runs
• From your local env
  • curl "https://test.wikipedia.org/{path}" -H "x-wikimedia-debug: k8s-mw-parsoid-codfw"

To revert the codebase to latest and discard the local changes run: sudo systemctl restart mw-experimental-mediawiki-image-update.service

Troubleshooting

Disk space issues

As mentioned on the script, https://wikitech.wikimedia.org/wiki/Parsoid/Common_Tasks#Freeing_disk_space explains how to free disk space.

Wrong commit on mw-parsoid

It can happen that, when starting rt-testing, the commit is not updated on mw-parsoid. One reason for that can be a permission issue when checking out the repository, which may not interrupt the script. This may show up as unexpected results in the rt-tests or in the logs ("I should not get these errors with this commit").

Warning:

It's probably a good idea to have sudo service parsoid-rt-client status running in a terminal somewhere during the duration of the operations to check that it doesn't gets restarted.

• Find the commit that one on the top of the list in https://parsoid-rt-tests.wikimedia.org/commits
• Check on mw-parsoid that the commit in /srv/parsoid-testing is the same as above
• If they are different, congratulations, you found out what was weird in your rt-testing.
• Fixing things and restarting:
  • Log in to ctt-rt-testing-01
  • If the rt-testing is still running, run sudo service parsoid-rt-client stop.
  • Check that it's indeed stopped with sudo service parsoid-rt-client status.
  • Clear the database results for the previous execution. The full procedure for this requires some tweaking/additional documentation.:
    • Start mysql with mysql -u testreduce -p testreduce, use the password in /etc/testreduce/parsoid-rt.settings.js
    • delete from results where commit_hash='<hash>'; (note: this can take a few minutes, probably up to 10)
    • delete from stats where commit_hash='<hash>';
    • delete from perfstats where commit_hash='<hash>';
    • SOMETHING TBD needs to be done on the pages table to clear the crashers - deleting the lines corresponding to the commit is probably a Bad Idea, it seems plausible that something along the lines of update pages claim_num_tries=0, claim_timestamp=null,latest_stat=null,latest_result=null,latest_score=0,num_fetch_errors=0 where claim_hash = <hash> might do the trick, but this needs to be confirmed.
    • It's also unclear whether it's necessary/useful/problematic to remove the corresponding line in thecommit. It seems necessary, if doing that, to restart the parsoid-rt server so that the "known commits" are cleared an the line gets re-added. Maybe. Probably just update the timestamp and be done with it.
• Restart the rt-testing as if nothing had happened.

Grafana dashboard

There is a dedicated grafana dashboard prepared from the SREs for the mw-parsoid instance here

Currently tested steps

• Done Start roundtrip testing for a specific commit
• Done Start regression testing for a specific commit
• Done Trigger a roundtrip-test locally using custom headers for routing
• Done Get shell to the running container for manually running commands from deployment node
• Done Copy remote file/dir from a running container to deployment node
• Done Manual edit of /srv/parsoid-testing in ssh mw-experimental.{eqiad/codfw}.wmnet
• Done Dashboard for mw-parsoid specific logs on logstash

Todo / Roadmap

Please look at the general Parsoid roadmap.

Server UI and other usability improvements

We recently changed the server to use a templating system to separate the code from the presentation. Now other improvements could be done on the presentation itself.

Ideas for improvement:

• Improve pairwise regressions/fixes interface on commits list bug 52407. Done!
• Flag certain types of regressions that we currently search for by eye: create views with
  • Regressions introducing exactly one semantic/syntactic diff into a perfect page, and
  • Other introductions of semantic diffs to pages that previously had only syntactic diffs.
• Improve diffing in results views:
  • Investigate other diffing libraries for speed,
  • Apply word based diffs on diffed lines,
  • Diff results pages between revisions to detect new semantic/syntactic errors,
  • Currently new diff content appears before old, which is confusing; change this.
• Have a "clear last run" script
• Automatically prune the results DB after each run, so (a) the latest run is guaranteed to be preserved, and (b) we never have to manually run the free-disk-space script before starting an rt test run.