Page MenuHomePhabricator
Create Task
Maniphest T182597

Use EtcdConfig in production to allow automation of a datacenter switch
Closed, ResolvedPublic
Actions

Description

The switchover of the master datacenter in mediawiki requires the following steps:

  1. Set the databases in the master datacenter in read-only mode
  2. Switch the wmfMasterDatacenter variable to the new datacenter
  3. Switch the databases in the new master datacenter

each one of these changes requires traditionally a git commit and a deploy. Our goal is to make switching easier, and - per T149617 - we've chosen to use Etcd to that end.

We have now EtcdConfig that can do that inside mediawiki-config, we just need to thoroughly test it and bring that to production.

Our objectives are thus as follows

  • Test EtcdConfig in different failure scenarios T185078
  • Prepare conftool for safely editing mediawiki-config values (additional validation, separate etcd credentials)
  • Deploy EtcdConfig to production
  • Allow use of EtcdConfig to configure slave databases (stretch) T185084

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Enable use of EtcdConfig everywhere.operations/mediawiki-configmaster+1 -8
Icinga: add EtcdConfig sync check on MW hostsoperations/puppetproduction+6 -0
Icinga: use systemd::service for etcd_mw_configoperations/puppetproduction+1 -1
Icinga: add sync check for MW config on etcdoperations/puppetproduction+224 -2
Fetch data from etcd on every serveroperations/mediawiki-configmaster+14 -13
Fetch the last modified index in etcd.php, and expose it via siteinfo.operations/mediawiki-configmaster+9 -1
Expose the latest modified index seen by EtcdConfigmediawiki/corewmf/1.31.0-wmf.23+130 -96
Expose the latest modified index seen by EtcdConfigmediawiki/coremaster+130 -96
Enable EtcdConfig on the debug hostsoperations/mediawiki-configmaster+5 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

Joe created this task.Dec 11 2017, 3:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 11 2017, 3:35 PM
Volans triaged this task as Medium priority.Dec 19 2017, 11:33 AM
Volans subscribed.
Joe renamed this task from [DRAFT] Use EtcdConfig in production to allow automation of a datacenter switch to Use EtcdConfig in production to allow automation of a datacenter switch.Jan 8 2018, 2:37 PM
Volans updated the task description. (Show Details)Jan 17 2018, 10:00 AM
elukey subscribed.Feb 12 2018, 4:21 PM
gerritbot subscribed.Feb 16 2018, 4:26 PM

Change 411286 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[mediawiki/core@master] Expose the latest modified index seen by EtcdConfig

https://gerrit.wikimedia.org/r/411286

gerritbot added a project: Patch-For-Review.Feb 16 2018, 4:26 PM
gerritbot added a comment.Feb 16 2018, 4:54 PM

Change 411296 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Enable EtcdConfig on the debug hosts

https://gerrit.wikimedia.org/r/411296

gerritbot added a comment.Feb 21 2018, 2:41 PM

Change 411296 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable EtcdConfig on the debug hosts

https://gerrit.wikimedia.org/r/411296

gerritbot added a comment.Feb 22 2018, 11:36 AM

Change 413355 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] Icinga: add sync check for MW config on etcd

https://gerrit.wikimedia.org/r/413355

gerritbot added a comment.Feb 22 2018, 11:36 AM

Change 413356 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] Icinga: add EtcdConfig sync check on MW hosts

https://gerrit.wikimedia.org/r/413356

Joe closed subtask T185080: Prepare conftool for safely editing mediawiki-config values as Resolved.Mar 1 2018, 7:02 AM
gerritbot added a comment.Mar 2 2018, 3:59 AM

Change 411286 merged by jenkins-bot:
[mediawiki/core@master] Expose the latest modified index seen by EtcdConfig

https://gerrit.wikimedia.org/r/411286

gerritbot added a comment.Mar 5 2018, 11:27 AM

Change 416411 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[mediawiki/core@wmf/1.31.0-wmf.23] Expose the latest modified index seen by EtcdConfig

https://gerrit.wikimedia.org/r/416411

gerritbot added a comment.Mar 6 2018, 8:39 AM

Change 416470 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Fetch the last modified index in etcd.php, and expose it via siteinfo.

https://gerrit.wikimedia.org/r/416470

gerritbot added a comment.Mar 6 2018, 8:39 AM

Change 416482 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Fetch data from etcd on every server

https://gerrit.wikimedia.org/r/416482

gerritbot added a comment.Mar 6 2018, 8:39 AM

Change 416483 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Enable use of EtcdConfig everywhere.

https://gerrit.wikimedia.org/r/416483

ReleaseTaggerBot added a project: MW-1.31-release-notes (WMF-deploy-2018-03-06 (1.31.0-wmf.24)).Mar 6 2018, 12:01 PM
gerritbot added a comment.Mar 6 2018, 3:20 PM

Change 416411 merged by Giuseppe Lavagetto:
[mediawiki/core@wmf/1.31.0-wmf.23] Expose the latest modified index seen by EtcdConfig

https://gerrit.wikimedia.org/r/416411

gerritbot added a comment.Mar 6 2018, 3:42 PM

Change 416470 merged by jenkins-bot:
[operations/mediawiki-config@master] Fetch the last modified index in etcd.php, and expose it via siteinfo.

https://gerrit.wikimedia.org/r/416470

ReleaseTaggerBot edited projects, added MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)); removed MW-1.31-release-notes (WMF-deploy-2018-03-06 (1.31.0-wmf.24)).Mar 6 2018, 4:00 PM
gerritbot added a comment.Mar 6 2018, 4:05 PM

Change 416482 merged by jenkins-bot:
[operations/mediawiki-config@master] Fetch data from etcd on every server

https://gerrit.wikimedia.org/r/416482

Joe closed subtask T185078: Test EtcdConfig in different failure scenarios as Resolved.Mar 7 2018, 12:01 PM
gerritbot added a comment.Mar 8 2018, 10:39 AM

Change 413355 merged by Volans:
[operations/puppet@production] Icinga: add sync check for MW config on etcd

https://gerrit.wikimedia.org/r/413355

gerritbot added a comment.Mar 8 2018, 11:33 AM

Change 417232 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] Icinga: use systemd::service for etcd_mw_config

https://gerrit.wikimedia.org/r/417232

gerritbot added a comment.Mar 8 2018, 11:39 AM

Change 417232 merged by Giuseppe Lavagetto:
[operations/puppet@production] Icinga: use systemd::service for etcd_mw_config

https://gerrit.wikimedia.org/r/417232

gerritbot added a comment.Mar 8 2018, 12:31 PM

Change 413356 merged by Volans:
[operations/puppet@production] Icinga: add EtcdConfig sync check on MW hosts

https://gerrit.wikimedia.org/r/413356

gerritbot added a comment.Mar 8 2018, 2:48 PM

Change 416483 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable use of EtcdConfig everywhere.

https://gerrit.wikimedia.org/r/416483

Joe updated the task description. (Show Details)Mar 8 2018, 3:32 PM
Joe closed subtask T188922: EtcdConfig: add Icinga check as Resolved.

The main part of the goal is now done. We will try to work on the stretch part if we get to it.

ArielGlenn removed a project: Patch-For-Review.Nov 16 2018, 11:10 AM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:38 PM
Joe closed subtask T185084: Allow use of EtcdConfig to configure slave databases as Resolved.Aug 21 2019, 10:55 AM
Marostegui updated the task description. (Show Details)Aug 21 2019, 10:57 AM
Joe closed this task as Resolved.Mar 2 2022, 7:15 AM
Joe claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits