Page MenuHomePhabricator

Wikimedia-Multiple-active-datacentersTag
ArchivedPublic

Watchers (2)

Details

Description

Project archived and superseded by a subproject of Sustainability (Multiple-active-datacenters): https://phabricator.wikimedia.org/project/profile/2743/

Recent Activity

May 4 2017

Aklapper set the color for Wikimedia-Multiple-active-datacenters to Red.
May 4 2017, 7:53 AM

May 3 2017

Krinkle set the color for Wikimedia-Multiple-active-datacenters to Red.
May 3 2017, 7:59 PM
Krinkle archived Wikimedia-Multiple-active-datacenters.
May 3 2017, 7:51 PM
Krinkle edited Description on Wikimedia-Multiple-active-datacenters.
May 3 2017, 7:50 PM
Krinkle removed a project from T121440: Dedicated post-edit cache busting cookie to prevent stale reads (session consistency): Performance Issue.
May 3 2017, 7:31 PM · Performance-Team
Krinkle edited projects for T121440: Dedicated post-edit cache busting cookie to prevent stale reads (session consistency), added: Wikimedia-Multiple-active-datacenters; removed MW-1.27-release-notes.
May 3 2017, 7:29 PM · Performance-Team
Volans closed T160178: MediaWiki Datacenter Switchover automation as Resolved.

Resolving this after a successful MediaWiki switchover to codfw and switchback to eqiad using the automation software Switchdc (operations-switchdc on gerrit). The tracking task for improvements is T163363.

May 3 2017, 5:20 PM · Sustainability (MediaWiki-MultiDC), Patch-For-Review, DC-Switchover-Prep-Q3-2016-17, Epic, SRE
Volans closed T160178: MediaWiki Datacenter Switchover automation, a subtask of T154658: Prepare and improve the datacenter switchover procedure, as Resolved.
May 3 2017, 5:20 PM · Sustainability (MediaWiki-MultiDC), DC-Switchover-Prep-Q3-2016-17, Epic, SRE
Volans closed T160178: MediaWiki Datacenter Switchover automation, a subtask of T156100: DNS: dynamically generate entries for service discovery, as Resolved.
May 3 2017, 5:20 PM · Sustainability (MediaWiki-MultiDC), Patch-For-Review, Services (watching), Performance-Team, discovery-system, User-Joe, User-mobrovac, MediaWiki-Configuration, SRE, Wikimedia-Developer-Summit (2017)
Volans closed T160178: MediaWiki Datacenter Switchover automation, a subtask of T156924: Allow integration of data from etcd into the MediaWiki configuration, as Resolved.
May 3 2017, 5:20 PM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
jcrespo closed T155099: Database maintenance scheduled while eqiad datacenter is non primary (after the DC switchover), a subtask of T154658: Prepare and improve the datacenter switchover procedure, as Resolved.
May 3 2017, 4:43 PM · Sustainability (MediaWiki-MultiDC), DC-Switchover-Prep-Q3-2016-17, Epic, SRE
gerritbot added a comment to T160178: MediaWiki Datacenter Switchover automation.

Change 351313 merged by Giuseppe Lavagetto:
[operations/puppet@production] cache::text: switch all mediawiki to eqiad

May 3 2017, 2:09 PM · Sustainability (MediaWiki-MultiDC), Patch-For-Review, DC-Switchover-Prep-Q3-2016-17, Epic, SRE
gerritbot added a comment to T160178: MediaWiki Datacenter Switchover automation.

Change 351315 merged by Giuseppe Lavagetto:
[operations/puppet@production] discovery::app_routes: switch mediawiki to eqiad

May 3 2017, 2:09 PM · Sustainability (MediaWiki-MultiDC), Patch-For-Review, DC-Switchover-Prep-Q3-2016-17, Epic, SRE

May 2 2017

gerritbot added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Change 351539 merged by jenkins-bot:
[operations/mediawiki-config@master] wmf-config: readonly is set in etcd now

May 2 2017, 11:15 PM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
gerritbot added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Change 351132 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable EtcdConfig in production

May 2 2017, 11:03 PM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
gerritbot added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Change 351539 had a related patch set uploaded (by Volans; owner: Volans):
[operations/mediawiki-config@master] wmf-config: readonly is set in etcd now

May 2 2017, 10:54 PM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
gerritbot added a comment to T160178: MediaWiki Datacenter Switchover automation.

Change 351346 merged by Volans:
[operations/switchdc@master] t05_switch_datacenter: fix typo in DNS checks

May 2 2017, 5:30 PM · Sustainability (MediaWiki-MultiDC), Patch-For-Review, DC-Switchover-Prep-Q3-2016-17, Epic, SRE
gerritbot added a comment to T160178: MediaWiki Datacenter Switchover automation.

Change 351346 had a related patch set uploaded (by Volans; owner: Volans):
[operations/switchdc@master] t05_switch_datacenter: fix typo in DNS checks

May 2 2017, 5:28 PM · Sustainability (MediaWiki-MultiDC), Patch-For-Review, DC-Switchover-Prep-Q3-2016-17, Epic, SRE
gerritbot added a comment to T160178: MediaWiki Datacenter Switchover automation.

Change 351335 merged by Volans:
[operations/switchdc@master] t04: fix title

May 2 2017, 4:40 PM · Sustainability (MediaWiki-MultiDC), Patch-For-Review, DC-Switchover-Prep-Q3-2016-17, Epic, SRE
gerritbot added a comment to T160178: MediaWiki Datacenter Switchover automation.

Change 351335 had a related patch set uploaded (by Volans; owner: Volans):
[operations/switchdc@master] t04: fix title

May 2 2017, 4:35 PM · Sustainability (MediaWiki-MultiDC), Patch-For-Review, DC-Switchover-Prep-Q3-2016-17, Epic, SRE
gerritbot added a comment to T160178: MediaWiki Datacenter Switchover automation.

Change 351315 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] discovery::app_routes: switch mediawiki to eqiad

May 2 2017, 2:17 PM · Sustainability (MediaWiki-MultiDC), Patch-For-Review, DC-Switchover-Prep-Q3-2016-17, Epic, SRE
gerritbot added a comment to T160178: MediaWiki Datacenter Switchover automation.

Change 351313 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] cache::text: switch all mediawiki to eqiad

May 2 2017, 1:52 PM · Sustainability (MediaWiki-MultiDC), Patch-For-Review, DC-Switchover-Prep-Q3-2016-17, Epic, SRE
Volans added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

thanks @tstarling!

May 2 2017, 11:45 AM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
tstarling added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Is there an easy way I could check which version and/or value of an Etcd-driven MW-config variable is actually loaded/cached by the running application?

I'll add a siteinfo hook or something.

May 2 2017, 2:41 AM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
gerritbot added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Change 351232 merged by jenkins-bot:
[operations/mediawiki-config@master] Add $wmfMasterDatacenter to meta=siteinfo

May 2 2017, 2:29 AM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
ReleaseTaggerBot edited projects for T156924: Allow integration of data from etcd into the MediaWiki configuration, added: MW-1.29-release (WMF-deploy-2017-04-25_(1.29.0-wmf.21)); removed MW-1.29-release (WMF-deploy-2017-04-11_(1.29.0-wmf.20)).
May 2 2017, 2:00 AM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
gerritbot added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Change 351217 merged by jenkins-bot:
[mediawiki/core@wmf/1.29.0-wmf.21] [1.29.0-wmf.21] config: Use less generic cache key, and not fragmented by wiki

May 2 2017, 1:04 AM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
gerritbot added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Change 351232 had a related patch set uploaded (by Tim Starling; owner: Tim Starling):
[operations/mediawiki-config@master] Add $wmfMasterDatacenter to meta=siteinfo

May 2 2017, 12:48 AM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
ReleaseTaggerBot added a project to T156924: Allow integration of data from etcd into the MediaWiki configuration: MW-1.30-release-notes (WMF-deploy-2017-05-09_(1.30.0-wmf.1)).
May 2 2017, 12:00 AM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
ReleaseTaggerBot added a project to T92357: Fix database master queries from HTTP GET/HEAD before active-active multi-dc: MW-1.30-release-notes (WMF-deploy-2017-05-09_(1.30.0-wmf.1)).
May 2 2017, 12:00 AM · Wikimedia-Performance-recommendation, Sustainability (MediaWiki-MultiDC), MediaWiki-General

May 1 2017

gerritbot added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Change 351204 merged by jenkins-bot:
[mediawiki/core@master] config: Use less generic cache key, and not fragmented by wiki

May 1 2017, 11:31 PM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
gerritbot added a comment to T92357: Fix database master queries from HTTP GET/HEAD before active-active multi-dc.

Change 350969 merged by jenkins-bot:
[mediawiki/core@master] Avoid master queries in loadAndLazyInit() for miser mode

May 1 2017, 11:24 PM · Wikimedia-Performance-recommendation, Sustainability (MediaWiki-MultiDC), MediaWiki-General
gerritbot added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Change 351217 had a related patch set uploaded (by Tim Starling; owner: Krinkle):
[mediawiki/core@wmf/1.29.0-wmf.21] [1.29.0-wmf.21] config: Use less generic cache key, and not fragmented by wiki

May 1 2017, 11:21 PM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
gerritbot added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Change 351204 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] config: Use less generic cache key, and not fragmented by wiki

May 1 2017, 10:00 PM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
Krinkle added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Did the following testing: [..]

  • Confirmed that the APC cache was operational, by sending 100 requests to Special:BlankPage with ab while watching for inbound connections on the etcd server using tcpdump. There were 4 connections in 12 seconds. Connections could be seen coming in pairs with ~10s spacing between pairs.
May 1 2017, 9:49 PM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
tstarling added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Is there an easy way I could check which version and/or value of an Etcd-driven MW-config variable is actually loaded/cached by the running application?

May 1 2017, 9:35 PM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
Mattflaschen-WMF closed T163514: Labs undefined index due to data center switchover, a subtask of T154658: Prepare and improve the datacenter switchover procedure, as Resolved.
May 1 2017, 9:28 PM · Sustainability (MediaWiki-MultiDC), DC-Switchover-Prep-Q3-2016-17, Epic, SRE
Volans added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Is there an easy way I could check which version and/or value of an Etcd-driven MW-config variable is actually loaded/cached by the running application?

May 1 2017, 10:06 AM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
gerritbot added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Change 351132 had a related patch set uploaded (by Tim Starling; owner: Tim Starling):
[operations/mediawiki-config@master] Enable EtcdConfig in production

May 1 2017, 4:29 AM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
tstarling added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Did the following testing:

May 1 2017, 4:22 AM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
gerritbot added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Change 347537 merged by jenkins-bot:
[operations/mediawiki-config@master] Use EtcdConfig in beta cluster only

May 1 2017, 3:39 AM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE

Apr 29 2017

gerritbot added a comment to T92357: Fix database master queries from HTTP GET/HEAD before active-active multi-dc.

Change 350969 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Avoid master queries in loadAndLazyInit() for miser mode

Apr 29 2017, 5:11 AM · Wikimedia-Performance-recommendation, Sustainability (MediaWiki-MultiDC), MediaWiki-General

Apr 28 2017

Krinkle added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.
  1. Should MW work seamlessly if there is a temporary outage on the etcd service?
    • I think it should and we should use the stale cached data if that happens, assuming that all cached configuration is still valid.
Apr 28 2017, 8:00 PM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE
gerritbot added a comment to T160178: MediaWiki Datacenter Switchover automation.

Change 350818 merged by Volans:
[operations/switchdc@master] Mediawiki: update role name for maintenance

Apr 28 2017, 11:02 AM · Sustainability (MediaWiki-MultiDC), Patch-For-Review, DC-Switchover-Prep-Q3-2016-17, Epic, SRE
gerritbot added a project to T160178: MediaWiki Datacenter Switchover automation: Patch-For-Review.
Apr 28 2017, 10:59 AM · Sustainability (MediaWiki-MultiDC), Patch-For-Review, DC-Switchover-Prep-Q3-2016-17, Epic, SRE
gerritbot added a comment to T160178: MediaWiki Datacenter Switchover automation.

Change 350818 had a related patch set uploaded (by Volans; owner: Volans):
[operations/switchdc@master] Mediawiki: update role name for maintenance

Apr 28 2017, 10:58 AM · Sustainability (MediaWiki-MultiDC), Patch-For-Review, DC-Switchover-Prep-Q3-2016-17, Epic, SRE
Volans added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Regarding the implementation of the MW configuration, in particular CR https://gerrit.wikimedia.org/r/#/c/347537 (current patchset is #8), I think that we should first agree on the failure model, because I've seen different comments and approaches.

Apr 28 2017, 9:59 AM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE

Apr 25 2017

gerritbot added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

Change 349468 merged by Alexandros Kosiaris:
[operations/puppet@production] profile::conftool::master: make the git root dir a parameter

Apr 25 2017, 1:08 PM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE

Apr 23 2017

mmodell added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

This is awesome! Thanks @Joe for getting this working on beta.

Apr 23 2017, 1:13 AM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE

Apr 21 2017

Joe added a comment to T156924: Allow integration of data from etcd into the MediaWiki configuration.

To summarize, I think it is possible to test EtcdConfig in beta at this point with the limited deployment I made.

Apr 21 2017, 6:03 PM · Patch-For-Review, MediaWiki-Platform-Team-Archived (MWPT-Q4-Apr-Jun-2018), MediaWiki-Configuration, Performance-Team (Radar), Sustainability (MediaWiki-MultiDC), Services (watching), User-Joe, User-mobrovac, SRE