Page MenuHomePhabricator
Create Task
Maniphest T409085

⬆️🚦Remove direct references to a MediaWiki Backend service from Platform API and replace with logic that appropriately routes it
Open, Needs TriagePublic5 Estimated Story Points
Actions

Description

Following the POC produced in T407504 it was decided to allow the Platform API to directly select the appropriate backend MW service to use.

The following jobs need to be migrated:

The following command needs to be migrated:

  • RebuildQueryserviceData

They should utilise the POC helper.

Finally we should clean up the PLATFORM_MW_BACKEND_HOST environment variable its self.

For now the db version to backend host definition should remain (as per POC) in the api code. We decided only in the future might we want to factor this out into a configmap.

A/C:

  • Remove above references to PLATFORM_MW_BACKEND_HOST
  • Remove PLATFORM_MW_BACKEND_HOST from configuration, charts and deployment variables

Related Objects
Search...

Event Timeline

Tarrow created this task.Nov 3 2025, 4:03 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 3 2025, 4:03 PM
Tarrow renamed this task from ⬆️ Remove direct references to a Mediawiki Backend service from Platform API and replace with logic that appropriated routes it to ⬆️ Remove direct references to a Mediawiki Backend service from Platform API and replace with logic that appropriately routes it.Nov 3 2025, 4:03 PM
Tarrow set the point value for this task to 5.
Tarrow updated the task description. (Show Details)Nov 3 2025, 4:06 PM
Tarrow added a parent task: T385981: ⬆️ Create suitable MediaWiki 1.43 image.
Tarrow mentioned this in T409087: ⬆️🚦 Investigate for queryservice-updater backend mediawiki service routing.Nov 3 2025, 4:13 PM
Tarrow mentioned this in T407504: ⬆️ [Spike] Routing MW backend traffic based on MediaWiki version.
dena mentioned this in T409078: ⬆️🚦Deduplicate mapping of mediawiki backend versions.Nov 3 2025, 4:21 PM
dena claimed this task.Nov 4 2025, 10:49 AM
dena moved this task from To do to Doing on the Wikibase Cloud (Kanban Board) board.
dena added a comment.EditedNov 4 2025, 2:31 PM

Question about this one:

The following command needs to be migrated:

  • RebuildQueryserviceData

This command doesn't seem to use this env var in any way https://github.com/wbstack/api/blob/main/app/Console/Commands/RebuildQueryserviceData.php#L32
It queues instances of SpawnQueryserviceUpdaterJob, and those seem to not use the mediawiki backend? https://github.com/wbstack/api/blob/main/app/Jobs/SpawnQueryserviceUpdaterJob.php#L86

edit: found the issue, that class (and others) use this trait https://github.com/wbstack/api/blob/main/app/Traits/PageFetcher.php

dena added a comment.Nov 4 2025, 2:41 PM

I pushed first commits already to this draft PR https://github.com/wbstack/api/pull/988

WMDEGHPhabBot added a project: Patch-For-Review.Nov 4 2025, 5:25 PM

tarrow opened https://github.com/wbstack/api/pull/989

Introduce MediaWikiHostResolver

WMDEGHPhabBot added a comment.Nov 5 2025, 2:23 PM

tarrow closed https://github.com/wbstack/api/pull/989

Introduce MediaWikiHostResolver

dena renamed this task from ⬆️ Remove direct references to a Mediawiki Backend service from Platform API and replace with logic that appropriately routes it to ⬆️ 🚦Remove direct references to a Mediawiki Backend service from Platform API and replace with logic that appropriately routes it.Nov 7 2025, 8:58 AM
dena moved this task from Doing to In Review on the Wikibase Cloud (Kanban Board) board.Nov 7 2025, 4:05 PM
dena added a comment.EditedNov 7 2025, 5:00 PM

PR is ready for review - its a lot since it contains all Jobs https://github.com/wbstack/api/pull/988

I wonder if we want to see this running correctly before we do AC remove PLATFORM_MW_BACKEND_HOST from configuration, charts and deployment variables

There's also this PR which adds some error handling to the host resolver https://github.com/wbstack/api/pull/994

Reedy renamed this task from ⬆️ 🚦Remove direct references to a Mediawiki Backend service from Platform API and replace with logic that appropriately routes it to ⬆️🚦Remove direct references to a MediaSiki Backend service from Platform API and replace with logic that appropriately routes it.Nov 7 2025, 5:02 PM
Reedy updated the task description. (Show Details)
Reedy updated the task description. (Show Details)
Reedy updated the task description. (Show Details)
dena renamed this task from ⬆️🚦Remove direct references to a MediaSiki Backend service from Platform API and replace with logic that appropriately routes it to ⬆️🚦Remove direct references to a MediaWiki Backend service from Platform API and replace with logic that appropriately routes it.Nov 7 2025, 5:05 PM
Tarrow edited parent tasks, added: T409730: ⬆️🚦Make sure MediaWiki 1.43 DBs are only accessed by 1.43 image based pods and similar for 1.39; removed: T385981: ⬆️ Create suitable MediaWiki 1.43 image.Nov 10 2025, 1:45 PM
Ollie.Shotton_WMDE updated the task description. (Show Details)Nov 11 2025, 12:29 PM
dena added a comment.Nov 11 2025, 6:37 PM

Fixed some tests and repaired a git-oopsie - this is now ready to be reviewed more

Tarrow reassigned this task from dena to Ollie.Shotton_WMDE.Nov 12 2025, 2:50 PM
Tarrow added subscribers: Ollie.Shotton_WMDE, dang, dena.

Vague next steps:

Ollie.Shotton_WMDE added a comment.Nov 13 2025, 3:59 PM

Tested this locally:

  • ✅ UpdateWikiSiteStatsJob
    • Created mediawiki-139 and mediawiki-143 wikis and added some Items to existing mediawiki-139 and mediawiki-143 wikis
    • Ran the job: kubectl exec -ti deployments/api-app-backend -- php artisan job:dispatchNow UpdateWikiSiteStatsJob
    • No errors in the logs: kubectl logs deployments/api-app-backend
    • Saw that the wiki_site_stats table was updated
  • ✅ MediawikiInit
    • This job is run when a platform user creates a new wiki. I was able to create mediawiki-139 and mediawiki-143 wikis.
  • ✅ SiteStatsUpdateJob
    • Job ran successfully
$ kubectl exec -ti deployments/api-app-backend -- php artisan job:dispatchNow SiteStatsUpdateJob 1
[2025-11-13 15:33:01] local.INFO: App\Jobs\SiteStatsUpdateJob::handle: Updating stats for asdf1.wbaas.dev  
[2025-11-13 15:33:02] local.INFO: App\Jobs\SiteStatsUpdateJob::handle: Finished in: 1.3104979991913 s
    • create a new item and saw the mwt_14b5587ec1_site_stats table was updated (didn't have to dispatch the job)
  • ✅ RebuildQueryserviceData
    • Job ran successfully:
$ kubectl exec -ti deployments/api-app-backend -- php artisan wbs-qs:rebuild
[2025-11-13 15:48:21] local.INFO: Dispatched 1 job(s) for wiki asdf1.wbaas.dev.  
[2025-11-13 15:48:22] local.INFO: Dispatched 0 job(s) for wiki asdf2.wbaas.dev.  
[2025-11-13 15:48:22] local.INFO: Dispatched 1 job(s) for wiki asdf3.wbaas.dev.  
[2025-11-13 15:48:22] local.INFO: Dispatched 0 job(s) for wiki asdf4.wbaas.dev.  
[2025-11-13 15:48:22] local.INFO: Dispatched 1 job(s) for wiki asdf5.wbaas.dev.  
[2025-11-13 15:48:22] local.INFO: Dispatched 1 job(s) for wiki asdf6.wbaas.dev.  
[2025-11-13 15:48:22] local.INFO: Dispatched 0 job(s) for wiki asdf7.wbaas.dev.  
[2025-11-13 15:48:22] local.INFO: Done. Jobs dispatched: 4 Wikis processed: 7 Wikis skipped: 0 
[previous exception] [object] (RuntimeException(code: 0): wikibase-exampledata/v0/load call for asdf6.wbaas.dev was not successful:{\"message\":\"Error: exception of type Error\",\"httpCode\":500,\"httpReason\":\"Internal Server Error\"} at /var/www/html/app/Jobs/MediawikiSandboxLoadData.php:61)
[stacktrace]
#0 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): App\\Jobs\\MediawikiSandboxLoadData->handle(Object(App\\Services\\MediaWikiHostResolver))
#1 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#2 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))
#3 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#4 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#5 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\\Container\\Container->call(Array)
#6 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\\Bus\\Dispatcher->Illuminate\\Bus\\{closure}(Object(App\\Jobs\\MediawikiSandboxLoadData))
#7 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(App\\Jobs\\MediawikiSandboxLoadData))
#8 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#9 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(123): Illuminate\\Bus\\Dispatcher->dispatchNow(Object(App\\Jobs\\MediawikiSandboxLoadData), false)
#10 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\\Queue\\CallQueuedHandler->Illuminate\\Queue\\{closure}(Object(App\\Jobs\\MediawikiSandboxLoadData))
#11 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(App\\Jobs\\MediawikiSandboxLoadData))
#12 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(122): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#13 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(70): Illuminate\\Queue\\CallQueuedHandler->dispatchThroughMiddleware(Object(Illuminate\\Queue\\Jobs\\SyncJob), Object(App\\Jobs\\MediawikiSandboxLoadData))
#14 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(102): Illuminate\\Queue\\CallQueuedHandler->call(Object(Illuminate\\Queue\\Jobs\\SyncJob), Array)
#15 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/SyncQueue.php(43): Illuminate\\Queue\\Jobs\\Job->fire()
#16 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(254): Illuminate\\Queue\\SyncQueue->push(Object(App\\Jobs\\MediawikiSandboxLoadData))
#17 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(230): Illuminate\\Bus\\Dispatcher->pushCommandToQueue(Object(Illuminate\\Queue\\SyncQueue), Object(App\\Jobs\\MediawikiSandboxLoadData))
#18 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(95): Illuminate\\Bus\\Dispatcher->dispatchToQueue(Object(App\\Jobs\\MediawikiSandboxLoadData))
#19 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/helpers.php(404): Illuminate\\Bus\\Dispatcher->dispatchSync(Object(App\\Jobs\\MediawikiSandboxLoadData), NULL)
#20 /var/www/html/vendor/mxl/laravel-job/src/Commands/DispatchNow.php(18): dispatch_sync(Object(App\\Jobs\\MediawikiSandboxLoadData))
#21 /var/www/html/vendor/mxl/laravel-job/src/Commands/Dispatch.php(32): MichaelLedin\\LaravelJob\\Commands\\DispatchNow->dispatch(Object(App\\Jobs\\MediawikiSandboxLoadData))
#22 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): MichaelLedin\\LaravelJob\\Commands\\Dispatch->handle()
#23 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#24 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))
#25 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#26 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#27 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(211): Illuminate\\Container\\Container->call(Array)
#28 /var/www/html/vendor/symfony/console/Command/Command.php(326): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#29 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(180): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#30 /var/www/html/vendor/symfony/console/Application.php(1096): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#31 /var/www/html/vendor/symfony/console/Application.php(324): Symfony\\Component\\Console\\Application->doRunCommand(Object(MichaelLedin\\LaravelJob\\Commands\\DispatchNow), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#32 /var/www/html/vendor/symfony/console/Application.php(175): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#33 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(201): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#34 /var/www/html/artisan(35): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#35 {main}
"}
  • ❌ PlatformStatsSummaryJob
    • Code looks correct, failed to test locally but also doesn't work without this change
[2025-11-13 15:03:37] local.ERROR: SQLSTATE[HY000] [1045] Access denied for user ''@'10.244.1.96' (using password: NO) {"exception":"[object] (PDOException(code: 1045): SQLSTATE[HY000] [1045] Access denied for user ''@'10.244.1.96' (using password: NO) at /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php:65)
[stacktrace]
#0 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php(65): PDO->__construct('mysql:host=sql-...', NULL, Object(SensitiveParameterValue), Array)
#1 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php(44): Illuminate\\Database\\Connectors\\Connector->createPdoConnection('mysql:host=sql-...', NULL, '', Array)
#2 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connectors/MySqlConnector.php(24): Illuminate\\Database\\Connectors\\Connector->createConnection('mysql:host=sql-...', Array, Array)
#3 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connectors/ConnectionFactory.php(184): Illuminate\\Database\\Connectors\\MySqlConnector->connect(Array)
#4 [internal function]: Illuminate\\Database\\Connectors\\ConnectionFactory->Illuminate\\Database\\Connectors\\{closure}()
#5 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(1339): call_user_func(Object(Closure))
#6 /var/www/html/app/Jobs/PlatformStatsSummaryJob.php(182): Illuminate\\Database\\Connection->getPdo()
#7 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): App\\Jobs\\PlatformStatsSummaryJob->handle(Object(Illuminate\\Database\\DatabaseManager), Object(App\\Services\\MediaWikiHostResolver))
#8 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#9 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))
#10 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#11 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#12 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\\Container\\Container->call(Array)
#13 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\\Bus\\Dispatcher->Illuminate\\Bus\\{closure}(Object(App\\Jobs\\PlatformStatsSummaryJob))
#14 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(App\\Jobs\\PlatformStatsSummaryJob))
#15 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#16 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(123): Illuminate\\Bus\\Dispatcher->dispatchNow(Object(App\\Jobs\\PlatformStatsSummaryJob), false)
#17 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\\Queue\\CallQueuedHandler->Illuminate\\Queue\\{closure}(Object(App\\Jobs\\PlatformStatsSummaryJob))
#18 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(App\\Jobs\\PlatformStatsSummaryJob))
#19 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(122): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#20 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(70): Illuminate\\Queue\\CallQueuedHandler->dispatchThroughMiddleware(Object(Illuminate\\Queue\\Jobs\\SyncJob), Object(App\\Jobs\\PlatformStatsSummaryJob))
#21 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(102): Illuminate\\Queue\\CallQueuedHandler->call(Object(Illuminate\\Queue\\Jobs\\SyncJob), Array)
#22 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/SyncQueue.php(43): Illuminate\\Queue\\Jobs\\Job->fire()
#23 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(254): Illuminate\\Queue\\SyncQueue->push(Object(App\\Jobs\\PlatformStatsSummaryJob))
#24 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(230): Illuminate\\Bus\\Dispatcher->pushCommandToQueue(Object(Illuminate\\Queue\\SyncQueue), Object(App\\Jobs\\PlatformStatsSummaryJob))
#25 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(95): Illuminate\\Bus\\Dispatcher->dispatchToQueue(Object(App\\Jobs\\PlatformStatsSummaryJob))
#26 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/helpers.php(404): Illuminate\\Bus\\Dispatcher->dispatchSync(Object(App\\Jobs\\PlatformStatsSummaryJob), NULL)
#27 /var/www/html/vendor/mxl/laravel-job/src/Commands/DispatchNow.php(18): dispatch_sync(Object(App\\Jobs\\PlatformStatsSummaryJob))
#28 /var/www/html/vendor/mxl/laravel-job/src/Commands/Dispatch.php(32): MichaelLedin\\LaravelJob\\Commands\\DispatchNow->dispatch(Object(App\\Jobs\\PlatformStatsSummaryJob))
#29 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): MichaelLedin\\LaravelJob\\Commands\\Dispatch->handle()
#30 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#31 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))
#32 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#33 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#34 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(211): Illuminate\\Container\\Container->call(Array)
#35 /var/www/html/vendor/symfony/console/Command/Command.php(326): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#36 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(180): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#37 /var/www/html/vendor/symfony/console/Application.php(1096): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#38 /var/www/html/vendor/symfony/console/Application.php(324): Symfony\\Component\\Console\\Application->doRunCommand(Object(MichaelLedin\\LaravelJob\\Commands\\DispatchNow), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#39 /var/www/html/vendor/symfony/console/Application.php(175): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#40 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(201): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#41 /var/www/html/artisan(35): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#42 {main}
"}
  • ❌ CirrusSearchJob
    • Code looks correct, failed to test locally but also doesn't work without this change
$ kubectl exec -ti deployments/api-app-backend -- php artisan job:dispatchNow CirrusSearch\\QueueSearchIndexBatches 1
...
[previous exception] [object] (RuntimeException(code: 0): wbstackQueueSearchIndexBatches call for 3 was not successful:{\"wbstackQueueSearchIndexBatches\":{\"script\":\"extensions/CirrusSearch/maintenance/ForceSearchIndex.php\",\"return\":1,\"output\":[\"Elastica\\\\Exception\\\\Connection\\\\HttpException from line 186 of /var/www/html/w/vendor/ruflin/elastica/src/Transport/Http.php: Operation timed out\",\"#0 /var/www/html/w/vendor/ruflin/elastica/src/Request.php(178): Elastica\\\\Transport\\\\Http->exec(Object(Elastica\\\\Request), Array)\",\"#1 /var/www/html/w/vendor/ruflin/elastica/src/Client.php(513): Elastica\\\\Request->send()\",\"#2 /var/www/html/w/vendor/ruflin/elastica/src/Client.php(548): Elastica\\\\Client->request('mwdb_7e7ebc3036', 'HEAD', Array, Array)\",\"#3 /var/www/html/w/vendor/ruflin/elastica/src/Index.php(666): Elastica\\\\Client->requestEndpoint(Object(Elasticsearch\\\\Endpoints\\\\Indices\\\\Exists))\",\"#4 /var/www/html/w/vendor/ruflin/elastica/src/Index.php(452): Elastica\\\\Index->requestEndpoint(Object(Elasticsearch\\\\Endpoints\\\\Indices\\\\Exists))\",\"#5 /var/www/html/w/extensions/CirrusSearch/maintenance/ForceSearchIndex.php(379): Elastica\\\\Index->exists()\",\"#6 /var/www/html/w/extensions/CirrusSearch/maintenance/ForceSearchIndex.php(142): CirrusSearch\\\\Maintenance\\\\ForceSearchIndex->simpleCheckIndexes()\",\"#7 /var/www/html/w/maintenance/includes/MaintenanceRunner.php(309): CirrusSearch\\\\Maintenance\\\\ForceSearchIndex->execute()\",\"#8 /var/www/html/w/maintenance/doMaintenance.php(89): MediaWiki\\\\Maintenance\\\\MaintenanceRunner->run()\",\"#9 /var/www/html/w/extensions/CirrusSearch/maintenance/ForceSearchIndex.php(684): require_once('/var/www/html/w...')\",\"#10 {main}\"]}} at /var/www/html/app/Jobs/CirrusSearch/CirrusSearchJob.php:117)
[stacktrace]
#0 /var/www/html/app/Jobs/CirrusSearch/QueueSearchIndexBatches.php(68): App\\Jobs\\CirrusSearch\\CirrusSearchJob->validateSuccess(Array, '{\"wbstackQueueS...', '')
#1 /var/www/html/app/Jobs/CirrusSearch/CirrusSearchJob.php(90): App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches->handleResponse('{\"wbstackQueueS...', '')
#2 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): App\\Jobs\\CirrusSearch\\CirrusSearchJob->handle(Object(App\\Http\\Curl\\CurlRequest), Object(App\\Services\\MediaWikiHostResolver))
#3 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#4 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))
#5 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#6 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#7 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\\Container\\Container->call(Array)
#8 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\\Bus\\Dispatcher->Illuminate\\Bus\\{closure}(Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches))
#9 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches))
#10 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#11 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(123): Illuminate\\Bus\\Dispatcher->dispatchNow(Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches), false)
#12 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\\Queue\\CallQueuedHandler->Illuminate\\Queue\\{closure}(Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches))
#13 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches))
#14 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(122): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#15 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(70): Illuminate\\Queue\\CallQueuedHandler->dispatchThroughMiddleware(Object(Illuminate\\Queue\\Jobs\\SyncJob), Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches))
#16 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(102): Illuminate\\Queue\\CallQueuedHandler->call(Object(Illuminate\\Queue\\Jobs\\SyncJob), Array)
#17 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/SyncQueue.php(43): Illuminate\\Queue\\Jobs\\Job->fire()
#18 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(254): Illuminate\\Queue\\SyncQueue->push(Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches))
#19 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(230): Illuminate\\Bus\\Dispatcher->pushCommandToQueue(Object(Illuminate\\Queue\\SyncQueue), Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches))
#20 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(95): Illuminate\\Bus\\Dispatcher->dispatchToQueue(Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches))
#21 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/helpers.php(404): Illuminate\\Bus\\Dispatcher->dispatchSync(Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches), NULL)
#22 /var/www/html/vendor/mxl/laravel-job/src/Commands/DispatchNow.php(18): dispatch_sync(Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches))
#23 /var/www/html/vendor/mxl/laravel-job/src/Commands/Dispatch.php(32): MichaelLedin\\LaravelJob\\Commands\\DispatchNow->dispatch(Object(App\\Jobs\\CirrusSearch\\QueueSearchIndexBatches))
#24 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): MichaelLedin\\LaravelJob\\Commands\\Dispatch->handle()
#25 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#26 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))
#27 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#28 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#29 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(211): Illuminate\\Container\\Container->call(Array)
#30 /var/www/html/vendor/symfony/console/Command/Command.php(326): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#31 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(180): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#32 /var/www/html/vendor/symfony/console/Application.php(1096): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#33 /var/www/html/vendor/symfony/console/Application.php(324): Symfony\\Component\\Console\\Application->doRunCommand(Object(MichaelLedin\\LaravelJob\\Commands\\DispatchNow), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#34 /var/www/html/vendor/symfony/console/Application.php(175): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#35 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(201): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#36 /var/www/html/artisan(35): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#37 {main}
"}
Tarrow added a comment.Nov 13 2025, 4:15 PM

For the failing jobs can you retry them on a queue pod? i.e. kubectl exec -ti deployments/api-queue-default ... .

I'd expect some of these jobs to maybe fail on a "backend" pod. For example I could imagine that this PlatformStatsSummaryJob is missing sufficient credentials to read all the wikidbs from a backend pod.

Ollie.Shotton_WMDE added a comment.Nov 13 2025, 4:27 PM

Good thinking.

✅ PlatformStatsSummaryJob works on the queue pod:

$ kubectl exec -ti deployments/api-queue-default -- php artisan job:dispatchNow PlatformStatsSummaryJob
{"platform_summary_version":"v1","total":7,"deleted":0,"edited_last_90_days":7,"not_edited_last_90_days":0,"empty":0,"total_non_deleted_users":20,"total_non_deleted_active_users":4,"total_non_deleted_pages":12,"total_non_deleted_edits":12,"total_items_count":5,"total_properties_count":0,"wikis_created_PT24H":3,"users_created_PT24H":0,"wikis_created_P30D":7,"users_created_P30D":1}

❌ MediawikiSandboxLoadData and CirrusSearchJob still failed as above.

Tarrow reassigned this task from Ollie.Shotton_WMDE to dena.Nov 14 2025, 10:06 AM
dena moved this task from In Review to PM/UX Verification on the Wikibase Cloud (Kanban Board) board.Nov 14 2025, 11:48 AM
dena added a comment.EditedNov 14 2025, 12:08 PM
Tests on Staging

test wikis:

  • MW 1.43 - test-143-deer.wikibase.dev - wikiID 175
  • MW 1.39 - deerbase.wikibase.dev - wikiID 2

general method: dispatching a Job while looking at logs of mediawiki-backend pods via stern, confirming the requests end up at the right hosts

✅ UpdateWikiSiteStatsJob

kubectl exec -ti deployments/api-scheduler -- php artisan job:dispatch UpdateWikiSiteStatsJob

Both wikis got queried by the Job.

✅ SiteStatsUpdateJob

kubectl exec -ti deployments/api-scheduler -- php artisan job:dispatch SiteStatsUpdateJob 2
kubectl exec -ti deployments/api-scheduler -- php artisan job:dispatch SiteStatsUpdateJob 175

✅ MediawikiInit
I created a new wiki, could observe logs at the right container, and platform UI and the wiki itself reports it runs 1.43: https://test-creation-143.wikibase.dev/wiki/Special:Version

✅ RebuildQueryserviceData
I ran kubectl exec -ti deployments/api-app-backend -- php artisan wbs-qs:rebuild, which unfortunately kicked off the jobs for all staging wikis with a chunk size of 1 and thus spawned thousands of jobs. Still, everything seems to have worked as expected.

kubectl exec -ti deployments/api-app-backend -- php artisan wbs-qs:rebuild --domain 'test-143-deer.wikibase.dev'
kubectl exec -ti deployments/api-app-backend -- php artisan wbs-qs:rebuild --domain 'deerbase.wikibase.dev'

✅ PlatformStatsSummaryJob

kubectl exec -ti deployments/api-app-backend -- php artisan schedule:stats

✅ PollForMediaWikiJobsJob

kubectl exec -ti deployments/api-scheduler -- php artisan job:dispatch PollForMediaWikiJobsJob
Tarrow mentioned this in T410136: RebuildQueryserviceData defaults to creating too many jobs.Nov 14 2025, 12:16 PM
dena moved this task from PM/UX Verification to Waiting for Deploy to Staging on the Wikibase Cloud (Kanban Board) board.Nov 14 2025, 12:29 PM
Tarrow moved this task from Waiting for Deploy to Staging to Waiting for Deploy to Production on the Wikibase Cloud (Kanban Board) board.Nov 14 2025, 12:29 PM
dena added a comment.Nov 14 2025, 2:26 PM

CirrusSearchJob is an abstract class which gets extended by:

kubectl exec -ti deployments/api-scheduler -- php artisan job:dispatch CirrusSearch\\ForceSearchIndex id 175 0 1000
kubectl exec -ti deployments/api-scheduler -- php artisan job:dispatch CirrusSearch\\ForceSearchIndex id 2 0 1000

  • ❓ CirrusSearch\\QueueSearchIndexBatches

For some reason this seems to fail currently for 1.43 wikis only, when dispatched like this (is this )

kubectl exec -ti deployments/api-scheduler -- php artisan job:dispatch CirrusSearch\\QueueSearchIndexBatches 175

Since the modification here for the routing happened in the base class and it worked for one of them, I think it's fair to assume it works for all of them. The Job could fail for other reasons, especially since it works for 1.39 wikis.

Tarrow added a comment.Nov 17 2025, 8:46 AM

CirrusSearch\\ElasticSearchIndexInit

I agree that this probably isn't run on each Wiki creation; I'm trying to figure out if this this is ever called? I thought it might be called once, somewhere, in order to create the initial ES index that we then target all the aliases at.

CirrusSearch\\QueueSearchIndexBatches

I looked at this a bit on staging and in tinker manually tried Wiki::whereId(175)->with('settings')->with('wikiDb')->first() which did give me a nicely populated Wiki object for test-143-deer.wikibase.dev. I wonder still if this is about how we are dispatching this job or if there's a bug in this code that we didn't discover yet after some refactoring.

Tarrow moved this task from Waiting for Deploy to Production to Done on the Wikibase Cloud (Kanban Board) board.Nov 17 2025, 9:49 AM
dena mentioned this in T410608: ⬆️🚦 Fix concurrency issue in WikiController.Thu, Nov 20, 11:15 AM
Ollie.Shotton_WMDE removed a project: Patch-For-Review.Fri, Dec 5, 11:23 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits