Configuration for sparsql in Wikibase
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Dann.wu
	Oct 24 2024, 8:12 AM

Description

Hello! This is a question from me that I hope it will be a task for an expert to help out.
We have installed a Wikibase instance. It seems sparsql works, ie. we have access to http://domain:8834 . It takes queries, but very simple query as below return NO items although we have thousands of item in the system. Inserted similar to github, https://github.com/the-qa-company/WikibaseSync

"SELECT ?item ?itemLabel
WHERE
{

?item wdt:P18 wd:Q146. # Must be a country
SERVICE wikibase:label bd:serviceParam wikibase:language "[AUTO_LANGUAGE]". }

Any suggestions

Many thanks!

Related Objects

Mentioned Here: T376016: WBS Deploy-3: WDQS does not accept items

Event Timeline

Dann.wu created this task.Oct 24 2024, 8:12 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 24 2024, 8:12 AM

To make SPARQL run properly you need to (1) if the triplestore is empty, dump every item in RDF and import it to triplestore; (2) run the update script to get recent changes from the wiki.

In T378065#10257922, @Bugreporter wrote:

To make SPARQL run properly you need to (1) if the triplestore is empty, dump every item in RDF and import it to triplestore; (2) run the update script to get recent changes from the wiki.

We installed Wikibase docker, I thought this is automatic handled by the system?!

Hi @Dann.wu,

Thanks a lot for reporting this. We recently released Wikibase Suite Deploy with a bug, where the triple store (WDQS) was not populated T376016. It is possible, that you hit that bug. Do you mind trying our newest version that released a fix for that?

You can get it by following our Quickstart Docs but checking out the git tag deploy@3.0.2 in the step "Download WBS Deploy".

Hope that helps,
Let me know if you need further support,
Best,
Robert

• roti_WMDE claimed this task.Nov 5 2024, 2:48 PM

• roti_WMDE added a project: Wikibase Suite Team (Sprint-∞).

• roti_WMDE moved this task from Sprint Backlog to In Review on the Wikibase Suite Team (Sprint-∞) board.

Hey @roti_WMDE
Thanks for your response the git tag deploy@3.0.2 has a problem with wikibase image it breaks on the run.
can you please refer us to the last working stable version?

Hi @Adammaen,

Thanks for reporting back.

We actually just released deploy@3.0.3, T376016: WBS Deploy-3: WDQS does not accept items was not fixed in deploy@3.0.2 due to an oversight. Sorry for the confusion. Please give 3.0.3 a try.

Could you please describe the problem you have with wikibase on deploy@3.0.2? I would be interested to understand what is happening there.

BTW: accessing services on 88XX ports is not supported anymore, please always use port 80/443 exposed by Traefik.

Hope that helps,
Best,
Robert

Hej @roti_WMDE

Thanks for your quick response,
and regarding not using the other ports do you recommend or have an nginx configuration for the reverse proxy with Traefik?
because since I don't have access to subdomain I just exposed the ports 8880, 8834 and 8840 before.

and here is the docker logs it just crashes

wikibase-1            | MediaWiki 1.42.3 Updater
wikibase-1            |
wikibase-1            | Your composer.lock file is up to date with current dependencies!
wikibase-1            | Going to run database updates for my_wiki
wikibase-1            | Depending on the size of your database this may take a while!
wikibase-1            | Wikimedia\Rdbms\DBQueryError from line 1203 of /var/www/html/includes/libs/rdbms/database/Database.php: Error 1146: Table 'my_wiki.updatelog' doesn't exist
wikibase-1            | Function: MediaWiki\Installer\DatabaseUpdater::updateRowExists
wikibase-1            | Query: SELECT  1 AS X  FROM updatelog    WHERE ul_key = 'UpdateCollation::uppercase'  LIMIT 1
wikibase-1            |
wikibase-1            | #0 /var/www/html/includes/libs/rdbms/database/Database.php(1187): Wikimedia\Rdbms\Database->getQueryException('Table 'my_wiki....', 1146, 'SELECT  1 AS X ...', 'MediaWiki\\Insta...')
wikibase-1            | #1 /var/www/html/includes/libs/rdbms/database/Database.php(1161): Wikimedia\Rdbms\Database->getQueryExceptionAndLog('Table 'my_wiki....', 1146, 'SELECT  1 AS X ...', 'MediaWiki\\Insta...')
wikibase-1            | #2 /var/www/html/includes/libs/rdbms/database/Database.php(652): Wikimedia\Rdbms\Database->reportQueryError('Table 'my_wiki....', 1146, 'SELECT  1 AS X ...', 'MediaWiki\\Insta...', false)
wikibase-1            | #3 /var/www/html/includes/libs/rdbms/database/Database.php(1350): Wikimedia\Rdbms\Database->query(Object(Wikimedia\Rdbms\Query), 'MediaWiki\\Insta...')
wikibase-1            | #4 /var/www/html/includes/libs/rdbms/database/Database.php(1359): Wikimedia\Rdbms\Database->select(Array, Array, Array, 'MediaWiki\\Insta...', Array, Array)
wikibase-1            | #5 /var/www/html/includes/libs/rdbms/database/DBConnRef.php(119): Wikimedia\Rdbms\Database->selectRow(Array, Array, Array, 'MediaWiki\\Insta...', Array, Array)
wikibase-1            | #6 /var/www/html/includes/libs/rdbms/database/DBConnRef.php(369): Wikimedia\Rdbms\DBConnRef->__call('selectRow', Array)
wikibase-1            | #7 /var/www/html/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(777): Wikimedia\Rdbms\DBConnRef->selectRow(Array, Array, Array, 'MediaWiki\\Insta...', Array, Array)
wikibase-1            | #8 /var/www/html/includes/installer/DatabaseUpdater.php(638): Wikimedia\Rdbms\SelectQueryBuilder->fetchRow()
wikibase-1            | #9 /var/www/html/includes/installer/DatabaseUpdater.php(1197): MediaWiki\Installer\DatabaseUpdater->updateRowExists('UpdateCollation...')
wikibase-1            | #10 /var/www/html/includes/installer/DatabaseUpdater.php(557): MediaWiki\Installer\DatabaseUpdater->doCollationUpdate()
wikibase-1            | #11 /var/www/html/maintenance/update.php(189): MediaWiki\Installer\DatabaseUpdater->doUpdates(Array)
wikibase-1            | #12 /var/www/html/maintenance/includes/MaintenanceRunner.php(698): UpdateMediaWiki->execute()
wikibase-1            | #13 /var/www/html/maintenance/doMaintenance.php(100): MediaWiki\Maintenance\MaintenanceRunner->run()
wikibase-1            | #14 /var/www/html/maintenance/update.php(302): require_once('/var/www/html/m...')
wikibase-1            | #15 {main}
wikibase-1 exited with code 1
wikibase-1            | MediaWiki 1.42.3 Updater
wikibase-1            |
wikibase-1            | Your composer.lock file is up to date with current dependencies!
wikibase-1            | Going to run database updates for my_wiki
wikibase-1            | Depending on the size of your database this may take a while!
wikibase-1            | Wikimedia\Rdbms\DBQueryError from line 1203 of /var/www/html/includes/libs/rdbms/database/Database.php: Error 1146: Table 'my_wiki.updatelog' doesn't exist
wikibase-1            | Function: MediaWiki\Installer\DatabaseUpdater::updateRowExists
wikibase-1            | Query: SELECT  1 AS X  FROM updatelog    WHERE ul_key = 'UpdateCollation::uppercase'  LIMIT 1
wikibase-1            |
wikibase-1            | #0 /var/www/html/includes/libs/rdbms/database/Database.php(1187): Wikimedia\Rdbms\Database->getQueryException('Table 'my_wiki....', 1146, 'SELECT  1 AS X ...', 'MediaWiki\\Insta...')
wikibase-1            | #1 /var/www/html/includes/libs/rdbms/database/Database.php(1161): Wikimedia\Rdbms\Database->getQueryExceptionAndLog('Table 'my_wiki....', 1146, 'SELECT  1 AS X ...', 'MediaWiki\\Insta...')
wikibase-1            | #2 /var/www/html/includes/libs/rdbms/database/Database.php(652): Wikimedia\Rdbms\Database->reportQueryError('Table 'my_wiki....', 1146, 'SELECT  1 AS X ...', 'MediaWiki\\Insta...', false)
wikibase-1            | #3 /var/www/html/includes/libs/rdbms/database/Database.php(1350): Wikimedia\Rdbms\Database->query(Object(Wikimedia\Rdbms\Query), 'MediaWiki\\Insta...')
wikibase-1            | #4 /var/www/html/includes/libs/rdbms/database/Database.php(1359): Wikimedia\Rdbms\Database->select(Array, Array, Array, 'MediaWiki\\Insta...', Array, Array)
wikibase-1            | #5 /var/www/html/includes/libs/rdbms/database/DBConnRef.php(119): Wikimedia\Rdbms\Database->selectRow(Array, Array, Array, 'MediaWiki\\Insta...', Array, Array)
wikibase-1            | #6 /var/www/html/includes/libs/rdbms/database/DBConnRef.php(369): Wikimedia\Rdbms\DBConnRef->__call('selectRow', Array)
wikibase-1            | #7 /var/www/html/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(777): Wikimedia\Rdbms\DBConnRef->selectRow(Array, Array, Array, 'MediaWiki\\Insta...', Array, Array)
wikibase-1            | #8 /var/www/html/includes/installer/DatabaseUpdater.php(638): Wikimedia\Rdbms\SelectQueryBuilder->fetchRow()
wikibase-1            | #9 /var/www/html/includes/installer/DatabaseUpdater.php(1197): MediaWiki\Installer\DatabaseUpdater->updateRowExists('UpdateCollation...')
wikibase-1            | #10 /var/www/html/includes/installer/DatabaseUpdater.php(557): MediaWiki\Installer\DatabaseUpdater->doCollationUpdate()
wikibase-1            | #11 /var/www/html/maintenance/update.php(189): MediaWiki\Installer\DatabaseUpdater->doUpdates(Array)
wikibase-1            | #12 /var/www/html/maintenance/includes/MaintenanceRunner.php(698): UpdateMediaWiki->execute()
wikibase-1            | #13 /var/www/html/maintenance/doMaintenance.php(100): MediaWiki\Maintenance\MaintenanceRunner->run()
wikibase-1            | #14 /var/www/html/maintenance/update.php(302): require_once('/var/www/html/m...')
wikibase-1            | #15 {main}
elasticsearch-1       | {"type": "server", "timestamp": "2024-11-07T09:47:33,640Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "1cb3e7200261", "message": "[controller/256] [Main.cc@114] controller (64 bit): Version 7.10.2 (Build 40a3af639d4698) Copyright (c) 2021 Elasticsearch BV" }
Gracefully stopping... (press Ctrl+C again to force)
dependency failed to start: container wbs-deploy-wikibase-1 is unhealthy

Thank you for the quick answer. My colleague tried 3.0.2, it crashed. Now he is trying 3.0.3. We will let you know the result.

I am wonder if you know an older version that works?

Thanks again for your answer!

Dann

From: roti_WMDE <no-reply@phabricator.wikimedia.org>
Sent: 07 November 2024 13:34:15
To: Phabricator
Cc: Dann Wu
Subject: [Maniphest] [Commented On] T378065: Configuration for sparsql in Wikibase

roti_WMDE added a comment. View Taskhttps://phabricator.wikimedia.org/T378065

Hi @Adammaen https://phabricator.wikimedia.org/p/Adammaen/,

Thanks for reporting back.

We actually just released deploy@3.0.3, T376016: WBS Deploy-3: WDQS does not accept items was not fixed in deploy@3.0.2 due to an oversight. Sorry for the confusion. Please give 3.0.3 a try.

Could you please describe the problem you have with wikibase on deploy@3.0.2? I would be interested to understand what is happening there.

BTW: accessing services on 88XX ports is not supported anymore, please always use port 80/443 exposed by Traefik.

Hope that helps,
Best,
Robert

TASK DETAIL
https://phabricator.wikimedia.org/T378065

EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: roti_WMDE
Cc: Adammaen, roti_WMDE, Bugreporter, Aklapper, Dann.wu, RickiJay-WMDE, jon_amar-WMDE, jdfraine, Olea, darthmon_wmde, KimKelting, Lydia_Pintscher

VARNING: Klicka inte på länkar och öppna inte bilagor om du inte känner igen avsändaren och vet att innehållet är säkert.
CAUTION: Do not click on links or open attachments unless you recognise the sender and know the content is safe.

När du har kontakt med oss på Uppsala universitet med e-post så innebär det att vi behandlar dina personuppgifter. För att läsa mer om hur vi gör det kan du läsa här: http://www.uu.se/om-uu/dataskydd-personuppgifter/

E-mailing Uppsala University means that we will process your personal data. For more information on how this is performed, please read here: http://www.uu.se/en/about-uu/data-protection-policy

Hi @Dann.wu,

Thanks a lot for you message.

The error you are reporting

wikibase-1            | Wikimedia\Rdbms\DBQueryError from line 1203 of /var/www/html/includes/libs/rdbms/database/Database.php: Error 1146: Table 'my_wiki.updatelog' doesn't exist

is probably related to setting up a fresh wikibase suite instance with an existing config/LocalSettings.php file. This is currently not supported. When starting a fresh wikibase suite instance (fresh as in "no volumes exist"), make sure to move your config/LocalSettings.php out of the way too. If you did not change any configuration besides .env yet, you can just remove all files in config/*. This will allow the MediaWiki installer to generate a fresh configuration. If you had modifications in config, feel free to add them back after the first launch.

Hope that helps,
Best,
Robert

darthmon_wmde changed the task status from Open to In Progress.Nov 14 2024, 5:30 PM

darthmon_wmde moved this task from In Review to Doing on the Wikibase Suite Team (Sprint-∞) board.

darthmon_wmde moved this task from Doing to In Review on the Wikibase Suite Team (Sprint-∞) board.

Hi again,

My colleague has tried 4.0.3 and considered the suggestion given above. Still we got errors on sparsql errors from the webbrowser,

Error handling response: TypeError: Cannot read properties of undefined (reading 'id')

at <URL>

Loading messages from: i18n/en.json

Thanks!

Regards/Dann

Hi @Dann.wu,

Thanks a lot for reporting back.

My colleague has tried 4.0.3

Sorry, can you please explain what you mean by 4.0.3? Wikibase Suite does not have this version in any component yet.

Error handling response: TypeError: Cannot read properties of undefined (reading 'id')

That sounds like a completely different error compared to the original report. Can you please a elaborate bit what you are trying to do? Maybe add a screenshot? This would help me to understand where you are at.

Thanks a lot,
Best,
Robert

Hi @roti_WMDE,

Please try some queries on https://wdqs-frontend.almeda.engelska.uu.se

select ?item ?itemLabel where {
values ?url {"https://www.wikidata.org/entity/Q262"}

?item wdt:P1 ?url.
SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE]". }

}

select ?item ?itemLabel where {

?item wdt:P4 wd:Q1.
SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE]". }

}

Hi @Dann.wu,

Thanks a lot for the link. That helps a lot.

One issue I can observe on your instance is that your WDQS DB is not properly synced to your MediaWiki DB.

E.g. on MediaWiki I can spot Q5 which is not available on WDQS (while Q9 is available). If you updated your instance after creating Q5 and before creating Q9, this could be the reason. The simplest solution to resync all the data from MediaWiki to WDQS might be to delete your WDQS database. Make sure to have a backup if you cannot risk loosing data! To delete your WDQS DB and resync from MediaWiki, do:

cd path/to/deploy
docker compose down
docker volume rm wbs-deploy_wdqs-data
docker compose up

Now, Q5 should be available on WDQS too. Test it here.

A second issue could be a confusion about SPARQL prefixes.

Your queries above use wd: and wdt: as prefixes. There has been some confusion about that on our side too recently, but these prefixes are allocated for Wikidata. Your wikibase instance should define own prefixes. This is also the suggestion for wikibase.cloud. Find some more details here.

So if you decide your prefix base should be uuea, you could write the following query:

PREFIX uuea: <https://wikibase.almeda.engelska.uu.se/entity/>
PREFIX uueat: <https://wikibase.almeda.engelska.uu.se/prop/direct/>

SELECT ?item ?itemLabel WHERE {
  ?item uueat:P4 uuea:Q1.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE]". }
}

This already works on your instance!

Hope that helps,
Best,
Robert

Hi @roti_WMDE,

I am so grateful for your answer. How about using Openrefine on our installed instance, does it need the configured namespace for using all functions in OpenRefine, including cluster and reconciling?

Best

Dann

Hi @Dann.wu,

I am glad the answer appears helpful to you. I will now close this ticket as I assume the initial issue is resolved. Feel free to re-open if MediaWiki-WDQS sync issues persist. If other issues appear, please open a new ticket so we track tasks and topics cleanly. Ideally, please use this ticket template for that.

Regarding OpenRefine, we currently do not officially support it and I personally do not have any knowledge about it. Unfortunately, I cannot help here. The OpenRefine Wikimedia or the Wikibase community Telegram groups might be a good place to ask about that.

Best,
Robert

• roti_WMDE closed this task as Resolved.Nov 29 2024, 11:04 AM

• roti_WMDE moved this task from In Review to Done on the Wikibase Suite Team (Sprint-∞) board.

• roti_WMDE moved this task from Sprint-∞ to Archive (closed) on the Wikibase Suite Team board.Dec 10 2024, 1:07 PM

• roti_WMDE edited projects, added Wikibase Suite Team; removed Wikibase Suite Team (Sprint-∞).

Configuration for sparsql in WikibaseClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Configuration for sparsql in Wikibase
Closed, ResolvedPublic
Actions