Wikidata integration + Refactoring Core to accept new formats #155

hadyelsahar · 2014-01-23T15:12:38Z

This pull request is mainly the output of GSoC2013 Wikidata Integration inside DBpedia Project

added JsonParser to Parse Json Pages Extracted from Wikidata
added multiple Extractors to Extract Triples out of Wikidata Json Dumps
updating the ontology file by adding Wikidata to DBpedia mappings for Properties and Types in the Mappings wiki
the core refactoring issue Refactor core to accept new formats #38 to easily accept new Page formats Like Json in case of Wikidata

there are two main Issues left to be done after the merge :

1 - there are two warnings left for JsonLift: 'withFilter' method does not yet exist on net.liftweb.json.JsonAST.JValue, using `filter' method instead, try mvn clean install to see them

2- the Design of the Wikidata Extractors and JsonWikiParser , were temporary design to cope with the old core , now the new core is redesigned so what we should do :

JsonNode should hold a JsonObject, not a list of Nodes
move all the extra code from JsonParser (that generates AST) into the individual extractors Every extractor should define at the beginning variables with the paths that are of interest (e.g. json \ "claims" \ ...) and later reuse these variables.
Create org.dbpedia.extraction.util.JSONUtils where you can store all the Json related custom function you create
adapt the Wikidata Extractors to new Wikidata Json Changes

various changes

…xtractor[WikiPage]

final Live merge

ninniuz · 2014-01-24T09:27:20Z

core/src/main/scala/org/dbpedia/extraction/destinations/DBpediaDatasets.scala

Naming coherence:

val WikidataNamespaceSameAs = new Dataset("wikidata-namespace-sameas")

ninniuz · 2014-01-24T11:00:44Z

core/src/main/scala/org/dbpedia/extraction/mappings/CategoryLabelExtractor.scala

Should use WikiPage.sourceIri (which does not exist).
WikiPage has a revision property though, so a valid (and coherent with current framework) sourceIri could be:

WikiPage.title.pageIri + "?oldid=" + revision

similar to PageNode.sourceUri

PageNode.title.pageIri does not give enough context to this quad.

ninniuz · 2014-01-24T11:39:58Z

core/src/main/scala/org/dbpedia/extraction/mappings/WikiParseExtractor.scala

Remove space after WikiPage

fix review comments

ninniuz · 2014-01-30T11:16:33Z

core/src/main/scala/org/dbpedia/extraction/mappings/WikidataExtractor.scala

Is this method doing anything?

no it's the dummy Wikidata Extractor Structure that Dimitris created in the beginning of the task , we can remove the whole file it's obsolete anyways

go ahead and remove it Hady

…ata.dbpedia.org/resource/XXX

hadyelsahar · 2014-02-23T07:37:53Z

last edits :
1- handeled Andreas last two comments considering owl:sameas property and page.wikiPage.SourceUri

2- wikidata uris to be : "http://www.wikidata.dbpedia.org/resource/" instead of ""http://www.wikidata.org/entity/"
it was already implemented but i think we overwritten it when merging or something

Wikidata integration + Refactoring Core to accept new formats

hadyelsahar and others added 19 commits December 11, 2013 20:01

draft core refactoring for wikidata

908cfce

fixed type conflicts

3124d7d

added wikiPageExtractor trait to match extractors that accepts WikiPage

e668fff

various changes

16ee866

Merge pull request #3 from jimkont/wikidata

7ce3d27

various changes

added wikipage extractor

1b7f815

various changes by dimitris

2811e1f

changed updated ArticlePageExtractor in prop file

fed7493

fixed issues in Sever Module to deal with Parser -> Option

dd6f3ca

fixed bugs in server module due to chainging Extractor[PageNode] to E…

b0671ab

…xtractor[WikiPage]

import scala.language.reflectiveCalls

9d96186

adapt for the new parser output

e9976d4

remove obsolete classes

f5eb297

throw exception on parsing error

909a741

merge with latest master

edab2b6

skipped HomepageExtractor test on merge

3c73aa8

comment unneeded constants (todo: remove all & reuse Namespace class)

d5b578f

Adapt Live core to work with the refactored framework

6209a69

Merge pull request #4 from jimkont/wikidata

563287b

final Live merge

ninniuz reviewed Jan 24, 2014
View reviewed changes

rename Extractor with a more representative name

8984a53

ninniuz reviewed Jan 24, 2014
View reviewed changes

jimkont added 5 commits January 24, 2014 13:11

fix HomepageExtractorTest (regression)

0724dbe

Use revision URI as context in WikiPageExtractors

848b195

Naming coherence (review)

28ca4fe

whitespace consistency

e50eefe

fix comment

115159e

adjust extraction properties with the renamed ProvenanceExtractor

460f3d0

ninniuz reviewed Jan 24, 2014
View reviewed changes

core/src/main/scala/org/dbpedia/extraction/mappings/WikiParseExtractor.scala Outdated

Copy link

Contributor

ninniuz Jan 24, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove space after WikiPage

hadyelsahar added 8 commits January 24, 2014 08:57

Merge pull request #5 from jimkont/wikidata

8f16d5f

fix review comments

added comments to Datasets Destinations

e535730

remove extra space after Wikipage

867d0aa

remove extra space

6cf62af

remove extra space after JsonNode

50da5c0

removing extra spaces

eda16c1

removing extra spaces before WikiPage

31bf421

removal of extraspace

52b2f09

ninniuz reviewed Jan 30, 2014
View reviewed changes

hadyelsahar added 6 commits February 23, 2014 06:03

deleted wikidataExtractor template file

8bdd1ef

aaa

a67749b

remove wikidataExtractor template

bdb4101

add owl:sameas as ontologyproperty instead of string

54c6b7d

added edited namespace of wikdata entities inside dbpedia to be wikid…

bc2b269

…ata.dbpedia.org/resource/XXX

page.wikiPage.sourceUri instead of page.wikiPage.title.pageIri

6b661cd

jimkont added a commit that referenced this pull request Mar 1, 2014

Merge pull request #155 from hadyelsahar/wikidata

fb89d3b

Wikidata integration + Refactoring Core to accept new formats

jimkont merged commit fb89d3b into dbpedia:master Mar 1, 2014

jimkont modified the milestone: pastReleases Mar 19, 2015

jimkont added a commit to jimkont/extraction-framework that referenced this pull request Mar 26, 2015

Merge pull request dbpedia#155 from hadyelsahar/wikidata

7227a67

Wikidata integration + Refactoring Core to accept new formats

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Wikidata integration + Refactoring Core to accept new formats #155

Wikidata integration + Refactoring Core to accept new formats #155

Uh oh!

hadyelsahar commented Jan 23, 2014

Uh oh!

ninniuz Jan 24, 2014

Uh oh!

ninniuz Jan 24, 2014

Uh oh!

ninniuz Jan 24, 2014

Uh oh!

ninniuz Jan 30, 2014

Uh oh!

hadyelsahar Jan 30, 2014

Uh oh!

jimkont Feb 22, 2014

Uh oh!

hadyelsahar Feb 23, 2014

Uh oh!

hadyelsahar commented Feb 23, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Wikidata integration + Refactoring Core to accept new formats #155

Wikidata integration + Refactoring Core to accept new formats #155

Uh oh!

Conversation

hadyelsahar commented Jan 23, 2014

This pull request is mainly the output of GSoC2013 Wikidata Integration inside DBpedia Project

there are two main Issues left to be done after the merge :

Uh oh!

ninniuz Jan 24, 2014

Choose a reason for hiding this comment

Uh oh!

ninniuz Jan 24, 2014

Choose a reason for hiding this comment

Uh oh!

ninniuz Jan 24, 2014

Choose a reason for hiding this comment

Uh oh!

ninniuz Jan 30, 2014

Choose a reason for hiding this comment

Uh oh!

hadyelsahar Jan 30, 2014

Choose a reason for hiding this comment

Uh oh!

jimkont Feb 22, 2014

Choose a reason for hiding this comment

Uh oh!

hadyelsahar Feb 23, 2014

Choose a reason for hiding this comment

Uh oh!

hadyelsahar commented Feb 23, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants