Mercurial > p > roundup > code
annotate roundup/backends/indexer_common.py @ 6588:91ab3e0ffcd0
Summary: Add test cases for sqlite fts
Add support for using the FTS5 full text query engine for sqlite.
Also stubbed out some sections for adding postgresql FTS support as
well.
Added nee indexer type native-fts. It is not selected by default. The
indexer=native is used if no indexer is set. This prevents an upgrade
from seeming to wipe out the native index if upgraded and
indexer=native is not explicitly set.
Docs updated. Also changed section headers to sentence case for the
current release notes.
Indexing backend can control if the full text search phrase is broken
into a list of words or passed intact. For backends with query
languages (sqlite and can be enabled for whoosh and xapian) we do not
want the phrase "tokenized" on whitespace.
This also updates the rdbms database version to version 7 to add FTS
table. I will be using the same version when I add postgresql. If
somebody runs this version on postgresql, they will have to manually
add the fts tables for postgresql if they want to use it.
Added a new renderError method to client. This allows errors to be
reported still using page.html rather than raw html. It also supports
templates for any error code. If no template for the error code
(e.g. 400) is found, the error in raw html with no page frame is
shown.
New IndexerQueryError exception to pass back message about query syntax
errors.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Sun, 23 Jan 2022 18:57:45 -0500 |
| parents | 9d209d2b34ae |
| children | 39189dd94f2c |
| rev | line source |
|---|---|
|
4089
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4017
diff
changeset
|
1 import re |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
2 |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
3 from roundup import hyperdb |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
4 |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
5 STOPWORDS = [ |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
6 "A", "AND", "ARE", "AS", "AT", "BE", "BUT", "BY", |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
7 "FOR", "IF", "IN", "INTO", "IS", "IT", |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
8 "NO", "NOT", "OF", "ON", "OR", "SUCH", |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
9 "THAT", "THE", "THEIR", "THEN", "THERE", "THESE", |
|
3997
edbb89730dc2
Fix indexer handling of indexed Link properties
Richard Jones <richard@users.sourceforge.net>
parents:
3751
diff
changeset
|
10 "THEY", "THIS", "TO", "WAS", "WILL", "WITH" |
|
3092
a8c2371f45b6
Some cleanup:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
3088
diff
changeset
|
11 ] |
|
a8c2371f45b6
Some cleanup:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
3088
diff
changeset
|
12 |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
13 def _isLink(propclass): |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
14 return (isinstance(propclass, hyperdb.Link) or |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
15 isinstance(propclass, hyperdb.Multilink)) |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
16 |
|
3613
5f4db2650da3
implement close() on all indexers [SF#1242477]
Richard Jones <richard@users.sourceforge.net>
parents:
3544
diff
changeset
|
17 class Indexer: |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
18 def __init__(self, db): |
|
4089
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4017
diff
changeset
|
19 self.stopwords = set(STOPWORDS) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
20 for word in db.config[('main', 'indexer_stopwords')]: |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
21 self.stopwords.add(word) |
|
4252
2ff6f39aa391
Indexers behaviour made more consistent regarding length of indexed words...
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4089
diff
changeset
|
22 # Do not index anything longer than 25 characters since that'll be |
|
2ff6f39aa391
Indexers behaviour made more consistent regarding length of indexed words...
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4089
diff
changeset
|
23 # gibberish (encoded text or somesuch) or shorter than 2 characters |
|
2ff6f39aa391
Indexers behaviour made more consistent regarding length of indexed words...
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4089
diff
changeset
|
24 self.minlength = 2 |
|
2ff6f39aa391
Indexers behaviour made more consistent regarding length of indexed words...
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4089
diff
changeset
|
25 self.maxlength = 25 |
|
6353
9d209d2b34ae
Add indexer_language to change stemmer for xapian FTS indexer
John Rouillard <rouilj@ieee.org>
parents:
5395
diff
changeset
|
26 self.language = db.config[('main','indexer_language')] |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
27 # Some indexers have a query language. If that is the case, |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
28 # we don't parse the user supplied query into a wordlist. |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
29 self.query_language = False |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
30 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
31 def is_stopword(self, word): |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
32 return word in self.stopwords |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
33 |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
34 def getHits(self, search_terms, klass): |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
35 return self.find(search_terms) |
|
3997
edbb89730dc2
Fix indexer handling of indexed Link properties
Richard Jones <richard@users.sourceforge.net>
parents:
3751
diff
changeset
|
36 |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
37 def search(self, search_terms, klass, ignore={}): |
|
4089
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4017
diff
changeset
|
38 """Display search results looking for [search, terms] associated |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
39 with the hyperdb Class "klass". Ignore hits on {class: property}. |
|
4089
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4017
diff
changeset
|
40 """ |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
41 # do the index lookup |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
42 hits = self.getHits(search_terms, klass) |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
43 if not hits: |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
44 return {} |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
45 |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
46 designator_propname = {} |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5388
diff
changeset
|
47 for nm, propclass in klass.getprops().items(): |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
48 if _isLink(propclass): |
|
3751
44603dd791b7
full-text search wasn't coping with multiple multilinks to the same class
Richard Jones <richard@users.sourceforge.net>
parents:
3718
diff
changeset
|
49 designator_propname.setdefault(propclass.classname, |
|
44603dd791b7
full-text search wasn't coping with multiple multilinks to the same class
Richard Jones <richard@users.sourceforge.net>
parents:
3718
diff
changeset
|
50 []).append(nm) |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
51 |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
52 # build a dictionary of nodes and their associated messages |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
53 # and files |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
54 nodeids = {} # this is the answer |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
55 propspec = {} # used to do the klass.find |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5388
diff
changeset
|
56 for l in designator_propname.values(): |
|
3751
44603dd791b7
full-text search wasn't coping with multiple multilinks to the same class
Richard Jones <richard@users.sourceforge.net>
parents:
3718
diff
changeset
|
57 for propname in l: |
|
44603dd791b7
full-text search wasn't coping with multiple multilinks to the same class
Richard Jones <richard@users.sourceforge.net>
parents:
3718
diff
changeset
|
58 propspec[propname] = {} # used as a set (value doesn't matter) |
|
3718
0d561b24ceff
support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents:
3613
diff
changeset
|
59 |
|
0d561b24ceff
support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents:
3613
diff
changeset
|
60 # don't unpack hits entries as sqlite3's Row can't be unpacked :( |
|
0d561b24ceff
support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents:
3613
diff
changeset
|
61 for entry in hits: |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
62 # skip this result if we don't care about this class/property |
|
3718
0d561b24ceff
support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents:
3613
diff
changeset
|
63 classname = entry[0] |
|
0d561b24ceff
support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents:
3613
diff
changeset
|
64 property = entry[2] |
|
4357
13b3155869e0
Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents:
4281
diff
changeset
|
65 if (classname, property) in ignore: |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
66 continue |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
67 |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
68 # if it's a property on klass, it's easy |
|
3998
20c9a1cefb39
make sure item ids are str()
Richard Jones <richard@users.sourceforge.net>
parents:
3997
diff
changeset
|
69 # (make sure the nodeid is str() not unicode() as returned by some |
|
20c9a1cefb39
make sure item ids are str()
Richard Jones <richard@users.sourceforge.net>
parents:
3997
diff
changeset
|
70 # backends as that can cause problems down the track) |
|
20c9a1cefb39
make sure item ids are str()
Richard Jones <richard@users.sourceforge.net>
parents:
3997
diff
changeset
|
71 nodeid = str(entry[1]) |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
72 if classname == klass.classname: |
|
4357
13b3155869e0
Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents:
4281
diff
changeset
|
73 if nodeid not in nodeids: |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
74 nodeids[nodeid] = {} |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
75 continue |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
76 |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
77 # make sure the class is a linked one, otherwise ignore |
|
4357
13b3155869e0
Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents:
4281
diff
changeset
|
78 if classname not in designator_propname: |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
79 continue |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
80 |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
81 # it's a linked class - set up to do the klass.find |
|
3751
44603dd791b7
full-text search wasn't coping with multiple multilinks to the same class
Richard Jones <richard@users.sourceforge.net>
parents:
3718
diff
changeset
|
82 for linkprop in designator_propname[classname]: |
|
44603dd791b7
full-text search wasn't coping with multiple multilinks to the same class
Richard Jones <richard@users.sourceforge.net>
parents:
3718
diff
changeset
|
83 propspec[linkprop][nodeid] = 1 |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
84 |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
85 # retain only the meaningful entries |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4357
diff
changeset
|
86 for propname, idset in list(propspec.items()): |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
87 if not idset: |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
88 del propspec[propname] |
|
3751
44603dd791b7
full-text search wasn't coping with multiple multilinks to the same class
Richard Jones <richard@users.sourceforge.net>
parents:
3718
diff
changeset
|
89 |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
90 # klass.find tells me the klass nodeids the linked nodes relate to |
|
3997
edbb89730dc2
Fix indexer handling of indexed Link properties
Richard Jones <richard@users.sourceforge.net>
parents:
3751
diff
changeset
|
91 propdefs = klass.getprops() |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
92 for resid in klass.find(**propspec): |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
93 resid = str(resid) |
|
4017
605f4a7910b4
Small performance-improvement and bug-fix for indexer:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3998
diff
changeset
|
94 if resid in nodeids: |
|
605f4a7910b4
Small performance-improvement and bug-fix for indexer:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3998
diff
changeset
|
95 continue # we ignore duplicate resids |
|
605f4a7910b4
Small performance-improvement and bug-fix for indexer:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3998
diff
changeset
|
96 nodeids[resid] = {} |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
97 node_dict = nodeids[resid] |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
98 # now figure out where it came from |
|
4357
13b3155869e0
Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents:
4281
diff
changeset
|
99 for linkprop in propspec: |
|
3997
edbb89730dc2
Fix indexer handling of indexed Link properties
Richard Jones <richard@users.sourceforge.net>
parents:
3751
diff
changeset
|
100 v = klass.get(resid, linkprop) |
|
edbb89730dc2
Fix indexer handling of indexed Link properties
Richard Jones <richard@users.sourceforge.net>
parents:
3751
diff
changeset
|
101 # the link might be a Link so deal with a single result or None |
|
edbb89730dc2
Fix indexer handling of indexed Link properties
Richard Jones <richard@users.sourceforge.net>
parents:
3751
diff
changeset
|
102 if isinstance(propdefs[linkprop], hyperdb.Link): |
|
edbb89730dc2
Fix indexer handling of indexed Link properties
Richard Jones <richard@users.sourceforge.net>
parents:
3751
diff
changeset
|
103 if v is None: continue |
|
edbb89730dc2
Fix indexer handling of indexed Link properties
Richard Jones <richard@users.sourceforge.net>
parents:
3751
diff
changeset
|
104 v = [v] |
|
edbb89730dc2
Fix indexer handling of indexed Link properties
Richard Jones <richard@users.sourceforge.net>
parents:
3751
diff
changeset
|
105 for nodeid in v: |
|
4357
13b3155869e0
Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents:
4281
diff
changeset
|
106 if nodeid in propspec[linkprop]: |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
107 # OK, this node[propname] has a winner |
|
4357
13b3155869e0
Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents:
4281
diff
changeset
|
108 if linkprop not in node_dict: |
|
3058
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
109 node_dict[linkprop] = [nodeid] |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
110 else: |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
111 node_dict[linkprop].append(nodeid) |
|
1c063814d567
Move search method duplicated in indexer_dbm and indexer_tsearch2...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
diff
changeset
|
112 return nodeids |
|
3613
5f4db2650da3
implement close() on all indexers [SF#1242477]
Richard Jones <richard@users.sourceforge.net>
parents:
3544
diff
changeset
|
113 |
|
5096
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
114 def get_indexer(config, db): |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
115 indexer_name = getattr(config, "INDEXER", "") |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
116 if not indexer_name: |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
117 # Try everything |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
118 try: |
|
5388
d26921b851c3
Python 3 preparation: make relative imports explicit.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5096
diff
changeset
|
119 from .indexer_xapian import Indexer |
|
5096
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
120 return Indexer(db) |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
121 except ImportError: |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
122 pass |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
123 |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
124 try: |
|
5388
d26921b851c3
Python 3 preparation: make relative imports explicit.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5096
diff
changeset
|
125 from .indexer_whoosh import Indexer |
|
5096
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
126 return Indexer(db) |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
127 except ImportError: |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
128 pass |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
129 |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
130 indexer_name = "native" # fallback to native full text search |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
131 |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
132 if indexer_name == "xapian": |
|
5388
d26921b851c3
Python 3 preparation: make relative imports explicit.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5096
diff
changeset
|
133 from .indexer_xapian import Indexer |
|
5096
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
134 return Indexer(db) |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
135 |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
136 if indexer_name == "whoosh": |
|
5388
d26921b851c3
Python 3 preparation: make relative imports explicit.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5096
diff
changeset
|
137 from .indexer_whoosh import Indexer |
|
5096
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
138 return Indexer(db) |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
139 |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
140 if indexer_name == "native-fts": |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
141 if db.dbtype not in ("sqlite", "postgres"): |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
142 raise AssertionError("Indexer native-fts is configured, but only sqlite and postgres support it. Database is: %r" % db.dbtype) |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
143 |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
144 if db.dbtype == "sqlite": |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
145 from roundup.backends.indexer_sqlite_fts import Indexer |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
146 return Indexer(db) |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
147 |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
148 if db.dbtype == "postgres": |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
149 raise NotImplementedError("Postgres FTS not available") |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
150 from roundup.backends.indexer_postgres_fts import Indexer |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
151 return Indexer(db) |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6353
diff
changeset
|
152 |
|
5096
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
153 if indexer_name == "native": |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
154 # load proper native indexing based on database type |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
155 if db.dbtype == "anydbm": |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
156 from roundup.backends.indexer_dbm import Indexer |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
157 return Indexer(db) |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
158 |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
159 if db.dbtype in ("sqlite", "postgres", "mysql"): |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
160 from roundup.backends.indexer_rdbms import Indexer |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
161 return Indexer(db) |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
162 |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
163 raise AssertionError("Invalid indexer: %r" %(indexer_name)) |
|
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
4687
diff
changeset
|
164 |
