Mercurial > p > roundup > code
annotate roundup/backends/indexer_rdbms.py @ 5543:bc3e00a3d24b
MySQL backend fixes for Python 3.
With Python 2, text sent to and from MySQL is treated as bytes in
Python. The database may be recorded by MySQL as having some other
encoding (latin1 being the default in some MySQL versions - Roundup
does not set an encoding explicitly, unlike in back_postgresql), but
as long as MySQL's notion of the connection encoding agrees with its
notion of the database encoding, no conversions actually take place
and the bytes are stored and returned as-is.
With Python 3, text sent to and from MySQL is treated as Python
Unicode strings. When the database and connection encoding is latin1,
that means the bytes stored in the database under Python 2 are
interpreted as latin1 and converted from that to Unicode, producing
incorrect results for any non-ASCII characters; furthermore, if trying
to store new non-ASCII data in the database under Python 3, any
non-latin1 characters produce errors.
This patch arranges for both the connection and database character
sets to be UTF-8 when using Python 3, and documents a need to export
and import the database when moving from Python 2 to Python 3 with
this backend.
| author | Joseph Myers <jsm@polyomino.org.uk> |
|---|---|
| date | Sun, 16 Sep 2018 16:19:20 +0000 |
| parents | 56c9bcdea47f |
| children | 8bda74ee7070 |
| rev | line source |
|---|---|
|
4089
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4008
diff
changeset
|
1 """ This implements the full-text indexer over two RDBMS tables. The first |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
2 is a mapping of words to occurance IDs. The second maps the IDs to (Class, |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
3 propname, itemid) instances. |
|
4089
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4008
diff
changeset
|
4 """ |
|
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4008
diff
changeset
|
5 import re |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
6 |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3414
diff
changeset
|
7 from roundup.backends.indexer_common import Indexer as IndexerBase |
|
5416
56c9bcdea47f
Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5391
diff
changeset
|
8 from roundup.anypy.strings import us2u, u2s |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
9 |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3414
diff
changeset
|
10 class Indexer(IndexerBase): |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
11 def __init__(self, db): |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3414
diff
changeset
|
12 IndexerBase.__init__(self, db) |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
13 self.db = db |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
14 self.reindex = 0 |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
15 |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
16 def close(self): |
|
4089
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4008
diff
changeset
|
17 """close the indexing database""" |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
18 # just nuke the circular reference |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
19 self.db = None |
|
3295
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
20 |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
21 def save_index(self): |
|
4089
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4008
diff
changeset
|
22 """Save the changes to the index.""" |
|
3295
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
23 # not necessary - the RDBMS connection will handle this for us |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
24 pass |
|
3331
7bc09d5d9544
perform word splitting in unicode for national characters support
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3295
diff
changeset
|
25 |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
26 def force_reindex(self): |
|
4089
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4008
diff
changeset
|
27 """Force a reindexing of the database. This essentially |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
28 empties the tables ids and index and sets a flag so |
|
4089
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4008
diff
changeset
|
29 that the databases are reindexed""" |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
30 self.reindex = 1 |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
31 |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
32 def should_reindex(self): |
|
4089
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4008
diff
changeset
|
33 """returns True if the indexes need to be rebuilt""" |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
34 return self.reindex |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
35 |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
36 def add_text(self, identifier, text, mime_type='text/plain'): |
|
4089
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4008
diff
changeset
|
37 """ "identifier" is (classname, itemid, property) """ |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
38 if mime_type != 'text/plain': |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
39 return |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
40 |
|
3988
b50446ff16f4
fix comment, thanks alex
Richard Jones <richard@users.sourceforge.net>
parents:
3981
diff
changeset
|
41 # Ensure all elements of the identifier are strings 'cos the itemid |
|
3981
82e116d515d2
another psql 8.3 compat fix
Richard Jones <richard@users.sourceforge.net>
parents:
3718
diff
changeset
|
42 # column is varchar even if item ids may be numbers elsewhere in the |
|
82e116d515d2
another psql 8.3 compat fix
Richard Jones <richard@users.sourceforge.net>
parents:
3718
diff
changeset
|
43 # code. ugh. |
|
4008
0bf9f8ae7d1b
fix bug introduced in 1.4.5 in RDBMS full-text indexing;
Richard Jones <richard@users.sourceforge.net>
parents:
3988
diff
changeset
|
44 identifier = tuple(map(str, identifier)) |
|
3981
82e116d515d2
another psql 8.3 compat fix
Richard Jones <richard@users.sourceforge.net>
parents:
3718
diff
changeset
|
45 |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
46 # first, find the id of the (classname, itemid, property) |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
47 a = self.db.arg |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
48 sql = 'select _textid from __textids where _class=%s and '\ |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
49 '_itemid=%s and _prop=%s'%(a, a, a) |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
50 self.db.cursor.execute(sql, identifier) |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
51 r = self.db.cursor.fetchone() |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
52 if not r: |
|
3617
f12722c7b9ee
improvements
Richard Jones <richard@users.sourceforge.net>
parents:
3544
diff
changeset
|
53 # not previously indexed |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
54 id = self.db.newid('__textids') |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
55 sql = 'insert into __textids (_textid, _class, _itemid, _prop)'\ |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
56 ' values (%s, %s, %s, %s)'%(a, a, a, a) |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
57 self.db.cursor.execute(sql, (id, ) + identifier) |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
58 else: |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
59 id = int(r[0]) |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
60 # clear out any existing indexed values |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
61 sql = 'delete from __words where _textid=%s'%a |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
62 self.db.cursor.execute(sql, (id, )) |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
63 |
|
3617
f12722c7b9ee
improvements
Richard Jones <richard@users.sourceforge.net>
parents:
3544
diff
changeset
|
64 # ok, find all the unique words in the text |
|
5416
56c9bcdea47f
Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5391
diff
changeset
|
65 text = us2u(text, "replace") |
|
4284
86e8b2a615fa
Fix Issue2550609, hopefully for real, this time.
Stefan Seefeld <stefan@seefeld.name>
parents:
4283
diff
changeset
|
66 text = text.upper() |
|
5416
56c9bcdea47f
Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5391
diff
changeset
|
67 wordlist = [u2s(w) |
| 4282 | 68 for w in re.findall(r'(?u)\b\w{%d,%d}\b' |
| 69 % (self.minlength, self.maxlength), text)] | |
|
4089
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4008
diff
changeset
|
70 words = set() |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
71 for word in wordlist: |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3414
diff
changeset
|
72 if self.is_stopword(word): continue |
|
3617
f12722c7b9ee
improvements
Richard Jones <richard@users.sourceforge.net>
parents:
3544
diff
changeset
|
73 words.add(word) |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
74 |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
75 # for each word, add an entry in the db |
|
3617
f12722c7b9ee
improvements
Richard Jones <richard@users.sourceforge.net>
parents:
3544
diff
changeset
|
76 sql = 'insert into __words (_word, _textid) values (%s, %s)'%(a, a) |
|
f12722c7b9ee
improvements
Richard Jones <richard@users.sourceforge.net>
parents:
3544
diff
changeset
|
77 words = [(word, id) for word in words] |
|
3618
b31a2e35be80
pysqlite 1.1.6 does not allow to pass a list of tuples to cursor.execute().
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3617
diff
changeset
|
78 self.db.cursor.executemany(sql, words) |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
79 |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
80 def find(self, wordlist): |
|
4089
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4008
diff
changeset
|
81 """look up all the words in the wordlist. |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
82 If none are found return an empty dictionary |
|
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
83 * more rules here |
|
4089
eddb82d0964c
Add compatibility package to allow us to deal with Python versions 2.3..2.6.
Richard Jones <richard@users.sourceforge.net>
parents:
4008
diff
changeset
|
84 """ |
|
3033
f8d0fd056ac0
fix indexer searching with no valid words [SF#1086787]
Richard Jones <richard@users.sourceforge.net>
parents:
2872
diff
changeset
|
85 if not wordlist: |
|
4008
0bf9f8ae7d1b
fix bug introduced in 1.4.5 in RDBMS full-text indexing;
Richard Jones <richard@users.sourceforge.net>
parents:
3988
diff
changeset
|
86 return [] |
|
3033
f8d0fd056ac0
fix indexer searching with no valid words [SF#1086787]
Richard Jones <richard@users.sourceforge.net>
parents:
2872
diff
changeset
|
87 |
|
4252
2ff6f39aa391
Indexers behaviour made more consistent regarding length of indexed words...
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4091
diff
changeset
|
88 l = [word.upper() for word in wordlist |
|
2ff6f39aa391
Indexers behaviour made more consistent regarding length of indexed words...
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4091
diff
changeset
|
89 if self.minlength <= len(word) <= self.maxlength] |
|
2ff6f39aa391
Indexers behaviour made more consistent regarding length of indexed words...
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4091
diff
changeset
|
90 l = [word for word in l if not self.is_stopword(word)] |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
91 |
|
3048
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
92 if not l: |
|
4008
0bf9f8ae7d1b
fix bug introduced in 1.4.5 in RDBMS full-text indexing;
Richard Jones <richard@users.sourceforge.net>
parents:
3988
diff
changeset
|
93 return [] |
|
3048
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
94 |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
95 if self.db.implements_intersect: |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
96 # simple AND search |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
97 sql = 'select distinct(_textid) from __words where _word=%s'%self.db.arg |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
98 sql = '\nINTERSECT\n'.join([sql]*len(l)) |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
99 self.db.cursor.execute(sql, tuple(l)) |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
100 r = self.db.cursor.fetchall() |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
101 if not r: |
|
4008
0bf9f8ae7d1b
fix bug introduced in 1.4.5 in RDBMS full-text indexing;
Richard Jones <richard@users.sourceforge.net>
parents:
3988
diff
changeset
|
102 return [] |
|
3048
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
103 a = ','.join([self.db.arg] * len(r)) |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
104 sql = 'select _class, _itemid, _prop from __textids '\ |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
105 'where _textid in (%s)'%a |
|
3718
0d561b24ceff
support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents:
3618
diff
changeset
|
106 self.db.cursor.execute(sql, tuple([int(row[0]) for row in r])) |
|
3048
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
107 |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
108 else: |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
109 # A more complex version for MySQL since it doesn't implement INTERSECT |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
110 |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
111 # Construct SQL statement to join __words table to itself |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
112 # multiple times. |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
113 sql = """select distinct(__words1._textid) |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
114 from __words as __words1 %s |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
115 where __words1._word=%s %s""" |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
116 |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
117 join_tmpl = ' left join __words as __words%d using (_textid) \n' |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
118 match_tmpl = ' and __words%d._word=%s \n' |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
119 |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
120 join_list = [] |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
121 match_list = [] |
|
5391
a391a071d045
Python 3 preparation: use range() instead of xrange().
Joseph Myers <jsm@polyomino.org.uk>
parents:
4687
diff
changeset
|
122 for n in range(len(l) - 1): |
|
3048
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
123 join_list.append(join_tmpl % (n + 2)) |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
124 match_list.append(match_tmpl % (n + 2, self.db.arg)) |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
125 |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
126 sql = sql%(' '.join(join_list), self.db.arg, ' '.join(match_list)) |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
127 self.db.cursor.execute(sql, l) |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
128 |
|
4357
13b3155869e0
Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents:
4284
diff
changeset
|
129 r = [x[0] for x in self.db.cursor.fetchall()] |
|
3048
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
130 if not r: |
|
4008
0bf9f8ae7d1b
fix bug introduced in 1.4.5 in RDBMS full-text indexing;
Richard Jones <richard@users.sourceforge.net>
parents:
3988
diff
changeset
|
131 return [] |
|
3048
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
132 |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
133 a = ','.join([self.db.arg] * len(r)) |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
134 sql = 'select _class, _itemid, _prop from __textids '\ |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
135 'where _textid in (%s)'%a |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
136 |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
137 self.db.cursor.execute(sql, tuple(map(int, r))) |
|
d9b4224f955c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3033
diff
changeset
|
138 |
|
3076
2817a4db901d
Change indexer_common.search() to take a list of nodeids...
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
3048
diff
changeset
|
139 return self.db.cursor.fetchall() |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
140 |
