Mercurial > p > roundup > code
annotate roundup/backends/rdbms_common.py @ 7665:6f841a55eabf
fix: database error handling during import of a non-user item
The code to handle the case of retired and active users causing errors
during import can get called on other db errors as well (e.g. out of
memory in postgresql).
The code that trys to detect/repair the out of order user case hard
coded the db.user class. This crashed when a db error occurred when
loading another data object (e.g. msg). Fix the crash by using the
proper db object class for the object class being loaded.
Credit to Norbert Schlemmer for finding this.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Wed, 18 Oct 2023 15:49:53 -0400 |
| parents | 506c86823abb |
| children | 78585199552a 4af0d235b570 |
| rev | line source |
|---|---|
|
3554
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
1 # |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/) |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
3 # This module is free software, and you may redistribute it and/or modify |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
4 # under the same terms as Python, so long as this copyright message and |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
5 # disclaimer are retained in their original form. |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
6 # |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
10 # POSSIBILITY OF SUCH DAMAGE. |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
11 # |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS" |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
17 # |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
18 """ Relational database (SQL) backend common code. |
|
1244
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
19 |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
20 Basics: |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
21 |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
22 - map roundup classes to relational tables |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
23 - automatically detect schema changes and modify the table schemas |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
24 appropriately (we store the "database version" of the schema in the |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
25 database itself as the only row of the "schema" table) |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
26 - multilinks (which represent a many-to-many relationship) are handled through |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
27 intermediate tables |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
28 - journals are stored adjunct to the per-class tables |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
29 - table names and columns have "_" prepended so the names can't clash with |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
30 restricted names (like "order") |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
31 - retirement is determined by the __retired__ column being > 0 |
|
1244
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
32 |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
33 Database-specific changes may generally be pushed out to the overridable |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
34 sql_* methods, since everything else should be fairly generic. There's |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
35 probably a bit of work to be done if a database is used that actually |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
36 honors column typing, since the initial databases don't (sqlite stores |
|
1528
96cd422532ef
bye bye gadfly - you served your purpose well [SF#701127]
Richard Jones <richard@users.sourceforge.net>
parents:
1523
diff
changeset
|
37 everything as a string.) |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
38 |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
39 The schema of the hyperdb being mapped to the database is stored in the |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
40 database itself as a repr()'ed dictionary of information about each Class |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
41 that maps to a table. If that information differs from the hyperdb schema, |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
42 then we update it. We also store in the schema dict a version which |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
43 allows us to upgrade the database schema when necessary. See upgrade_db(). |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
44 |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
45 To force a unqiueness constraint on the key properties we put the item |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
46 id into the __retired__ column duing retirement (so it's 0 for "active" |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
47 items) and place a unqiueness constraint on key + __retired__. This is |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
48 particularly important for the users class where multiple users may |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
49 try to have the same username, with potentially many retired users with |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
50 the same name. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
51 """ |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
52 __docformat__ = 'restructuredtext' |
| 1165 | 53 |
| 54 # standard python modules | |
| 6965 | 55 import copy |
| 56 import datetime | |
| 57 import logging | |
| 58 import os | |
| 59 import re | |
| 60 import time | |
| 61 | |
| 62 from hashlib import md5 | |
| 1165 | 63 |
| 64 # roundup modules | |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
65 from roundup import hyperdb, date, password, roundupdb, security, support |
| 6965 | 66 from roundup.anypy.strings import b2s, bs2b, us2s, repr_export, eval_import |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
67 from roundup.backends.blobfiles import FileStorage |
|
5096
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
5086
diff
changeset
|
68 from roundup.backends.indexer_common import get_indexer |
|
6915
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
69 from roundup.backends.indexer_common import Indexer as CommonIndexer |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
70 from roundup.backends.sessions_rdbms import Sessions, OneTimeKeys |
|
1499
8ee69708da0c
added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1496
diff
changeset
|
71 from roundup.date import Range |
| 6965 | 72 from roundup.hyperdb import String, Password, Date, Interval, Link, \ |
| 73 Multilink, DatabaseError, Boolean, Number, Integer | |
| 74 from roundup.i18n import _ | |
|
6401
8bc5faeb7677
Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6400
diff
changeset
|
75 from roundup.mlink_expr import compile_expression |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
76 |
|
2692
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
77 # dummy value meaning "argument not passed" |
|
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
78 _marker = [] |
|
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
79 |
| 6965 | 80 # python 3 doesn't have a unicode type |
| 81 try: | |
| 82 unicode # noqa: F821 | |
| 83 except NameError: | |
| 84 unicode = str | |
| 85 | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
86 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
87 def _num_cvt(num): |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
88 num = str(num) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
89 try: |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
90 return int(num) |
| 6002 | 91 except ValueError: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
92 return float(num) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
93 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
94 |
|
2472
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
95 def _bool_cvt(value): |
|
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
96 if value in ('TRUE', 'FALSE'): |
|
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
97 return {'TRUE': 1, 'FALSE': 0}[value] |
|
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
98 # assume it's a number returned from the db API |
|
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
99 return int(value) |
|
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
100 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
101 |
|
4473
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
102 def date_to_hyperdb_value(d): |
|
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
103 """ convert date d to a roundup date """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
104 if isinstance(d, datetime.datetime): |
|
4473
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
105 return date.Date(d) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
106 return date.Date(str(d).replace(' ', '.')) |
|
4473
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
107 |
|
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
108 |
|
2634
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
109 def connection_dict(config, dbnamestr=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
110 """ Used by Postgresql and MySQL to detemine the keyword args for |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
111 opening the database connection.""" |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
112 d = {} |
|
2634
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
113 if dbnamestr: |
|
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
114 d[dbnamestr] = config.RDBMS_NAME |
|
3099
519b92df37dc
handle ~/.my.cnf files for MySQL defaults [SF#1096031]
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
115 for name in ('host', 'port', 'password', 'user', 'read_default_group', |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
116 'read_default_file'): |
|
2634
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
117 cvar = 'RDBMS_'+name.upper() |
|
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
118 if config[cvar] is not None: |
|
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
119 d[name] = config[cvar] |
|
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
120 return d |
|
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
121 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
122 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
123 class IdListOptimizer: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
124 """ To prevent flooding the SQL parser of the underlaying |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
125 db engine with "x IN (1, 2, 3, ..., <large number>)" collapses |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
126 these cases to "x BETWEEN 1 AND <large number>". |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
127 """ |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
128 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
129 def __init__(self): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
130 self.ranges = [] |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
131 self.singles = [] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
132 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
133 def append(self, nid): |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
134 """ Invariant: nid are ordered ascending """ |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
135 if self.ranges: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
136 last = self.ranges[-1] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
137 if last[1] == nid-1: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
138 last[1] = nid |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
139 return |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
140 if self.singles: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
141 last = self.singles[-1] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
142 if last == nid-1: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
143 self.singles.pop() |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
144 self.ranges.append([last, nid]) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
145 return |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
146 self.singles.append(nid) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
147 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
148 def where(self, field, placeholder): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
149 ranges = self.ranges |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
150 singles = self.singles |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
151 |
| 6965 | 152 if not singles and not ranges: return "(1=0)", [] # noqa: E701 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
153 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
154 if ranges: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
155 between = '%s BETWEEN %s AND %s' % ( |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
156 field, placeholder, placeholder) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
157 stmnt = [between] * len(ranges) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
158 else: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
159 stmnt = [] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
160 if singles: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
161 stmnt.append('%s in (%s)' % ( |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
162 field, ','.join([placeholder]*len(singles)))) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
163 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
164 return '(%s)' % ' OR '.join(stmnt), sum(ranges, []) + singles |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
165 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
166 def __str__(self): |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
167 return "ranges: %r / singles: %r" % (self.ranges, self.singles) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
168 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
169 |
| 1165 | 170 class Database(FileStorage, hyperdb.Database, roundupdb.Database): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
171 """ Wrapper around an SQL database that presents a hyperdb interface. |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
172 |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
173 - some functionality is specific to the actual SQL database, hence |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
174 the sql_* methods that are NotImplemented |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
175 - we keep a cache of the latest N row fetches (where N is |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
176 configurable). |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
177 """ |
| 1165 | 178 def __init__(self, config, journaltag=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
179 """ Open the database and load the schema from it. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
180 """ |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3872
diff
changeset
|
181 FileStorage.__init__(self, config.UMASK) |
|
6658
408fd477761f
Add i18n object to roundupdb.Database
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6599
diff
changeset
|
182 roundupdb.Database.__init__(self) |
| 1165 | 183 self.config, self.journaltag = config, journaltag |
| 184 self.dir = config.DATABASE | |
| 185 self.classes = {} | |
|
6915
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
186 # Assign the indexer base class here. During schema |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
187 # generation in open_connection, the min/max size for FTS |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
188 # tokens is used when creating the database tables for |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
189 # indexer=native full text search. These tables are always |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
190 # created as part of the schema so that the admin can choose |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
191 # indexer=native at some later date and "things will just |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
192 # work" (TM). |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
193 # |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
194 # We would like to use get_indexer() to return the real |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
195 # indexer class. However indexer=native-fts for postgres |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
196 # requires a database connection (conn) to be defined when |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
197 # calling get_indexer. The call to open_connection creates the |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
198 # conn but also creates the schema if it is missing. |
|
6918
cb2ed1e8c852
Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents:
6915
diff
changeset
|
199 if self.config['INDEXER'] != 'native-fts': |
|
cb2ed1e8c852
Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents:
6915
diff
changeset
|
200 self.indexer = get_indexer(config, self) |
|
cb2ed1e8c852
Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents:
6915
diff
changeset
|
201 else: |
|
cb2ed1e8c852
Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents:
6915
diff
changeset
|
202 self.indexer = CommonIndexer(self) |
| 1165 | 203 self.security = security.Security(self) |
| 204 | |
| 205 # additional transaction support for external files and the like | |
| 206 self.transactions = [] | |
| 207 | |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
208 # keep a cache of the N most recently retrieved rows of any kind |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
209 # (classname, nodeid) = row |
|
4074
e039f3cbbb96
Make RDBMS cache-size configurable.
Stefan Seefeld <stefan@seefeld.name>
parents:
4062
diff
changeset
|
210 self.cache_size = config.RDBMS_CACHE_SIZE |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
211 self.clearCache() |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
212 self.stats = {'cache_hits': 0, 'cache_misses': 0, 'get_items': 0, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
213 'filtering': 0} |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
214 |
|
5041
5251e97b1de0
Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents:
4995
diff
changeset
|
215 # make sure the database directory exists |
|
5251e97b1de0
Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents:
4995
diff
changeset
|
216 if not os.path.isdir(self.config.DATABASE): |
|
5251e97b1de0
Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents:
4995
diff
changeset
|
217 os.makedirs(self.config.DATABASE) |
|
5251e97b1de0
Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents:
4995
diff
changeset
|
218 |
|
1333
80d27b7d6db5
implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents:
1304
diff
changeset
|
219 # database lock |
|
80d27b7d6db5
implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents:
1304
diff
changeset
|
220 self.lockfile = None |
|
80d27b7d6db5
implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents:
1304
diff
changeset
|
221 |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
222 # Uppercase to not collide with Class names |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
223 self.Session = None |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
224 self.Otk = None |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
225 |
| 1165 | 226 # open a connection to the database, creating the "conn" attribute |
|
2082
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
227 self.open_connection() |
| 1165 | 228 |
|
6915
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
229 # If indexer is native-fts, conn to db must be available. |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
230 # so we set the real self.indexer value here, after db is open. |
|
6918
cb2ed1e8c852
Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents:
6915
diff
changeset
|
231 if self.config['INDEXER'] == 'native-fts': |
|
cb2ed1e8c852
Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents:
6915
diff
changeset
|
232 self.indexer = get_indexer(config, self) |
|
6915
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
233 |
|
1181
49aebf5a8691
some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents:
1176
diff
changeset
|
234 def clearCache(self): |
|
49aebf5a8691
some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents:
1176
diff
changeset
|
235 self.cache = {} |
|
49aebf5a8691
some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents:
1176
diff
changeset
|
236 self.cache_lru = [] |
|
4652
dfbc0cfa9811
Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4517
diff
changeset
|
237 # upcall is necessary! |
|
dfbc0cfa9811
Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4517
diff
changeset
|
238 roundupdb.Database.clearCache(self) |
|
1181
49aebf5a8691
some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents:
1176
diff
changeset
|
239 |
|
2082
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
240 def getSessionManager(self): |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
241 if not self.Session: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
242 self.Session = Sessions(self) |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
243 return self.Session |
|
2082
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
244 |
|
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
245 def getOTKManager(self): |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
246 if not self.Otk: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
247 self.Otk = OneTimeKeys(self) |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
248 return self.Otk |
|
2082
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
249 |
|
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
250 def open_connection(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
251 """ Open a connection to the database, creating it if necessary. |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
252 |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
253 Must call self.load_dbschema() |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
254 """ |
|
5236
0a20e1965f4b
Pylint checks bark about:
John Rouillard <rouilj@ieee.org>
parents:
5232
diff
changeset
|
255 raise NotImplementedError |
| 1165 | 256 |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
257 def sql(self, sql, args=None, cursor=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
258 """ Execute the sql with the optional args. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
259 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
260 self.log_debug('SQL %r %r' % (sql, args)) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
261 if not cursor: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
262 cursor = self.cursor |
| 1165 | 263 if args: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
264 cursor.execute(sql, args) |
| 1165 | 265 else: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
266 cursor.execute(sql) |
| 1165 | 267 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
268 def sql_fetchone(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
269 """ Fetch a single row. If there's nothing to fetch, return None. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
270 """ |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
271 return self.cursor.fetchone() |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
272 |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
273 def sql_fetchall(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
274 """ Fetch all rows. If there's nothing to fetch, return []. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
275 """ |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
276 return self.cursor.fetchall() |
| 1165 | 277 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
278 def sql_fetchiter(self): |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
279 """ Fetch all row as a generator |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
280 """ |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
281 while True: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
282 row = self.cursor.fetchone() |
| 6965 | 283 if not row: break # noqa: E701 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
284 yield row |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
285 |
|
4849
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
286 def search_stringquote(self, value): |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
287 """ Quote a search string to escape magic search characters |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
288 '%' and '_', also need to quote '\' (first) |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
289 Then put '%' around resulting string for LIKE (or ILIKE) operator |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
290 """ |
|
4849
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
291 v = value.replace('\\', '\\\\') |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
292 v = v.replace('%', '\\%') |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
293 v = v.replace('_', '\\_') |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
294 return '%' + v + '%' |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
295 |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
296 def init_dbschema(self): |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
297 self.database_schema = { |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
298 'version': self.current_db_version, |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
299 'tables': {} |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
300 } |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
301 |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
302 def load_dbschema(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
303 """ Load the schema definition that the database currently implements |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
304 """ |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
305 self.cursor.execute('select schema from schema') |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
306 schema = self.cursor.fetchone() |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
307 if schema: |
| 6002 | 308 # bandit - schema is trusted |
| 309 self.database_schema = eval(schema[0]) # nosec | |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
310 else: |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
311 self.database_schema = {} |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
312 |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
313 def save_dbschema(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
314 """ Save the schema definition that the database currently implements |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
315 """ |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
316 s = repr(self.database_schema) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
317 self.sql('delete from schema') |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
318 self.sql('insert into schema values (%s)' % self.arg, (s,)) |
| 1165 | 319 |
| 320 def post_init(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
321 """ Called once the schema initialisation has finished. |
| 1165 | 322 |
| 323 We should now confirm that the schema defined by our "classes" | |
| 324 attribute actually matches the schema in the database. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
325 """ |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
326 super(Database, self).post_init() |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
327 |
|
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
328 # upgrade the database for column type changes, new internal |
|
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
329 # tables, etc. |
|
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
330 save = self.upgrade_db() |
| 3553 | 331 |
| 332 # handle changes in the schema | |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
333 tables = self.database_schema['tables'] |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
334 for classname, spec in self.classes.items(): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
335 if classname in tables: |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
336 dbspec = tables[classname] |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
337 if self.update_class(spec, dbspec): |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
338 tables[classname] = spec.schema() |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
339 save = 1 |
| 1165 | 340 else: |
| 341 self.create_class(spec) | |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
342 tables[classname] = spec.schema() |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
343 save = 1 |
| 1165 | 344 |
|
7038
f524ddc27af8
replace for a,b in x.items() with for a in x.keys()
John Rouillard <rouilj@ieee.org>
parents:
7007
diff
changeset
|
345 for classname in list(tables.keys()): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
346 if classname not in self.classes: |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
347 self.drop_class(classname, tables[classname]) |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
348 del tables[classname] |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
349 save = 1 |
| 1165 | 350 |
| 351 # update the database version of the schema | |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
352 if save: |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
353 self.save_dbschema() |
| 1165 | 354 |
| 355 # reindex the db if necessary | |
| 356 if self.indexer.should_reindex(): | |
| 357 self.reindex() | |
| 358 | |
| 359 # commit | |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
2089
diff
changeset
|
360 self.sql_commit() |
| 1165 | 361 |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
362 # update this number when we need to make changes to the SQL structure |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
363 # of the backend database |
|
6806
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
364 current_db_version = 8 |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
365 db_version_updated = False |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
366 |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
367 def upgrade_db(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
368 """ Update the SQL database to reflect changes in the backend code. |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
369 |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
370 Return boolean whether we need to save the schema. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
371 """ |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
372 version = self.database_schema.get('version', 1) |
|
3993
1eba65a4b0aa
Don't run old code on newer database
Richard Jones <richard@users.sourceforge.net>
parents:
3979
diff
changeset
|
373 if version > self.current_db_version: |
|
1eba65a4b0aa
Don't run old code on newer database
Richard Jones <richard@users.sourceforge.net>
parents:
3979
diff
changeset
|
374 raise DatabaseError('attempting to run rev %d DATABASE with rev ' |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
375 '%d CODE!' % (version, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
376 self.current_db_version)) |
|
2081
fb4bf55b94d7
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2077
diff
changeset
|
377 if version == self.current_db_version: |
|
fb4bf55b94d7
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2077
diff
changeset
|
378 # nothing to do |
|
fb4bf55b94d7
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2077
diff
changeset
|
379 return 0 |
|
fb4bf55b94d7
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2077
diff
changeset
|
380 |
|
2429
e28a5d5d95a7
better upgrading
Richard Jones <richard@users.sourceforge.net>
parents:
2424
diff
changeset
|
381 if version < 2: |
| 4062 | 382 self.log_info('upgrade to version 2') |
|
2100
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
383 # change the schema structure |
|
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
384 self.database_schema = {'tables': self.database_schema} |
|
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
385 |
|
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
386 # version 1 didn't have the actor column (note that in |
|
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
387 # MySQL this will also transition the tables to typed columns) |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
388 self.add_new_columns_v2() |
|
2100
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
389 |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
390 # version 1 doesn't have the OTK, session and indexing in the |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
391 # database |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
392 self.create_version_2_tables() |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
393 |
|
2429
e28a5d5d95a7
better upgrading
Richard Jones <richard@users.sourceforge.net>
parents:
2424
diff
changeset
|
394 if version < 3: |
| 4062 | 395 self.log_info('upgrade to version 3') |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
396 self.fix_version_2_tables() |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
397 |
|
2721
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
398 if version < 4: |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
399 self.log_info('upgrade to version 4') |
|
2721
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
400 self.fix_version_3_tables() |
|
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
401 |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
402 if version < 5: |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
403 self.log_info('upgrade to version 5') |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
404 self.fix_version_4_tables() |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
405 |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
406 if version < 6: |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
407 self.log_info('upgrade to version 6') |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
408 self.fix_version_5_tables() |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
409 |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
410 if version < 7: |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
411 self.log_info('upgrade to version 7') |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
412 self.fix_version_6_tables() |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
413 |
|
6806
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
414 if version < 8: |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
415 self.log_info('upgrade to version 8') |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
416 self.fix_version_7_tables() |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
417 |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
418 self.database_schema['version'] = self.current_db_version |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
419 self.db_version_updated = True |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
420 return 1 |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
421 |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
422 def fix_version_2_tables(self): |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
423 # Default (used by sqlite): NOOP |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
424 pass |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
425 |
|
2721
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
426 def fix_version_3_tables(self): |
|
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
427 # drop the shorter VARCHAR OTK column and add a new TEXT one |
|
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
428 for name in ('otk', 'session'): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
429 self.sql('DELETE FROM %ss' % name) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
430 self.sql('ALTER TABLE %ss DROP %s_value' % (name, name)) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
431 self.sql('ALTER TABLE %ss ADD %s_value TEXT' % (name, name)) |
|
2721
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
432 |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
433 def fix_version_4_tables(self): |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
434 # note this is an explicit call now |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
435 c = self.cursor |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
436 for cn, klass in self.classes.items(): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
437 c.execute('select id from _%s where __retired__<>0' % (cn,)) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
438 for (id,) in c.fetchall(): |
| 6965 | 439 c.execute('update _%s set __retired__=%s where id=%s' % ( |
| 440 cn, | |
| 441 self.arg, self.arg), (id, id)) | |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
442 |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
443 if klass.key: |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
444 self.add_class_key_required_unique_constraint(cn, klass.key) |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
445 |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
446 def fix_version_5_tables(self): |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
447 # Default (used by sqlite, postgres): NOOP |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
448 # mysql overrides this because it is missing |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
449 # _<class>_key_retired_idx index used to make |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
450 # sure that the key is unique if it was created |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
451 # as version 5. |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
452 pass |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
453 |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
454 def fix_version_6_tables(self): |
|
6599
39189dd94f2c
issue2551189 - increase size of words in full text index.
John Rouillard <rouilj@ieee.org>
parents:
6588
diff
changeset
|
455 # Default (used by nobody): NOOP |
|
39189dd94f2c
issue2551189 - increase size of words in full text index.
John Rouillard <rouilj@ieee.org>
parents:
6588
diff
changeset
|
456 # Each backend mysql, postgres, sqlite overrides this |
|
39189dd94f2c
issue2551189 - increase size of words in full text index.
John Rouillard <rouilj@ieee.org>
parents:
6588
diff
changeset
|
457 # You would think ALTER commands would be the same but nooo. |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
458 pass |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
459 |
|
6806
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
460 def fix_version_7_tables(self): |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
461 # Default (used by sqlite): NOOP |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
462 # Each backend mysql, postgres overrides this |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
463 # You would think ALTER commands would be the same but nooo. |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
464 pass |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
465 |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
466 def _convert_journal_tables(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
467 """Get current journal table contents, drop the table and re-create""" |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
468 c = self.cursor |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
469 cols = ','.join('nodeid date tag action params'.split()) |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
470 for klass in self.classes.values(): |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
471 # slurp and drop |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
472 sql = 'select %s from %s__journal order by date' % \ |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
473 (cols, klass.classname) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
474 c.execute(sql) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
475 contents = c.fetchall() |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
476 self.drop_journal_table_indexes(klass.classname) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
477 c.execute('drop table %s__journal' % klass.classname) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
478 |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
479 # re-create and re-populate |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
480 self.create_journal_table(klass) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
481 a = self.arg |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
482 sql = 'insert into %s__journal (%s) values (%s,%s,%s,%s,%s)' % ( |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
483 klass.classname, cols, a, a, a, a, a) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
484 for row in contents: |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
485 # no data conversion needed |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
486 self.cursor.execute(sql, row) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
487 |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
488 def _convert_string_properties(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
489 """Get current Class tables that contain String properties, and |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
490 convert the VARCHAR columns to TEXT""" |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
491 c = self.cursor |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
492 for klass in self.classes.values(): |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
493 # slurp and drop |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
494 cols, mls = self.determine_columns(list(klass.properties.items())) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
495 scols = ','.join([i[0] for i in cols]) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
496 sql = 'select id,%s from _%s' % (scols, klass.classname) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
497 c.execute(sql) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
498 contents = c.fetchall() |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
499 self.drop_class_table_indexes(klass.classname, klass.getkey()) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
500 c.execute('drop table _%s' % klass.classname) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
501 |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
502 # re-create and re-populate |
|
2424
74474ec41050
argh! backwards compat
Richard Jones <richard@users.sourceforge.net>
parents:
2416
diff
changeset
|
503 self.create_class_table(klass, create_sequence=0) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
504 a = ','.join([self.arg for i in range(len(cols)+1)]) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
505 sql = 'insert into _%s (id,%s) values (%s)' % (klass.classname, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
506 scols, a) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
507 for row in contents: |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
508 l = [] |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
509 for entry in row: |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
510 # mysql will already be a string - psql needs "help" |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
511 if entry is not None and not isinstance(entry, type('')): |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
512 entry = str(entry) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
513 l.append(entry) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
514 self.cursor.execute(sql, l) |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
515 |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
516 def refresh_database(self): |
|
1951
767ff2a03eee
more unit tests to improve coverage (up from 85% to 88% for anydbm! :)
Richard Jones <richard@users.sourceforge.net>
parents:
1926
diff
changeset
|
517 self.post_init() |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
518 |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
519 def reindex(self, classname=None, show_progress=False): |
|
2650
d68a444fcce3
roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents:
2649
diff
changeset
|
520 if classname: |
|
d68a444fcce3
roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents:
2649
diff
changeset
|
521 classes = [self.getclass(classname)] |
|
d68a444fcce3
roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents:
2649
diff
changeset
|
522 else: |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
523 classes = list(self.classes.values()) |
|
2650
d68a444fcce3
roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents:
2649
diff
changeset
|
524 for klass in classes: |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
525 if show_progress: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
526 for nodeid in support.Progress('Reindex %s' % klass.classname, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
527 klass.list()): |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
528 klass.index(nodeid) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
529 else: |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
530 for nodeid in klass.list(): |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
531 klass.index(nodeid) |
|
3295
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3239
diff
changeset
|
532 self.indexer.save_index() |
| 1165 | 533 |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
534 def checkpoint_data(self): |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
535 """Call if you need to commit the state of the database |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
536 so you can try to fix the error rather than rolling back |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
537 |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
538 Needed for postgres when importing data. |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
539 """ |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
540 pass |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
541 |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
542 def restore_connection_on_error(self): |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
543 """on a database error/exception recover the db connection |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
544 if left in an unusable state (e.g. postgres requires |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
545 a rollback). |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
546 """ |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
547 pass |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
548 |
|
5175
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
549 # Used here in the generic backend to determine if the database |
|
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
550 # supports 'DOUBLE PRECISION' for floating point numbers. |
|
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
551 implements_double_precision = True |
|
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
552 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
553 hyperdb_to_sql_datatypes = { |
| 6965 | 554 hyperdb.String : 'TEXT', # noqa: E203 |
| 555 hyperdb.Date : 'TIMESTAMP', # noqa: E203 | |
| 556 hyperdb.Link : 'INTEGER', # noqa: E203 | |
| 557 hyperdb.Interval : 'VARCHAR(255)', # noqa: E203 | |
| 558 hyperdb.Password : 'VARCHAR(255)', # noqa: E203 | |
| 559 hyperdb.Boolean : 'BOOLEAN', # noqa: E203 | |
| 560 hyperdb.Number : 'REAL', # noqa: E203 | |
| 561 hyperdb.Integer : 'INTEGER', # noqa: E203 | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
562 } |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
563 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
564 def hyperdb_to_sql_datatype(self, propclass, prop=None): |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
565 |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
566 datatype = self.hyperdb_to_sql_datatypes.get(propclass) |
|
5175
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
567 if self.implements_double_precision and prop and \ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
568 isinstance(prop, Number) and prop.use_double: |
|
5175
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
569 datatype = 'DOUBLE PRECISION' |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
570 if datatype: |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
571 return datatype |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
572 |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
573 for k, v in self.hyperdb_to_sql_datatypes.items(): |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
574 if issubclass(propclass, k): |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
575 return v |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
576 |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
577 raise ValueError('%r is not a hyperdb property class' % propclass) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
578 |
| 1165 | 579 def determine_columns(self, properties): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
580 """ Figure the column names and multilink properties from the spec |
| 1165 | 581 |
| 582 "properties" is a list of (name, prop) where prop may be an | |
| 583 instance of a hyperdb "type" _or_ a string repr of that type. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
584 """ |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
585 cols = [ |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
586 ('_actor', self.hyperdb_to_sql_datatype(hyperdb.Link)), |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
587 ('_activity', self.hyperdb_to_sql_datatype(hyperdb.Date)), |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
588 ('_creator', self.hyperdb_to_sql_datatype(hyperdb.Link)), |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
589 ('_creation', self.hyperdb_to_sql_datatype(hyperdb.Date)), |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
590 ] |
| 1165 | 591 mls = [] |
| 592 # add the multilinks separately | |
| 593 for col, prop in properties: | |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
594 # Computed props are not in the db |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
595 if prop.computed: |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
596 continue |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
597 |
| 1165 | 598 if isinstance(prop, Multilink): |
| 599 mls.append(col) | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
600 continue |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
601 |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
602 if isinstance(prop, type('')): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
603 raise ValueError("string property spec!") |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
604 # and prop.find('Multilink') != -1: |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
605 # mls.append(col) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
606 |
|
5175
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
607 datatype = self.hyperdb_to_sql_datatype(prop.__class__, prop) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
608 cols.append(('_'+col, datatype)) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
609 |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
610 # Intervals stored as two columns |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
611 if isinstance(prop, Interval): |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
612 cols.append(('__'+col+'_int__', 'BIGINT')) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
613 |
| 1165 | 614 cols.sort() |
| 615 return cols, mls | |
| 616 | |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
617 def update_class(self, spec, old_spec, force=0): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
618 """ Determine the differences between the current spec and the |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
619 database version of the spec, and update where necessary. |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
620 |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
621 If 'force' is true, update the database anyway. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
622 """ |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
623 new_spec = spec.schema() |
|
1515
a516bbb9896b
fixed rdbms table update detection logic [SF#703297]
Richard Jones <richard@users.sourceforge.net>
parents:
1500
diff
changeset
|
624 new_spec[1].sort() |
|
a516bbb9896b
fixed rdbms table update detection logic [SF#703297]
Richard Jones <richard@users.sourceforge.net>
parents:
1500
diff
changeset
|
625 old_spec[1].sort() |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
626 if not force and new_spec == old_spec: |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
627 # no changes |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
628 return 0 |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
629 |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
630 if not self.config.RDBMS_ALLOW_ALTER: |
|
6133
e7cb0147e6fe
Make format strings use named placeholders rather than %s/%r
John Rouillard <rouilj@ieee.org>
parents:
6118
diff
changeset
|
631 raise DatabaseError(_( |
| 6965 | 632 'ALTER operation disallowed: %(old)r -> %(new)r.' % { |
|
6133
e7cb0147e6fe
Make format strings use named placeholders rather than %s/%r
John Rouillard <rouilj@ieee.org>
parents:
6118
diff
changeset
|
633 'old': old_spec, 'new': new_spec})) |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
634 |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
635 logger = logging.getLogger('roundup.hyperdb.backend') |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
636 logger.info('update_class %s' % spec.classname) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
637 |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
638 logger.debug('old_spec %r' % (old_spec,)) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
639 logger.debug('new_spec %r' % (new_spec,)) |
| 1165 | 640 |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
641 # detect key prop change for potential index change |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
642 keyprop_changes = {} |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
643 if new_spec[0] != old_spec[0]: |
|
3554
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
644 if old_spec[0]: |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
645 keyprop_changes['remove'] = old_spec[0] |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
646 if new_spec[0]: |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
647 keyprop_changes['add'] = new_spec[0] |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
648 |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
649 # detect multilinks that have been removed, and drop their table |
|
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
650 old_has = {} |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
651 for name, prop in old_spec[1]: |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
652 old_has[name] = 1 |
|
6392
99455aeec1ae
Fix dropping of multilinks on schema change
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6332
diff
changeset
|
653 if name in spec.properties and not spec.properties[name].computed: |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
654 continue |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
655 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
656 if prop.find('Multilink to') != -1: |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
657 # first drop indexes. |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
658 self.drop_multilink_table_indexes(spec.classname, name) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
659 |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
660 # now the multilink table itself |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
661 sql = 'drop table %s_%s' % (spec.classname, name) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
662 else: |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
663 # if this is the key prop, drop the index first |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
664 if old_spec[0] == prop: |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
665 self.drop_class_table_key_index(spec.classname, name) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
666 del keyprop_changes['remove'] |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
667 |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
668 # drop the column |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
669 sql = 'alter table _%s drop column _%s' % (spec.classname, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
670 name) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
671 |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
672 self.sql(sql) |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
673 |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
674 # if we didn't remove the key prop just then, but the key prop has |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
675 # changed, we still need to remove the old index |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
676 if 'remove' in keyprop_changes: |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
677 self.drop_class_table_key_index(spec.classname, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
678 keyprop_changes['remove']) |
| 1165 | 679 |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
680 # add new columns |
|
2196
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
681 for propname, prop in new_spec[1]: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
682 if propname in old_has: |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
683 continue |
|
2196
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
684 prop = spec.properties[propname] |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
685 if isinstance(prop, Multilink): |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
686 self.create_multilink_table(spec, propname) |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
687 else: |
|
2727
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
688 # add the column |
|
5175
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
689 coltype = self.hyperdb_to_sql_datatype(prop.__class__, prop) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
690 sql = 'alter table _%s add column _%s %s' % ( |
|
2727
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
691 spec.classname, propname, coltype) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
692 self.sql(sql) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
693 |
|
2727
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
694 # extra Interval column |
|
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
695 if isinstance(prop, Interval): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
696 sql = 'alter table _%s add column __%s_int__ BIGINT' % ( |
|
2727
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
697 spec.classname, propname) |
|
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
698 self.sql(sql) |
|
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
699 |
|
2196
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
700 # if the new column is a key prop, we need an index! |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
701 if new_spec[0] == propname: |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
702 self.create_class_table_key_index(spec.classname, propname) |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
703 del keyprop_changes['add'] |
| 1165 | 704 |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
705 # if we didn't add the key prop just then, but the key prop has |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
706 # changed, we still need to add the new index |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
707 if 'add' in keyprop_changes: |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
708 self.create_class_table_key_index(spec.classname, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
709 keyprop_changes['add']) |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
710 |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
711 return 1 |
| 1165 | 712 |
|
3016
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
713 def determine_all_columns(self, spec): |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
714 """Figure out the columns from the spec and also add internal columns |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
715 |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
716 """ |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
717 cols, mls = self.determine_columns(list(spec.properties.items())) |
|
3082
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
718 |
|
3016
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
719 # add on our special columns |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
720 cols.append(('id', 'INTEGER PRIMARY KEY')) |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
721 cols.append(('__retired__', 'INTEGER DEFAULT 0')) |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
722 return cols, mls |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
723 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
724 def create_class_table(self, spec): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
725 """Create the class table for the given Class "spec". Creates the |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
726 indexes too.""" |
|
3016
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
727 cols, mls = self.determine_all_columns(spec) |
| 1165 | 728 |
| 729 # create the base table | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
730 scols = ','.join(['%s %s' % x for x in cols]) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
731 sql = 'create table _%s (%s)' % (spec.classname, scols) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
732 self.sql(sql) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
733 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
734 self.create_class_table_indexes(spec) |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
735 |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
736 return cols, mls |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
737 |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
738 def create_class_table_indexes(self, spec): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
739 """ create the class table for the given spec |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
740 """ |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
741 # create __retired__ index |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
742 index_sql2 = 'create index _%s_retired_idx on _%s(__retired__)' % ( |
|
1836
94e430ad4fdb
make the RDBMS common backend and the SQLite and MYsql backend create...
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1800
diff
changeset
|
743 spec.classname, spec.classname) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
744 self.sql(index_sql2) |
| 1165 | 745 |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
746 # create index for key property |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
747 if spec.key: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
748 index_sql3 = 'create index _%s_%s_idx on _%s(_%s)' % ( |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
749 spec.classname, spec.key, |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
750 spec.classname, spec.key) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
751 self.sql(index_sql3) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
752 |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
753 # and the unique index for key / retired(id) |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
754 self.add_class_key_required_unique_constraint(spec.classname, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
755 spec.key) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
756 |
|
2228
1d1362c54c94
Some doc / comment fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
2217
diff
changeset
|
757 # TODO: create indexes on (selected?) Link property columns, as |
|
1d1362c54c94
Some doc / comment fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
2217
diff
changeset
|
758 # they're more likely to be used for lookup |
|
1d1362c54c94
Some doc / comment fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
2217
diff
changeset
|
759 |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
760 def add_class_key_required_unique_constraint(self, cn, key): |
|
3969
905faf52a51f
fix mysql breakage in 1.4.2
Richard Jones <richard@users.sourceforge.net>
parents:
3967
diff
changeset
|
761 sql = '''create unique index _%s_key_retired_idx |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
762 on _%s(__retired__, _%s)''' % (cn, cn, key) |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
763 try: |
|
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
764 self.sql(sql) |
| 6002 | 765 except Exception: # nosec |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
766 # XXX catch e.g.: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
767 # _sqlite.DatabaseError: index _status_key_retired_idx |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
768 # already exists |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
769 pass |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
770 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
771 def drop_class_table_indexes(self, cn, key): |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
772 # drop the old table indexes first |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
773 l = ['_%s_id_idx' % cn, '_%s_retired_idx' % cn] |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
774 if key: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
775 l.append('_%s_%s_idx' % (cn, key)) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
776 |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
777 table_name = '_%s' % cn |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
778 for index_name in l: |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
779 if not self.sql_index_exists(table_name, index_name): |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
780 continue |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
781 index_sql = 'drop index '+index_name |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
782 self.sql(index_sql) |
| 1165 | 783 |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
784 def create_class_table_key_index(self, cn, key): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
785 """ create the class table for the given spec |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
786 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
787 sql = 'create index _%s_%s_idx on _%s(_%s)' % (cn, key, cn, key) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
788 self.sql(sql) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
789 |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
790 def drop_class_table_key_index(self, cn, key): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
791 table_name = '_%s' % cn |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
792 index_name = '_%s_%s_idx' % (cn, key) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
793 if self.sql_index_exists(table_name, index_name): |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
794 sql = 'drop index '+index_name |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
795 self.sql(sql) |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
796 |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
797 # and now the retired unique index too |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
798 index_name = '_%s_key_retired_idx' % cn |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
799 if self.sql_index_exists(table_name, index_name): |
|
3967
b1e81ad3fa6a
don't recompute the index name
Richard Jones <richard@users.sourceforge.net>
parents:
3963
diff
changeset
|
800 sql = 'drop index '+index_name |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
801 self.sql(sql) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
802 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
803 def create_journal_table(self, spec): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
804 """ create the journal table for a class given the spec and |
| 1165 | 805 already-determined cols |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
806 """ |
| 1165 | 807 # journal table |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
808 sql = """create table %s__journal ( |
|
3224
1fd11a9803bb
use backend datatype for journal timestamps.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3155
diff
changeset
|
809 nodeid integer, date %s, tag varchar(255), |
| 6965 | 810 action varchar(255), params text)""" % ( |
| 811 spec.classname, | |
| 812 self.hyperdb_to_sql_datatype(hyperdb.Date)) | |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
813 self.sql(sql) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
814 self.create_journal_table_indexes(spec) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
815 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
816 def create_journal_table_indexes(self, spec): |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
817 # index on nodeid |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
818 sql = 'create index %s_journ_idx on %s__journal(nodeid)' % ( |
|
1836
94e430ad4fdb
make the RDBMS common backend and the SQLite and MYsql backend create...
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1800
diff
changeset
|
819 spec.classname, spec.classname) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
820 self.sql(sql) |
| 1165 | 821 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
822 def drop_journal_table_indexes(self, classname): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
823 index_name = '%s_journ_idx' % classname |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
824 if not self.sql_index_exists('%s__journal' % classname, index_name): |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
825 return |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
826 index_sql = 'drop index '+index_name |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
827 self.sql(index_sql) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
828 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
829 def create_multilink_table(self, spec, ml): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
830 """ Create a multilink table for the "ml" property of the class |
| 1165 | 831 given by the spec |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
832 """ |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
833 # create the table |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
834 sql = 'create table %s_%s (linkid INTEGER, nodeid INTEGER)' % ( |
| 1165 | 835 spec.classname, ml) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
836 self.sql(sql) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
837 self.create_multilink_table_indexes(spec, ml) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
838 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
839 def create_multilink_table_indexes(self, spec, ml): |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
840 # create index on linkid |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
841 index_sql = 'create index %s_%s_l_idx on %s_%s(linkid)' % ( |
|
2100
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
842 spec.classname, ml, spec.classname, ml) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
843 self.sql(index_sql) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
844 |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
845 # create index on nodeid |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
846 index_sql = 'create index %s_%s_n_idx on %s_%s(nodeid)' % ( |
|
2100
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
847 spec.classname, ml, spec.classname, ml) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
848 self.sql(index_sql) |
| 1165 | 849 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
850 def drop_multilink_table_indexes(self, classname, ml): |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
851 l = [ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
852 '%s_%s_l_idx' % (classname, ml), |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
853 '%s_%s_n_idx' % (classname, ml) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
854 ] |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
855 table_name = '%s_%s' % (classname, ml) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
856 for index_name in l: |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
857 if not self.sql_index_exists(table_name, index_name): |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
858 continue |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
859 index_sql = 'drop index %s' % index_name |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
860 self.sql(index_sql) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
861 |
| 1165 | 862 def create_class(self, spec): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
863 """ Create a database table according to the given spec. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
864 """ |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
865 |
|
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
866 if not self.config.RDBMS_ALLOW_CREATE: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
867 raise DatabaseError(_('CREATE operation disallowed: "%s".' % |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
868 spec.classname)) |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
869 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
870 cols, mls = self.create_class_table(spec) |
|
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
871 self.create_journal_table(spec) |
| 1165 | 872 |
| 873 # now create the multilink tables | |
| 874 for ml in mls: | |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
875 self.create_multilink_table(spec, ml) |
| 1165 | 876 |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
877 def drop_class(self, cn, spec): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
878 """ Drop the given table from the database. |
| 1165 | 879 |
| 880 Drop the journal and multilink tables too. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
881 """ |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
882 |
|
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
883 if not self.config.RDBMS_ALLOW_DROP: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
884 raise DatabaseError(_('DROP operation disallowed: "%s".' % cn)) |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
885 |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
886 properties = spec[1] |
| 1165 | 887 # figure the multilinks |
| 888 mls = [] | |
|
3924
21d3d7eeea8c
assorted pyflakes fixes
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3906
diff
changeset
|
889 for propname, prop in properties: |
| 1165 | 890 if isinstance(prop, Multilink): |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
891 mls.append(propname) |
| 1165 | 892 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
893 # drop class table and indexes |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
894 self.drop_class_table_indexes(cn, spec[0]) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
895 |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
896 self.drop_class_table(cn) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
897 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
898 # drop journal table and indexes |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
899 self.drop_journal_table_indexes(cn) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
900 sql = 'drop table %s__journal' % cn |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
901 self.sql(sql) |
| 1165 | 902 |
| 903 for ml in mls: | |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
904 # drop multilink table and indexes |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
905 self.drop_multilink_table_indexes(cn, ml) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
906 sql = 'drop table %s_%s' % (spec.classname, ml) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
907 self.sql(sql) |
| 1165 | 908 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
909 def drop_class_table(self, cn): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
910 sql = 'drop table _%s' % cn |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
911 self.sql(sql) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
912 |
| 1165 | 913 # |
| 914 # Classes | |
| 915 # | |
| 916 def __getattr__(self, classname): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
917 """ A convenient way of calling self.getclass(classname). |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
918 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
919 if classname in self.classes: |
| 1165 | 920 return self.classes[classname] |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
921 raise AttributeError(classname) |
| 1165 | 922 |
| 923 def addclass(self, cl): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
924 """ Add a Class to the hyperdatabase. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
925 """ |
| 1165 | 926 cn = cl.classname |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
927 if cn in self.classes: |
| 6965 | 928 raise ValueError(_('Class "%s" already defined.' % cn)) |
| 1165 | 929 self.classes[cn] = cl |
| 930 | |
|
2076
2a4309450202
security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents:
2075
diff
changeset
|
931 # add default Edit and View permissions |
| 6965 | 932 self.security.addPermission( |
| 933 name="Create", klass=cn, | |
|
2649
1df7d4a41da4
Buncha stuff (sorry about the large checkin):
Richard Jones <richard@users.sourceforge.net>
parents:
2640
diff
changeset
|
934 description="User is allowed to create "+cn) |
| 6965 | 935 self.security.addPermission( |
| 936 name="Edit", klass=cn, | |
|
2076
2a4309450202
security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents:
2075
diff
changeset
|
937 description="User is allowed to edit "+cn) |
| 6965 | 938 self.security.addPermission( |
| 939 name="View", klass=cn, | |
|
2076
2a4309450202
security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents:
2075
diff
changeset
|
940 description="User is allowed to access "+cn) |
| 6965 | 941 self.security.addPermission( |
| 942 name="Retire", klass=cn, | |
|
4517
f8e85cf5f0fe
how odd, the Retire permission wasn't being registered;
Richard Jones <richard@users.sourceforge.net>
parents:
4490
diff
changeset
|
943 description="User is allowed to retire "+cn) |
|
2076
2a4309450202
security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents:
2075
diff
changeset
|
944 |
| 1165 | 945 def getclasses(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
946 """ Return a list of the names of all existing classes. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
947 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
948 return sorted(self.classes) |
| 1165 | 949 |
| 950 def getclass(self, classname): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
951 """Get the Class object representing a particular class. |
| 1165 | 952 |
| 953 If 'classname' is not a valid class name, a KeyError is raised. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
954 """ |
| 1165 | 955 try: |
| 956 return self.classes[classname] | |
| 957 except KeyError: | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
958 raise KeyError('There is no class called "%s"' % classname) |
| 1165 | 959 |
| 960 def clear(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
961 """Delete all database contents. |
| 1165 | 962 |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
963 Note: I don't commit here, which is different behaviour to the |
|
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
964 "nuke from orbit" behaviour in the dbs. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
965 """ |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
966 logging.getLogger('roundup.hyperdb.backend').info('clear') |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
967 for cn in self.classes: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
968 sql = 'delete from _%s' % cn |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
969 self.sql(sql) |
|
7007
05d81de0d92d
in clear() clear journal tables as well.
John Rouillard <rouilj@ieee.org>
parents:
6965
diff
changeset
|
970 sql = 'delete from %s__journal' % cn |
|
05d81de0d92d
in clear() clear journal tables as well.
John Rouillard <rouilj@ieee.org>
parents:
6965
diff
changeset
|
971 self.sql(sql) |
| 1165 | 972 |
| 973 # | |
| 974 # Nodes | |
| 975 # | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
976 |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
977 hyperdb_to_sql_value = { |
| 6965 | 978 hyperdb.String : str, # noqa: E203 |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
979 # fractional seconds by default |
| 6965 | 980 hyperdb.Date : lambda x: x.formal(sep=' ', # noqa: E203 |
| 981 sec='%06.3f'), | |
| 982 hyperdb.Link : int, # noqa: E203 | |
| 983 hyperdb.Interval : str, # noqa: E203 | |
| 984 hyperdb.Password : str, # noqa: E203 | |
| 985 hyperdb.Boolean : lambda x: x and 'TRUE' or 'FALSE', # noqa: E203 | |
| 986 hyperdb.Number : lambda x: x, # noqa: E203 | |
| 987 hyperdb.Integer : lambda x: x, # noqa: E203 | |
| 988 # used in journal marshalling | |
| 989 hyperdb.Multilink : lambda x: x, # noqa: E203 | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
990 } |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
991 |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
992 def to_sql_value(self, propklass): |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
993 |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
994 fn = self.hyperdb_to_sql_value.get(propklass) |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
995 if fn: |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
996 return fn |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
997 |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
998 for k, v in self.hyperdb_to_sql_value.items(): |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
999 if issubclass(propklass, k): |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1000 return v |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1001 |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1002 raise ValueError('%r is not a hyperdb property class' % propklass) |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1003 |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1004 def _cache_del(self, key): |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1005 del self.cache[key] |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1006 self.cache_lru.remove(key) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1007 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1008 def _cache_refresh(self, key): |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1009 self.cache_lru.remove(key) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1010 self.cache_lru.insert(0, key) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1011 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1012 def _cache_save(self, key, node): |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1013 self.cache[key] = node |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1014 # update the LRU |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1015 self.cache_lru.insert(0, key) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1016 if len(self.cache_lru) > self.cache_size: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1017 del self.cache[self.cache_lru.pop()] |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1018 |
| 1165 | 1019 def addnode(self, classname, nodeid, node): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1020 """ Add the specified node to its class's db. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1021 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1022 self.log_debug('addnode %s%s %r' % (classname, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1023 nodeid, node)) |
|
1528
96cd422532ef
bye bye gadfly - you served your purpose well [SF#701127]
Richard Jones <richard@users.sourceforge.net>
parents:
1523
diff
changeset
|
1024 |
|
96cd422532ef
bye bye gadfly - you served your purpose well [SF#701127]
Richard Jones <richard@users.sourceforge.net>
parents:
1523
diff
changeset
|
1025 # determine the column definitions and multilink tables |
| 1165 | 1026 cl = self.classes[classname] |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1027 cols, mls = self.determine_columns(list(cl.properties.items())) |
| 1165 | 1028 |
|
1189
23b8d1e87fe3
import fixes
Richard Jones <richard@users.sourceforge.net>
parents:
1187
diff
changeset
|
1029 # we'll be supplied these props if we're doing an import |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1030 values = node.copy() |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1031 if 'creator' not in values: |
|
1189
23b8d1e87fe3
import fixes
Richard Jones <richard@users.sourceforge.net>
parents:
1187
diff
changeset
|
1032 # add in the "calculated" properties (dupe so we don't affect |
|
23b8d1e87fe3
import fixes
Richard Jones <richard@users.sourceforge.net>
parents:
1187
diff
changeset
|
1033 # calling code's node assumptions) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1034 values['creation'] = values['activity'] = date.Date() |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1035 values['actor'] = values['creator'] = self.getuid() |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1036 |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1037 cl = self.classes[classname] |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1038 props = cl.getprops(protected=1) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1039 del props['id'] |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1040 |
| 1165 | 1041 # default the non-multilink columns |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1042 for col, prop in props.items(): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1043 if col not in values: |
|
1496
e6ac4e074acb
relaxed CVS importing (feature [SF#693277])
Richard Jones <richard@users.sourceforge.net>
parents:
1492
diff
changeset
|
1044 if isinstance(prop, Multilink): |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1045 values[col] = [] |
|
1496
e6ac4e074acb
relaxed CVS importing (feature [SF#693277])
Richard Jones <richard@users.sourceforge.net>
parents:
1492
diff
changeset
|
1046 else: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1047 values[col] = None |
| 1165 | 1048 |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1049 # clear this node out of the cache if it's in there |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1050 key = (classname, nodeid) |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1051 if key in self.cache: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1052 self._cache_del(key) |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1053 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1054 # figure the values to insert |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1055 vals = [] |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1056 for col, _dt in cols: |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1057 # this is somewhat dodgy.... |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1058 if col.endswith('_int__'): |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1059 # XXX eugh, this test suxxors |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1060 value = values[col[2:-6]] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1061 # this is an Interval special "int" column |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1062 if value is not None: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1063 vals.append(value.as_seconds()) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1064 else: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1065 vals.append(value) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1066 continue |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1067 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1068 prop = props[col[1:]] |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1069 value = values[col[1:]] |
|
2472
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
1070 if value is not None: |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1071 value = self.to_sql_value(prop.__class__)(value) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1072 vals.append(value) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1073 vals.append(nodeid) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1074 vals = tuple(vals) |
| 1165 | 1075 |
| 1076 # make sure the ordering is correct for column name -> column value | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1077 s = ','.join([self.arg for x in cols]) + ',%s' % self.arg |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1078 cols = ','.join([col for col, dt in cols]) + ',id' |
| 1165 | 1079 |
| 1080 # perform the inserts | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1081 sql = 'insert into _%s (%s) values (%s)' % (classname, cols, s) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1082 self.sql(sql, vals) |
| 1165 | 1083 |
| 1084 # insert the multilink rows | |
| 1085 for col in mls: | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1086 t = '%s_%s' % (classname, col) |
| 1165 | 1087 for entry in node[col]: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1088 sql = 'insert into %s (linkid, nodeid) values (%s,%s)' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1089 t, self.arg, self.arg) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1090 self.sql(sql, (entry, nodeid)) |
| 1165 | 1091 |
| 6965 | 1092 def setnode(self, classname, nodeid, values, multilink_changes=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1093 """ Change the specified node. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1094 """ |
| 6965 | 1095 if multilink_changes is None: multilink_changes = {} # noqa: E701 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1096 self.log_debug('setnode %s%s %r' % (classname, nodeid, values)) |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1097 |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1098 # clear this node out of the cache if it's in there |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1099 key = (classname, nodeid) |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1100 if key in self.cache: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1101 self._cache_del(key) |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1102 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1103 cl = self.classes[classname] |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1104 props = cl.getprops() |
| 1165 | 1105 |
| 1106 cols = [] | |
| 1107 mls = [] | |
| 1108 # add the multilinks separately | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1109 for col in values: |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1110 prop = props[col] |
| 1165 | 1111 if isinstance(prop, Multilink): |
| 1112 mls.append(col) | |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1113 elif isinstance(prop, Interval): |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1114 # Intervals store the seconds value too |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1115 cols.append(col) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1116 # extra leading '_' added by code below |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1117 cols.append('_' + col + '_int__') |
| 1165 | 1118 else: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1119 cols.append(col) |
| 1165 | 1120 cols.sort() |
| 1121 | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1122 # figure the values to insert |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1123 vals = [] |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1124 for col in cols: |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1125 if col.endswith('_int__'): |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1126 # XXX eugh, this test suxxors |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1127 # Intervals store the seconds value too |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1128 col = col[1:-6] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1129 prop = props[col] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1130 value = values[col] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1131 if value is None: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1132 vals.append(None) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1133 else: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1134 vals.append(value.as_seconds()) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1135 else: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1136 prop = props[col] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1137 value = values[col] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1138 if value is None: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1139 e = None |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1140 else: |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1141 e = self.to_sql_value(prop.__class__)(value) |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1142 vals.append(e) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1143 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1144 vals.append(int(nodeid)) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1145 vals = tuple(vals) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1146 |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1147 # if there's any updates to regular columns, do them |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1148 if cols: |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1149 # make sure the ordering is correct for column name -> column value |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1150 s = ','.join(['_%s=%s' % (x, self.arg) for x in cols]) |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1151 cols = ','.join(cols) |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1152 |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1153 # perform the update |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1154 sql = 'update _%s set %s where id=%s' % (classname, s, self.arg) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1155 self.sql(sql, vals) |
| 1165 | 1156 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1157 # we're probably coming from an import, not a change |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1158 if not multilink_changes: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1159 for name in mls: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1160 prop = props[name] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1161 value = values[name] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1162 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1163 t = '%s_%s' % (classname, name) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1164 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1165 # clear out previous values for this node |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1166 # XXX numeric ids |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1167 self.sql('delete from %s where nodeid=%s' % (t, self.arg), |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1168 (nodeid,)) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1169 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1170 # insert the values for this node |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1171 for entry in values[name]: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1172 sql = 'insert into %s (linkid, nodeid) values (%s,%s)' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1173 t, self.arg, self.arg) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1174 # XXX numeric ids |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1175 self.sql(sql, (entry, nodeid)) |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1176 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1177 # we have multilink changes to apply |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1178 for col, (add, remove) in multilink_changes.items(): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1179 tn = '%s_%s' % (classname, col) |
| 1165 | 1180 if add: |
| 6965 | 1181 sql = 'insert into %s (nodeid, linkid) values (%s,%s)' % ( |
| 1182 tn, self.arg, self.arg) | |
| 1165 | 1183 for addid in add: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1184 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1185 self.sql(sql, (int(nodeid), int(addid))) |
| 1165 | 1186 if remove: |
|
4039
f44ec45cedb4
Fuse multiple DELETE calls into one for multilinks.
Stefan Seefeld <stefan@seefeld.name>
parents:
4038
diff
changeset
|
1187 s = ','.join([self.arg]*len(remove)) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1188 sql = 'delete from %s where nodeid=%s and linkid in (%s)' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1189 tn, self.arg, s) |
|
4039
f44ec45cedb4
Fuse multiple DELETE calls into one for multilinks.
Stefan Seefeld <stefan@seefeld.name>
parents:
4038
diff
changeset
|
1190 # XXX numeric ids |
|
f44ec45cedb4
Fuse multiple DELETE calls into one for multilinks.
Stefan Seefeld <stefan@seefeld.name>
parents:
4038
diff
changeset
|
1191 self.sql(sql, [int(nodeid)] + remove) |
| 1165 | 1192 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1193 sql_to_hyperdb_value = { |
| 6965 | 1194 hyperdb.String : us2s, # noqa: E203 |
| 1195 hyperdb.Date : date_to_hyperdb_value, # noqa: E203 | |
| 1196 # hyperdb.Link : int, # XXX numeric ids | |
| 1197 hyperdb.Link : str, # noqa: E203 | |
| 1198 hyperdb.Interval : date.Interval, # noqa: E203 | |
| 1199 hyperdb.Password : lambda x: password.Password(encrypted=x), # noqa: E203 | |
| 1200 hyperdb.Boolean : _bool_cvt, # noqa: E203 | |
| 1201 hyperdb.Number : _num_cvt, # noqa: E203 | |
| 1202 hyperdb.Integer : int, # noqa: E203 | |
| 1203 # used in journal marshalling | |
| 1204 hyperdb.Multilink : lambda x: x, # noqa: E203 | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1205 } |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1206 |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1207 def to_hyperdb_value(self, propklass): |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1208 |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1209 fn = self.sql_to_hyperdb_value.get(propklass) |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1210 if fn: |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1211 return fn |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1212 |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1213 for k, v in self.sql_to_hyperdb_value.items(): |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1214 if issubclass(propklass, k): |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1215 return v |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1216 |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1217 raise ValueError('%r is not a hyperdb property class' % propklass) |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1218 |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1219 def _materialize_multilink(self, classname, nodeid, node, propname): |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1220 """ evaluation of single Multilink (lazy eval may have skipped this) |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1221 """ |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1222 if propname not in node: |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1223 prop = self.getclass(classname).properties[propname] |
| 6965 | 1224 tn = prop.table_name |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1225 lid = prop.linkid_name |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1226 nid = prop.nodeid_name |
| 6965 | 1227 w = '' |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1228 joi = '' |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1229 if prop.computed: |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1230 if isinstance(prop.rev_property, Link): |
| 6965 | 1231 w = ' and %s.__retired__=0' % tn |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1232 else: |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1233 tn2 = '_' + prop.classname |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1234 joi = ', %s' % tn2 |
| 6965 | 1235 w = ' and %s.%s=%s.id and %s.__retired__=0' % ( |
| 1236 tn, lid, tn2, tn2) | |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1237 cursor = self.sql_new_cursor(name='_materialize_multilink') |
| 6965 | 1238 sql = 'select %s from %s%s where %s=%s%s' % (lid, tn, joi, nid, |
| 1239 self.arg, w) | |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1240 self.sql(sql, (nodeid,), cursor) |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1241 # Reduce this to only the first row (the ID), this can save a |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1242 # lot of space for large query results (not using fetchall) |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1243 node[propname] = [str(x) for x in sorted(int(r[0]) for r in cursor)] |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1244 cursor.close() |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1245 |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1246 def _materialize_multilinks(self, classname, nodeid, node, props=None): |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1247 """ get all Multilinks of a node (lazy eval may have skipped this) |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1248 """ |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1249 cl = self.classes[classname] |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1250 props = props or [pn for (pn, p) in cl.properties.items() |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1251 if isinstance(p, Multilink)] |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1252 for propname in props: |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1253 if propname not in node: |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1254 self._materialize_multilink(classname, nodeid, node, propname) |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1255 |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1256 def getnode(self, classname, nodeid, fetch_multilinks=True): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1257 """ Get a node from the database. |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1258 For optimisation optionally we don't fetch multilinks |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1259 (lazy Multilinks). |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1260 But for internal database operations we need them. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1261 """ |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1262 # see if we have this node cached |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1263 key = (classname, nodeid) |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1264 if key in self.cache: |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1265 # push us back to the top of the LRU |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1266 self._cache_refresh(key) |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1267 if __debug__: |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1268 self.stats['cache_hits'] += 1 |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1269 # return the cached information |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1270 if fetch_multilinks: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1271 self._materialize_multilinks(classname, nodeid, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1272 self.cache[key]) |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1273 return self.cache[key] |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1274 |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1275 if __debug__: |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1276 self.stats['cache_misses'] += 1 |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1277 start_t = time.time() |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1278 |
| 1165 | 1279 # figure the columns we're fetching |
| 1280 cl = self.classes[classname] | |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1281 cols, mls = self.determine_columns(list(cl.properties.items())) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1282 scols = ','.join([col for col, dt in cols]) |
| 1165 | 1283 |
| 1284 # perform the basic property fetch | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1285 sql = 'select %s from _%s where id=%s' % (scols, classname, self.arg) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1286 self.sql(sql, (nodeid,)) |
| 1165 | 1287 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1288 values = self.sql_fetchone() |
| 1165 | 1289 if values is None: |
|
6197
2492e2e17371
Change text from no such <class> node <node id> to drop "node"
John Rouillard <rouilj@ieee.org>
parents:
6179
diff
changeset
|
1290 raise IndexError('no such %s %s' % (classname, nodeid)) |
| 1165 | 1291 |
| 1292 # make up the node | |
| 1293 node = {} | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1294 props = cl.getprops(protected=1) |
| 1165 | 1295 for col in range(len(cols)): |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1296 name = cols[col][0][1:] |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1297 if name.endswith('_int__'): |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1298 # XXX eugh, this test suxxors |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1299 # ignore the special Interval-as-seconds column |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1300 continue |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1301 value = values[col] |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1302 if value is not None: |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1303 value = self.to_hyperdb_value(props[name].__class__)(value) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1304 node[name] = value |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1305 |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1306 if fetch_multilinks and mls: |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1307 self._materialize_multilinks(classname, nodeid, node, mls) |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1308 |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1309 # save off in the cache |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1310 key = (classname, nodeid) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1311 self._cache_save(key, node) |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1312 |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1313 if __debug__: |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1314 self.stats['get_items'] += (time.time() - start_t) |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1315 |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1316 return node |
| 1165 | 1317 |
| 1318 def destroynode(self, classname, nodeid): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1319 """Remove a node from the database. Called exclusively by the |
| 1165 | 1320 destroy() method on Class. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1321 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1322 message = 'destroynode %s%s' % (classname, nodeid) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1323 logging.getLogger('roundup.hyperdb.backend').info(message) |
| 1165 | 1324 |
| 1325 # make sure the node exists | |
| 1326 if not self.hasnode(classname, nodeid): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1327 raise IndexError('%s has no node %s' % (classname, nodeid)) |
| 1165 | 1328 |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1329 # see if we have this node cached |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1330 if (classname, nodeid) in self.cache: |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1331 del self.cache[(classname, nodeid)] |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1332 |
| 1165 | 1333 # see if there's any obvious commit actions that we should get rid of |
| 1334 for entry in self.transactions[:]: | |
| 1335 if entry[1][:2] == (classname, nodeid): | |
| 1336 self.transactions.remove(entry) | |
| 1337 | |
| 1338 # now do the SQL | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1339 sql = 'delete from _%s where id=%s' % (classname, self.arg) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1340 self.sql(sql, (nodeid,)) |
| 1165 | 1341 |
| 1342 # remove from multilnks | |
| 1343 cl = self.getclass(classname) | |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1344 x, mls = self.determine_columns(list(cl.properties.items())) |
| 1165 | 1345 for col in mls: |
| 1346 # get the link ids | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1347 sql = 'delete from %s_%s where nodeid=%s' % (classname, col, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1348 self.arg) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
1349 self.sql(sql, (nodeid,)) |
| 1165 | 1350 |
| 1351 # remove journal entries | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1352 sql = 'delete from %s__journal where nodeid=%s' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1353 classname, self.arg) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1354 self.sql(sql, (nodeid,)) |
| 1165 | 1355 |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
1356 # cleanup any blob filestorage when we commit |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1357 self.transactions.append((FileStorage.destroy, (self, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1358 classname, nodeid))) |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
1359 |
| 1165 | 1360 def hasnode(self, classname, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1361 """ Determine if the database has a given node. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1362 """ |
|
5086
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1363 # nodeid (aka link) is sql type INTEGER. max positive value |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1364 # for INTEGER is 2^31-1 for Postgres and MySQL. The max |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1365 # positive value for SqLite is 2^63 -1, so arguably this check |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1366 # needs to adapt for the type of the RDBMS. For right now, |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1367 # choose lowest common denominator. |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1368 if int(nodeid) >= 2**31: |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1369 # value out of range return false |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1370 return 0 |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1371 |
| 4031 | 1372 # If this node is in the cache, then we do not need to go to |
| 1373 # the database. (We don't consider this an LRU hit, though.) | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1374 if (classname, nodeid) in self.cache: |
| 4031 | 1375 # Return 1, not True, to match the type of the result of |
| 1376 # the SQL operation below. | |
| 1377 return 1 | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1378 sql = 'select count(*) from _%s where id=%s' % (classname, self.arg) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1379 self.sql(sql, (nodeid,)) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1380 return int(self.cursor.fetchone()[0]) |
| 1165 | 1381 |
| 1382 def countnodes(self, classname): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1383 """ Count the number of nodes that exist for a particular Class. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1384 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1385 sql = 'select count(*) from _%s' % classname |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1386 self.sql(sql) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1387 return self.cursor.fetchone()[0] |
| 1165 | 1388 |
| 1389 def addjournal(self, classname, nodeid, action, params, creator=None, | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1390 creation=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1391 """ Journal the Action |
| 1165 | 1392 'action' may be: |
| 1393 | |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1394 'set' -- 'params' is a dictionary of property values |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1395 'create' -- 'params' is an empty dictionary as of |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1396 Wed Nov 06 11:38:43 2002 +0000 |
| 1165 | 1397 'link' or 'unlink' -- 'params' is (classname, nodeid, propname) |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1398 'retired' or 'restored' -- 'params' is None |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1399 |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1400 'creator' -- the user performing the action, which defaults to |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1401 the current user. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1402 """ |
| 1165 | 1403 # handle supply of the special journalling parameters (usually |
| 1404 # supplied on importing an existing database) | |
| 1405 if creator: | |
| 1406 journaltag = creator | |
| 1407 else: | |
|
1800
a3b1b1dcf639
Use getuid(), not figure_curuserid()
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
1794
diff
changeset
|
1408 journaltag = self.getuid() |
| 1165 | 1409 if creation: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1410 journaldate = creation |
| 1165 | 1411 else: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1412 journaldate = date.Date() |
| 1165 | 1413 |
| 1414 # create the journal entry | |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1415 cols = 'nodeid,date,tag,action,params' |
| 1165 | 1416 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1417 self.log_debug('addjournal %s%s %r %s %s %r' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1418 classname, nodeid, journaldate, journaltag, action, params)) |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1419 |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1420 # make the journalled data marshallable |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1421 if isinstance(params, type({})): |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1422 self._journal_marshal(params, classname) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1423 |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
1424 params = repr_export(params) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1425 |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1426 dc = self.to_sql_value(hyperdb.Date) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1427 journaldate = dc(journaldate) |
| 1165 | 1428 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1429 self.save_journal(classname, cols, nodeid, journaldate, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1430 journaltag, action, params) |
| 1165 | 1431 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1432 def setjournal(self, classname, nodeid, journal): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1433 """Set the journal to the "journal" list.""" |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1434 # clear out any existing entries |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1435 self.sql('delete from %s__journal where nodeid=%s' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1436 classname, self.arg), (nodeid,)) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1437 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1438 # create the journal entry |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1439 cols = 'nodeid,date,tag,action,params' |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1440 |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1441 dc = self.to_sql_value(hyperdb.Date) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1442 for nodeid, journaldate, journaltag, action, params in journal: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1443 self.log_debug('addjournal %s%s %r %s %s %r' % ( |
|
4059
ef0b4396888a
Enhance and simplify logging.
Stefan Seefeld <stefan@seefeld.name>
parents:
4044
diff
changeset
|
1444 classname, nodeid, journaldate, journaltag, action, |
|
ef0b4396888a
Enhance and simplify logging.
Stefan Seefeld <stefan@seefeld.name>
parents:
4044
diff
changeset
|
1445 params)) |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1446 |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1447 # make the journalled data marshallable |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1448 if isinstance(params, type({})): |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1449 self._journal_marshal(params, classname) |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
1450 params = repr_export(params) |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1451 |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1452 self.save_journal(classname, cols, nodeid, dc(journaldate), |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1453 journaltag, action, params) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1454 |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1455 def _journal_marshal(self, params, classname): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1456 """Convert the journal params values into safely repr'able and |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1457 eval'able values.""" |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1458 properties = self.getclass(classname).getprops() |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1459 for param, value in params.items(): |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1460 if not value: |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1461 continue |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1462 property = properties[param] |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1463 cvt = self.to_sql_value(property.__class__) |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1464 if isinstance(property, Password): |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1465 params[param] = cvt(value) |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1466 elif isinstance(property, Date): |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1467 params[param] = cvt(value) |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1468 elif isinstance(property, Interval): |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1469 params[param] = cvt(value) |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1470 elif isinstance(property, Boolean): |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1471 params[param] = cvt(value) |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1472 |
| 1165 | 1473 def getjournal(self, classname, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1474 """ get the journal for id |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1475 """ |
| 1165 | 1476 # make sure the node exists |
| 1477 if not self.hasnode(classname, nodeid): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1478 raise IndexError('%s has no node %s' % (classname, nodeid)) |
| 1165 | 1479 |
| 1480 cols = ','.join('nodeid date tag action params'.split()) | |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1481 journal = self.load_journal(classname, cols, nodeid) |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1482 |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1483 # now unmarshal the data |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1484 dc = self.to_hyperdb_value(hyperdb.Date) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1485 res = [] |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1486 properties = self.getclass(classname).getprops() |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1487 for nodeid, date_stamp, user, action, params in journal: |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
1488 params = eval_import(params) |
|
2248
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1489 if isinstance(params, type({})): |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1490 for param, value in params.items(): |
|
2248
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1491 if not value: |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1492 continue |
|
2731
7c472ed6babf
handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents:
2727
diff
changeset
|
1493 property = properties.get(param, None) |
|
7c472ed6babf
handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents:
2727
diff
changeset
|
1494 if property is None: |
|
7c472ed6babf
handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents:
2727
diff
changeset
|
1495 # deleted property |
|
7c472ed6babf
handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents:
2727
diff
changeset
|
1496 continue |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1497 cvt = self.to_hyperdb_value(property.__class__) |
|
2248
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1498 if isinstance(property, Password): |
|
4483
22bc0426e348
Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4480
diff
changeset
|
1499 params[param] = password.JournalPassword(value) |
|
2248
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1500 elif isinstance(property, Date): |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1501 params[param] = cvt(value) |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1502 elif isinstance(property, Interval): |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1503 params[param] = cvt(value) |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1504 elif isinstance(property, Boolean): |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1505 params[param] = cvt(value) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1506 # XXX numeric ids |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1507 res.append((str(nodeid), dc(date_stamp), user, action, params)) |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1508 return res |
| 1165 | 1509 |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1510 def save_journal(self, classname, cols, nodeid, journaldate, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1511 journaltag, action, params): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1512 """ Save the journal entry to the database |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1513 """ |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1514 entry = (nodeid, journaldate, journaltag, action, params) |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1515 |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1516 # do the insert |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1517 a = self.arg |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1518 sql = 'insert into %s__journal (%s) values (%s,%s,%s,%s,%s)' % ( |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1519 classname, cols, a, a, a, a, a) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1520 self.sql(sql, entry) |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1521 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1522 def load_journal(self, classname, cols, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1523 """ Load the journal from the database |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1524 """ |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1525 # now get the journal entries |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1526 sql = 'select %s from %s__journal where nodeid=%s order by date' % ( |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1527 cols, classname, self.arg) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1528 self.sql(sql, (nodeid,)) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1529 return self.cursor.fetchall() |
| 1165 | 1530 |
| 1531 def pack(self, pack_before): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1532 """ Delete all journal entries except "create" before 'pack_before'. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1533 """ |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1534 date_stamp = self.to_sql_value(Date)(pack_before) |
| 1165 | 1535 |
| 1536 # do the delete | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1537 for classname in self.classes: |
| 1165 | 1538 sql = "delete from %s__journal where date<%s and "\ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1539 "action<>'create'" % (classname, self.arg) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1540 self.sql(sql, (date_stamp,)) |
| 1165 | 1541 |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1542 def sql_commit(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1543 """ Actually commit to the database. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1544 """ |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1545 logging.getLogger('roundup.hyperdb.backend').info('commit') |
|
3687
ff9f4ca42454
Postgres backend allows transaction collisions to be ignored when...
Richard Jones <richard@users.sourceforge.net>
parents:
3685
diff
changeset
|
1546 |
| 1165 | 1547 self.conn.commit() |
| 1548 | |
|
2374
31cb1014300c
Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents:
2362
diff
changeset
|
1549 # open a new cursor for subsequent work |
|
31cb1014300c
Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents:
2362
diff
changeset
|
1550 self.cursor = self.conn.cursor() |
|
31cb1014300c
Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents:
2362
diff
changeset
|
1551 |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1552 def sql_new_cursor(self, conn=None, *args, **kw): |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1553 """ Create new cursor, this may need additional parameters for |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1554 performance optimization for different backends. |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1555 """ |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1556 if conn is None: |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1557 conn = self.conn |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1558 return conn.cursor() |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1559 |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1560 def commit(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1561 """ Commit the current transactions. |
| 1165 | 1562 |
| 1563 Save all data changed since the database was opened or since the | |
| 1564 last commit() or rollback(). | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1565 """ |
| 1165 | 1566 # commit the database |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1567 self.sql_commit() |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1568 |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1569 # session and otk are committed with the db but not the other |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1570 # way round |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1571 if self.Session: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1572 self.Session.commit() |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1573 if self.Otk: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1574 self.Otk.commit() |
| 1165 | 1575 |
| 1576 # now, do all the other transaction stuff | |
| 1577 for method, args in self.transactions: | |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1578 method(*args) |
| 1165 | 1579 |
|
3295
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3239
diff
changeset
|
1580 # save the indexer |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3239
diff
changeset
|
1581 self.indexer.save_index() |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3239
diff
changeset
|
1582 |
| 1165 | 1583 # clear out the transactions |
| 1584 self.transactions = [] | |
| 1585 | |
|
4448
2784c239e6c8
clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4430
diff
changeset
|
1586 # clear the cache: Don't carry over cached values from one |
|
2784c239e6c8
clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4430
diff
changeset
|
1587 # transaction to the next (there may be other changes from other |
|
2784c239e6c8
clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4430
diff
changeset
|
1588 # transactions) |
|
2784c239e6c8
clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4430
diff
changeset
|
1589 self.clearCache() |
|
2784c239e6c8
clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4430
diff
changeset
|
1590 |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1591 def sql_rollback(self): |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1592 self.conn.rollback() |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1593 |
| 1165 | 1594 def rollback(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1595 """ Reverse all actions from the current transaction. |
| 1165 | 1596 |
| 1597 Undo all the changes made since the database was opened or the last | |
| 1598 commit() or rollback() was performed. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1599 """ |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1600 logging.getLogger('roundup.hyperdb.backend').info('rollback') |
| 1165 | 1601 |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1602 self.sql_rollback() |
| 1165 | 1603 |
| 1604 # roll back "other" transaction stuff | |
| 1605 for method, args in self.transactions: | |
| 1606 # delete temporary files | |
| 1607 if method == self.doStoreFile: | |
| 1608 self.rollbackStoreFile(*args) | |
| 1609 self.transactions = [] | |
| 1610 | |
|
1492
2fc7d4a8c9e7
fixed sqlite rollback/caching bug [SF#689383]
Richard Jones <richard@users.sourceforge.net>
parents:
1484
diff
changeset
|
1611 # clear the cache |
|
2fc7d4a8c9e7
fixed sqlite rollback/caching bug [SF#689383]
Richard Jones <richard@users.sourceforge.net>
parents:
1484
diff
changeset
|
1612 self.clearCache() |
|
2fc7d4a8c9e7
fixed sqlite rollback/caching bug [SF#689383]
Richard Jones <richard@users.sourceforge.net>
parents:
1484
diff
changeset
|
1613 |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1614 def sql_close(self): |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1615 logging.getLogger('roundup.hyperdb.backend').info('close') |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1616 self.conn.close() |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1617 |
| 1165 | 1618 def close(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1619 """ Close off the connection. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1620 """ |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
2089
diff
changeset
|
1621 self.indexer.close() |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1622 self.sql_close() |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1623 if self.Session: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1624 self.Session.close() |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1625 self.Session = None |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1626 if self.Otk: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1627 self.Otk.close() |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1628 self.Otk = None |
| 1165 | 1629 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1630 |
| 1165 | 1631 # |
| 1632 # The base Class class | |
| 1633 # | |
| 1634 class Class(hyperdb.Class): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1635 """ The handle to a particular class of nodes in a hyperdatabase. |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1636 |
| 1165 | 1637 All methods except __repr__ and getnode must be implemented by a |
| 1638 concrete backend Class. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1639 """ |
|
4849
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
1640 # For many databases the LIKE operator ignores case. |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
1641 # Postgres and Oracle have an ILIKE operator to support this. |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
1642 # We define the default here, can be changed in derivative class |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
1643 case_insensitive_like = 'LIKE' |
| 1165 | 1644 |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
1645 # For some databases (mysql) the = operator for strings ignores case. |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
1646 # We define the default here, can be changed in derivative class |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
1647 case_sensitive_equal = '=' |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
1648 |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1649 # Some DBs order NULL values last. Set this variable in the backend |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1650 # for prepending an order by clause for each attribute that causes |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1651 # correct sort order for NULLs. Examples: |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1652 # order_by_null_values = '(%s is not NULL)' |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1653 # order_by_null_values = 'notnull(%s)' |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1654 # The format parameter is replaced with the attribute. |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1655 order_by_null_values = None |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1656 |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1657 # Assuming DBs can do subselects, overwrite if they cannot. |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1658 supports_subselects = True |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1659 |
| 1165 | 1660 def schema(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1661 """ A dumpable version of the schema that we can store in the |
| 1165 | 1662 database |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1663 """ |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1664 return (self.key, |
| 6965 | 1665 [(x, repr(y)) for x, y in |
| 1666 self.properties.items() if not y.computed]) | |
| 1165 | 1667 |
| 1668 def enableJournalling(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1669 """Turn journalling on for this class |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1670 """ |
| 1165 | 1671 self.do_journal = 1 |
| 1672 | |
| 1673 def disableJournalling(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1674 """Turn journalling off for this class |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1675 """ |
| 1165 | 1676 self.do_journal = 0 |
| 1677 | |
| 1678 # Editing nodes: | |
| 1679 def create(self, **propvalues): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1680 """ Create a new node of this class and return its id. |
| 1165 | 1681 |
| 1682 The keyword arguments in 'propvalues' map property names to values. | |
| 1683 | |
| 1684 The values of arguments must be acceptable for the types of their | |
| 1685 corresponding properties or a TypeError is raised. | |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1686 |
| 1165 | 1687 If this class has a key property, it must be present and its value |
| 1688 must not collide with other key strings or a ValueError is raised. | |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1689 |
| 1165 | 1690 Any other properties on this class that are missing from the |
| 1691 'propvalues' dictionary are set to None. | |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1692 |
| 1165 | 1693 If an id in a link or multilink property does not refer to a valid |
| 1694 node, an IndexError is raised. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1695 """ |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
1696 self.fireAuditors('create', None, propvalues) |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
1697 newid = self.create_inner(**propvalues) |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
1698 self.fireReactors('create', newid, None) |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
1699 return newid |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1700 |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
1701 def create_inner(self, **propvalues): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1702 """ Called by create, in-between the audit and react calls. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1703 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1704 if 'id' in propvalues: |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1705 raise KeyError('"id" is reserved') |
| 1165 | 1706 |
| 1707 if self.db.journaltag is None: | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1708 raise DatabaseError(_('Database open read-only')) |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1709 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1710 if ('creator' in propvalues or 'actor' in propvalues or |
| 6965 | 1711 'creation' in propvalues or 'activity' in propvalues): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1712 raise KeyError('"creator", "actor", "creation" and ' |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1713 '"activity" are reserved') |
| 1165 | 1714 |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1715 for p in propvalues: |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1716 prop = self.properties[p] |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1717 if prop.computed: |
| 6965 | 1718 raise KeyError('"%s" is a computed property' % p) |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1719 |
| 1165 | 1720 # new node's id |
| 1721 newid = self.db.newid(self.classname) | |
| 1722 | |
| 1723 # validate propvalues | |
| 5809 | 1724 num_re = re.compile(r'^\d+$') |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1725 for key, value in propvalues.items(): |
| 1165 | 1726 if key == self.key: |
| 1727 try: | |
| 1728 self.lookup(value) | |
| 1729 except KeyError: | |
| 1730 pass | |
| 1731 else: | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1732 raise ValueError('node with key "%s" exists' % value) |
| 1165 | 1733 |
| 1734 # try to handle this property | |
| 1735 try: | |
| 1736 prop = self.properties[key] | |
| 1737 except KeyError: | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1738 raise KeyError('"%s" has no property "%s"' % (self.classname, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1739 key)) |
| 1165 | 1740 |
| 1741 if value is not None and isinstance(prop, Link): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1742 if not isinstance(value, type('')): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1743 raise ValueError('link value must be String') |
| 1165 | 1744 link_class = self.properties[key].classname |
| 1745 # if it isn't a number, it's a key | |
| 1746 if not num_re.match(value): | |
| 1747 try: | |
| 1748 value = self.db.classes[link_class].lookup(value) | |
| 1749 except (TypeError, KeyError): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1750 raise IndexError('new property "%s": %s not a %s' % ( |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1751 key, value, link_class)) |
| 1165 | 1752 elif not self.db.getclass(link_class).hasnode(value): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1753 raise IndexError('%s has no node %s' % (link_class, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1754 value)) |
| 1165 | 1755 |
| 1756 # save off the value | |
| 1757 propvalues[key] = value | |
| 1758 | |
| 1759 # register the link with the newly linked node | |
| 1760 if self.do_journal and self.properties[key].do_journal: | |
| 1761 self.db.addjournal(link_class, value, 'link', | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1762 (self.classname, newid, key)) |
| 1165 | 1763 |
| 1764 elif isinstance(prop, Multilink): | |
|
3872
34128a809e22
Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3869
diff
changeset
|
1765 if value is None: |
|
34128a809e22
Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3869
diff
changeset
|
1766 value = [] |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1767 if not hasattr(value, '__iter__') or \ |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1768 isinstance(value, type('')): |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1769 raise TypeError('new property "%s" not an iterable of ids' |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1770 % key) |
| 1165 | 1771 # clean up and validate the list of links |
| 1772 link_class = self.properties[key].classname | |
| 1773 l = [] | |
| 1774 for entry in value: | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1775 if not isinstance(entry, type('')): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1776 raise ValueError('"%s" multilink value (%r) ' |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1777 'must contain Strings' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1778 key, value)) |
| 1165 | 1779 # if it isn't a number, it's a key |
| 1780 if not num_re.match(entry): | |
| 1781 try: | |
| 1782 entry = self.db.classes[link_class].lookup(entry) | |
| 1783 except (TypeError, KeyError): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1784 raise IndexError('new property "%s": %s not a %s' % ( |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1785 key, entry, self.properties[key].classname)) |
| 1165 | 1786 l.append(entry) |
| 1787 value = l | |
| 1788 propvalues[key] = value | |
| 1789 | |
| 1790 # handle additions | |
| 1791 for nodeid in value: | |
| 1792 if not self.db.getclass(link_class).hasnode(nodeid): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1793 raise IndexError('%s has no node %s' % (link_class, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1794 nodeid)) |
| 1165 | 1795 # register the link with the newly linked node |
| 1796 if self.do_journal and self.properties[key].do_journal: | |
| 1797 self.db.addjournal(link_class, nodeid, 'link', | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1798 (self.classname, newid, key)) |
| 1165 | 1799 |
| 1800 elif isinstance(prop, String): | |
| 6965 | 1801 if not isinstance(value, (str, unicode)): |
| 1802 raise TypeError('new property "%s" not a string' % key) | |
|
2892
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
1803 if prop.indexme: |
|
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
1804 self.db.indexer.add_text((self.classname, newid, key), |
| 6965 | 1805 value) |
| 1165 | 1806 |
| 1807 elif isinstance(prop, Password): | |
| 1808 if not isinstance(value, password.Password): | |
| 6965 | 1809 raise TypeError('new property "%s" not a Password' % key) |
| 1165 | 1810 |
| 1811 elif isinstance(prop, Date): | |
| 1812 if value is not None and not isinstance(value, date.Date): | |
| 6965 | 1813 raise TypeError('new property "%s" not a Date' % key) |
| 1165 | 1814 |
| 1815 elif isinstance(prop, Interval): | |
| 1816 if value is not None and not isinstance(value, date.Interval): | |
| 6965 | 1817 raise TypeError('new property "%s" not an Interval' % key) |
| 1165 | 1818 |
| 1819 elif value is not None and isinstance(prop, Number): | |
| 1820 try: | |
| 1821 float(value) | |
| 1822 except ValueError: | |
| 6965 | 1823 raise TypeError('new property "%s" not numeric' % key) |
| 1165 | 1824 |
|
5067
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1825 elif value is not None and isinstance(prop, Integer): |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1826 try: |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1827 int(value) |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1828 except ValueError: |
| 6965 | 1829 raise TypeError('new property "%s" not integer' % key) |
|
5067
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1830 |
| 1165 | 1831 elif value is not None and isinstance(prop, Boolean): |
| 1832 try: | |
| 1833 int(value) | |
| 1834 except ValueError: | |
| 6965 | 1835 raise TypeError('new property "%s" not boolean' % key) |
| 1165 | 1836 |
| 1837 # make sure there's data where there needs to be | |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1838 for key, prop in self.properties.items(): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1839 if key in propvalues: |
| 1165 | 1840 continue |
| 1841 if key == self.key: | |
| 6965 | 1842 raise ValueError('key property "%s" is required' % key) |
| 1165 | 1843 if isinstance(prop, Multilink): |
| 1844 propvalues[key] = [] | |
| 1845 else: | |
| 1846 propvalues[key] = None | |
| 1847 | |
| 1848 # done | |
| 1849 self.db.addnode(self.classname, newid, propvalues) | |
| 1850 if self.do_journal: | |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
1851 self.db.addjournal(self.classname, newid, ''"create", {}) |
| 1165 | 1852 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1853 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1854 return str(newid) |
| 1165 | 1855 |
| 1856 def get(self, nodeid, propname, default=_marker, cache=1): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1857 """Get the value of a property on an existing node of this class. |
| 1165 | 1858 |
| 1859 'nodeid' must be the id of an existing node of this class or an | |
| 1860 IndexError is raised. 'propname' must be the name of a property | |
| 1861 of this class or a KeyError is raised. | |
| 1862 | |
|
1780
d2801a2b0a77
Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents:
1751
diff
changeset
|
1863 'cache' exists for backwards compatibility, and is not used. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1864 """ |
| 1165 | 1865 if propname == 'id': |
| 1866 return nodeid | |
| 1867 | |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1868 # get the node's dict |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1869 d = self.db.getnode(self.classname, nodeid, fetch_multilinks=False) |
|
4476
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1870 # handle common case -- that property is in dict -- first |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1871 # if None and one of creator/creation actor/activity return None |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1872 if propname in d: |
| 6965 | 1873 r = d[propname] |
|
4476
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1874 # return copy of our list |
| 6965 | 1875 if isinstance(r, list): |
|
4476
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1876 return r[:] |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1877 if r is not None: |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1878 return r |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1879 elif propname in ('creation', 'activity', 'creator', 'actor'): |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1880 return r |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1881 |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1882 # propname not in d: |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1883 if propname == 'creation' or propname == 'activity': |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1884 return date.Date() |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1885 if propname == 'creator' or propname == 'actor': |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1886 return self.db.getuid() |
| 1165 | 1887 |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1888 # get the property (raises KeyError if invalid) |
| 1165 | 1889 prop = self.properties[propname] |
| 1890 | |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1891 # lazy evaluation of Multilink |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1892 if propname not in d and isinstance(prop, Multilink): |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1893 self.db._materialize_multilink(self.classname, nodeid, d, propname) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1894 |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1895 # handle there being no value in the table for the property |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1896 if propname not in d or d[propname] is None: |
|
2692
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
1897 if default is _marker: |
| 1165 | 1898 if isinstance(prop, Multilink): |
| 1899 return [] | |
| 1900 else: | |
| 1901 return None | |
| 1902 else: | |
| 1903 return default | |
| 1904 | |
| 1905 # don't pass our list to other code | |
| 1906 if isinstance(prop, Multilink): | |
| 1907 return d[propname][:] | |
| 1908 | |
| 1909 return d[propname] | |
| 1910 | |
| 1911 def set(self, nodeid, **propvalues): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1912 """Modify a property on an existing node of this class. |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1913 |
| 1165 | 1914 'nodeid' must be the id of an existing node of this class or an |
| 1915 IndexError is raised. | |
| 1916 | |
| 1917 Each key in 'propvalues' must be the name of a property of this | |
| 1918 class or a KeyError is raised. | |
| 1919 | |
| 1920 All values in 'propvalues' must be acceptable types for their | |
| 1921 corresponding properties or a TypeError is raised. | |
| 1922 | |
| 1923 If the value of the key property is set, it must not collide with | |
| 1924 other key strings or a ValueError is raised. | |
| 1925 | |
| 1926 If the value of a Link or Multilink property contains an invalid | |
| 1927 node id, a ValueError is raised. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1928 """ |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1929 self.fireAuditors('set', nodeid, propvalues) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1930 oldvalues = copy.deepcopy(self.db.getnode(self.classname, nodeid)) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1931 propvalues = self.set_inner(nodeid, **propvalues) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1932 self.fireReactors('set', nodeid, oldvalues) |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1933 return propvalues |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1934 |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1935 def set_inner(self, nodeid, **propvalues): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1936 """ Called by set, in-between the audit and react calls. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1937 """ |
| 1165 | 1938 if not propvalues: |
| 1939 return propvalues | |
| 1940 | |
| 6965 | 1941 if ('creator' in propvalues or 'actor' in propvalues or |
| 1942 'creation' in propvalues or 'activity' in propvalues): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1943 raise KeyError('"creator", "actor", "creation" and ' |
| 6965 | 1944 '"activity" are reserved') |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1945 |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1946 if 'id' in propvalues: |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1947 raise KeyError('"id" is reserved') |
| 1165 | 1948 |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1949 for p in propvalues: |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1950 prop = self.properties[p] |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1951 if prop.computed: |
| 6965 | 1952 raise KeyError('"%s" is a computed property' % p) |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1953 |
| 1165 | 1954 if self.db.journaltag is None: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1955 raise DatabaseError(_('Database open read-only')) |
| 1165 | 1956 |
| 1957 node = self.db.getnode(self.classname, nodeid) | |
| 1958 if self.is_retired(nodeid): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1959 raise IndexError('Requested item is retired') |
| 5809 | 1960 num_re = re.compile(r'^\d+$') |
| 1165 | 1961 |
|
3082
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
1962 # make a copy of the values dictionary - we'll modify the contents |
|
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
1963 propvalues = propvalues.copy() |
|
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
1964 |
| 1165 | 1965 # if the journal value is to be different, store it in here |
| 1966 journalvalues = {} | |
| 1967 | |
| 1968 # remember the add/remove stuff for multilinks, making it easier | |
| 1969 # for the Database layer to do its stuff | |
| 1970 multilink_changes = {} | |
| 1971 | |
|
5112
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
1972 # omit quiet properties from history/changelog |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
1973 quiet_props = [] |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
1974 |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1975 for propname, value in list(propvalues.items()): |
| 1165 | 1976 # check to make sure we're not duplicating an existing key |
| 1977 if propname == self.key and node[propname] != value: | |
| 1978 try: | |
| 1979 self.lookup(value) | |
| 1980 except KeyError: | |
| 1981 pass | |
| 1982 else: | |
| 6965 | 1983 raise ValueError('node with key "%s" exists' % value) |
| 1165 | 1984 |
| 1985 # this will raise the KeyError if the property isn't valid | |
| 1986 # ... we don't use getprops() here because we only care about | |
| 1987 # the writeable properties. | |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1988 try: |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1989 prop = self.properties[propname] |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1990 except KeyError: |
| 6965 | 1991 raise KeyError('"%s" has no property named "%s"' % ( |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1992 self.classname, propname)) |
| 1165 | 1993 |
| 1994 # if the value's the same as the existing value, no sense in | |
| 1995 # doing anything | |
|
1304
61ad556cfc8d
working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents:
1295
diff
changeset
|
1996 current = node.get(propname, None) |
|
61ad556cfc8d
working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents:
1295
diff
changeset
|
1997 if value == current: |
| 1165 | 1998 del propvalues[propname] |
| 1999 continue | |
|
1304
61ad556cfc8d
working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents:
1295
diff
changeset
|
2000 journalvalues[propname] = current |
| 1165 | 2001 |
| 2002 # do stuff based on the prop type | |
| 2003 if isinstance(prop, Link): | |
| 2004 link_class = prop.classname | |
| 2005 # if it isn't a number, it's a key | |
| 6965 | 2006 if value is not None and not isinstance(value, str): |
| 2007 raise ValueError('property "%s" link value be a string' % ( | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2008 propname)) |
| 6965 | 2009 if isinstance(value, str) and not num_re.match(value): |
| 1165 | 2010 try: |
| 2011 value = self.db.classes[link_class].lookup(value) | |
| 2012 except (TypeError, KeyError): | |
| 6965 | 2013 raise IndexError('new property "%s": %s not a %s' % ( |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2014 propname, value, prop.classname)) |
| 1165 | 2015 |
| 2016 if (value is not None and | |
| 2017 not self.db.getclass(link_class).hasnode(value)): | |
| 6965 | 2018 raise IndexError('%s has no node %s' % (link_class, |
| 2019 value)) | |
| 1165 | 2020 |
| 2021 if self.do_journal and prop.do_journal: | |
| 2022 # register the unlink with the old linked node | |
| 2023 if node[propname] is not None: | |
| 6965 | 2024 self.db.addjournal( |
| 2025 link_class, node[propname], | |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
2026 ''"unlink", (self.classname, nodeid, propname)) |
| 1165 | 2027 |
| 2028 # register the link with the newly linked node | |
| 2029 if value is not None: | |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
2030 self.db.addjournal(link_class, value, ''"link", |
| 6965 | 2031 (self.classname, nodeid, propname)) |
| 1165 | 2032 |
| 2033 elif isinstance(prop, Multilink): | |
|
3872
34128a809e22
Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3869
diff
changeset
|
2034 if value is None: |
|
34128a809e22
Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3869
diff
changeset
|
2035 value = [] |
| 6965 | 2036 if not hasattr(value, '__iter__') or isinstance(value, str): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2037 raise TypeError('new property "%s" not an iterable of' |
| 6965 | 2038 ' ids' % propname) |
| 1165 | 2039 link_class = self.properties[propname].classname |
| 2040 l = [] | |
| 2041 for entry in value: | |
| 2042 # if it isn't a number, it's a key | |
| 6965 | 2043 if not isinstance(entry, str): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2044 raise ValueError('new property "%s" link value ' |
| 6965 | 2045 'must be a string' % propname) |
| 1165 | 2046 if not num_re.match(entry): |
| 2047 try: | |
| 2048 entry = self.db.classes[link_class].lookup(entry) | |
| 2049 except (TypeError, KeyError): | |
| 6965 | 2050 raise IndexError( |
| 2051 'new property "%s": %s not a %s' % ( | |
| 2052 propname, entry, | |
| 2053 self.properties[propname].classname)) | |
| 1165 | 2054 l.append(entry) |
| 2055 value = l | |
| 2056 propvalues[propname] = value | |
| 2057 | |
| 2058 # figure the journal entry for this property | |
| 2059 add = [] | |
| 2060 remove = [] | |
| 2061 | |
| 2062 # handle removals | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2063 if propname in node: |
| 1165 | 2064 l = node[propname] |
| 2065 else: | |
| 2066 l = [] | |
| 2067 for id in l[:]: | |
| 2068 if id in value: | |
| 2069 continue | |
| 2070 # register the unlink with the old linked node | |
| 2071 if self.do_journal and self.properties[propname].do_journal: | |
| 2072 self.db.addjournal(link_class, id, 'unlink', | |
| 6965 | 2073 (self.classname, nodeid, propname)) |
| 1165 | 2074 l.remove(id) |
| 2075 remove.append(id) | |
| 2076 | |
| 2077 # handle additions | |
| 2078 for id in value: | |
|
4095
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2079 if id in l: |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2080 continue |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2081 # We can safely check this condition after |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2082 # checking that this is an addition to the |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2083 # multilink since the condition was checked for |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2084 # existing entries at the point they were added to |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2085 # the multilink. Since the hasnode call will |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2086 # result in a SQL query, it is more efficient to |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2087 # avoid the check if possible. |
| 1165 | 2088 if not self.db.getclass(link_class).hasnode(id): |
| 6965 | 2089 raise IndexError('%s has no node %s' % (link_class, |
| 2090 id)) | |
| 1165 | 2091 # register the link with the newly linked node |
| 2092 if self.do_journal and self.properties[propname].do_journal: | |
| 2093 self.db.addjournal(link_class, id, 'link', | |
| 6965 | 2094 (self.classname, nodeid, propname)) |
| 1165 | 2095 l.append(id) |
| 2096 add.append(id) | |
| 2097 | |
| 2098 # figure the journal entry | |
| 2099 l = [] | |
| 2100 if add: | |
| 2101 l.append(('+', add)) | |
| 2102 if remove: | |
| 2103 l.append(('-', remove)) | |
| 2104 multilink_changes[propname] = (add, remove) | |
| 2105 if l: | |
| 2106 journalvalues[propname] = tuple(l) | |
| 2107 | |
| 2108 elif isinstance(prop, String): | |
| 6965 | 2109 if value is not None and not isinstance(value, (str, unicode)): |
| 2110 raise TypeError( | |
| 2111 'new property "%s" not a string' % propname) | |
|
2892
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
2112 if prop.indexme: |
| 6965 | 2113 if value is None: value = '' # noqa: E701 |
| 2114 self.db.indexer.add_text( | |
| 2115 (self.classname, nodeid, propname), value) | |
| 1165 | 2116 |
| 2117 elif isinstance(prop, Password): | |
| 2118 if not isinstance(value, password.Password): | |
| 6965 | 2119 raise TypeError( |
| 2120 'new property "%s" not a Password' % propname) | |
| 1165 | 2121 propvalues[propname] = value |
|
4483
22bc0426e348
Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4480
diff
changeset
|
2122 journalvalues[propname] = \ |
|
22bc0426e348
Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4480
diff
changeset
|
2123 current and password.JournalPassword(current) |
| 1165 | 2124 |
| 2125 elif value is not None and isinstance(prop, Date): | |
| 2126 if not isinstance(value, date.Date): | |
| 6965 | 2127 raise TypeError('new property "%s" not a Date' % propname) |
| 1165 | 2128 propvalues[propname] = value |
| 2129 | |
| 2130 elif value is not None and isinstance(prop, Interval): | |
| 2131 if not isinstance(value, date.Interval): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2132 raise TypeError('new property "%s" not an ' |
| 6965 | 2133 'Interval' % propname) |
| 1165 | 2134 propvalues[propname] = value |
| 2135 | |
| 2136 elif value is not None and isinstance(prop, Number): | |
| 2137 try: | |
| 2138 float(value) | |
| 2139 except ValueError: | |
| 6965 | 2140 raise TypeError('new property "%s" not numeric' % propname) |
| 1165 | 2141 |
|
5067
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2142 elif value is not None and isinstance(prop, Integer): |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2143 try: |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2144 int(value) |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2145 except ValueError: |
| 6965 | 2146 raise TypeError('new property "%s" not integer' % propname) |
|
5067
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2147 |
| 1165 | 2148 elif value is not None and isinstance(prop, Boolean): |
| 2149 try: | |
| 2150 int(value) | |
| 2151 except ValueError: | |
| 6965 | 2152 raise TypeError('new property "%s" not boolean' % propname) |
| 1165 | 2153 |
|
5112
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2154 # record quiet properties to omit from history/changelog |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2155 if prop.quiet: |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2156 quiet_props.append(propname) |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2157 |
| 1165 | 2158 # nothing to do? |
| 2159 if not propvalues: | |
| 2160 return propvalues | |
| 2161 | |
|
3082
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
2162 # update the activity time |
|
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
2163 propvalues['activity'] = date.Date() |
|
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
2164 propvalues['actor'] = self.db.getuid() |
|
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
2165 |
|
3328
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2166 # do the set |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
2167 self.db.setnode(self.classname, nodeid, propvalues, multilink_changes) |
| 1165 | 2168 |
|
3328
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2169 # remove the activity props now they're handled |
|
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2170 del propvalues['activity'] |
|
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2171 del propvalues['actor'] |
|
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2172 |
|
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2173 # journal the set |
| 1165 | 2174 if self.do_journal: |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
2175 self.db.addjournal(self.classname, nodeid, ''"set", journalvalues) |
| 1165 | 2176 |
|
5112
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2177 # remove quiet properties from output |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2178 for propname in quiet_props: |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2179 if propname in propvalues: |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2180 del propvalues[propname] |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2181 |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
2182 return propvalues |
| 1165 | 2183 |
| 2184 def retire(self, nodeid): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2185 """Retire a node. |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2186 |
| 1165 | 2187 The properties on the node remain available from the get() method, |
| 2188 and the node's id is never reused. | |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2189 |
| 1165 | 2190 Retired nodes are not returned by the find(), list(), or lookup() |
| 2191 methods, and other nodes may reuse the values of their key properties. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2192 """ |
| 1165 | 2193 if self.db.journaltag is None: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2194 raise DatabaseError(_('Database open read-only')) |
| 1165 | 2195 |
|
1345
618aa9c37d65
fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents:
1333
diff
changeset
|
2196 self.fireAuditors('retire', nodeid, None) |
|
618aa9c37d65
fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents:
1333
diff
changeset
|
2197 |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2198 # use the arg for __retired__ to cope with any odd database type |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2199 # conversion (hello, sqlite) |
| 6965 | 2200 sql = 'update _%s set __retired__=%s where id=%s' % ( |
| 2201 self.classname, self.db.arg, self.db.arg) | |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2202 self.db.sql(sql, (nodeid, nodeid)) |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2203 if self.do_journal: |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
2204 self.db.addjournal(self.classname, nodeid, ''"retired", None) |
| 1165 | 2205 |
|
1345
618aa9c37d65
fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents:
1333
diff
changeset
|
2206 self.fireReactors('retire', nodeid, None) |
|
618aa9c37d65
fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents:
1333
diff
changeset
|
2207 |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2208 def restore(self, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2209 """Restore a retired node. |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2210 |
|
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2211 Make node available for all operations like it was before retirement. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2212 """ |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2213 if self.db.journaltag is None: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2214 raise DatabaseError(_('Database open read-only')) |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2215 |
|
1523
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2216 node = self.db.getnode(self.classname, nodeid) |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2217 # check if key property was overrided |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2218 key = self.getkey() |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2219 try: |
| 6965 | 2220 self.lookup(node[key]) |
|
1523
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2221 except KeyError: |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2222 pass |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2223 else: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2224 raise KeyError("Key property (%s) of retired node clashes " |
| 6965 | 2225 "with existing one (%s)" % (key, node[key])) |
|
1523
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2226 |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2227 self.fireAuditors('restore', nodeid, None) |
|
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2228 # use the arg for __retired__ to cope with any odd database type |
|
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2229 # conversion (hello, sqlite) |
| 6965 | 2230 sql = 'update _%s set __retired__=%s where id=%s' % ( |
| 2231 self.classname, self.db.arg, self.db.arg) | |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
2232 self.db.sql(sql, (0, nodeid)) |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2233 if self.do_journal: |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
2234 self.db.addjournal(self.classname, nodeid, ''"restored", None) |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2235 |
|
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2236 self.fireReactors('restore', nodeid, None) |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2237 |
| 1165 | 2238 def is_retired(self, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2239 """Return true if the node is rerired |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2240 """ |
| 6965 | 2241 sql = 'select __retired__ from _%s where id=%s' % (self.classname, |
| 2242 self.db.arg) | |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
2243 self.db.sql(sql, (nodeid,)) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2244 return int(self.db.sql_fetchone()[0]) > 0 |
| 1165 | 2245 |
| 2246 def destroy(self, nodeid): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2247 """Destroy a node. |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2248 |
| 1165 | 2249 WARNING: this method should never be used except in extremely rare |
| 2250 situations where there could never be links to the node being | |
| 2251 deleted | |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
2252 |
| 1165 | 2253 WARNING: use retire() instead |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
2254 |
| 1165 | 2255 WARNING: the properties of this node will not be available ever again |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
2256 |
| 1165 | 2257 WARNING: really, use retire() instead |
| 2258 | |
| 2259 Well, I think that's enough warnings. This method exists mostly to | |
| 2260 support the session storage of the cgi interface. | |
| 2261 | |
| 2262 The node is completely removed from the hyperdb, including all journal | |
| 2263 entries. It will no longer be available, and will generally break code | |
| 2264 if there are any references to the node. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2265 """ |
| 1165 | 2266 if self.db.journaltag is None: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2267 raise DatabaseError(_('Database open read-only')) |
| 1165 | 2268 self.db.destroynode(self.classname, nodeid) |
| 2269 | |
| 2270 # Locating nodes: | |
| 2271 def hasnode(self, nodeid): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2272 """Determine if the given nodeid actually exists |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2273 """ |
| 1165 | 2274 return self.db.hasnode(self.classname, nodeid) |
| 2275 | |
| 2276 def setkey(self, propname): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2277 """Select a String property of this class to be the key property. |
| 1165 | 2278 |
| 2279 'propname' must be the name of a String property of this class or | |
| 2280 None, or a TypeError is raised. The values of the key property on | |
| 2281 all existing nodes must be unique or a ValueError is raised. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2282 """ |
| 1165 | 2283 prop = self.getprops()[propname] |
| 2284 if not isinstance(prop, String): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2285 raise TypeError('key properties must be String') |
| 1165 | 2286 self.key = propname |
| 2287 | |
| 2288 def getkey(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2289 """Return the name of the key property for this class or None.""" |
| 1165 | 2290 return self.key |
| 2291 | |
| 2292 def lookup(self, keyvalue): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2293 """Locate a particular node by its key property and return its id. |
| 1165 | 2294 |
| 2295 If this class has no key property, a TypeError is raised. If the | |
| 2296 'keyvalue' matches one of the values for the key property among | |
| 2297 the nodes in this class, the matching node's id is returned; | |
| 2298 otherwise a KeyError is raised. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2299 """ |
| 1165 | 2300 if not self.key: |
| 6965 | 2301 raise TypeError('No key property set for class %s' % |
| 2302 self.classname) | |
| 1165 | 2303 |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2304 # use the arg to handle any odd database type conversion (hello, |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2305 # sqlite) |
| 6965 | 2306 sql = "select id from _%s where _%s=%s and __retired__=%s" % ( |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2307 self.classname, self.key, self.db.arg, self.db.arg) |
|
3977
732a37da3a10
Fix for postgres 8.3 compatibility (and bug) (patch [SF#2030479])
Richard Jones <richard@users.sourceforge.net>
parents:
3969
diff
changeset
|
2308 self.db.sql(sql, (str(keyvalue), 0)) |
| 1165 | 2309 |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
2310 # see if there was a result that's not retired |
|
1203
735adcbfc665
fix to SQL lookup() and retirement
Richard Jones <richard@users.sourceforge.net>
parents:
1195
diff
changeset
|
2311 row = self.db.sql_fetchone() |
|
735adcbfc665
fix to SQL lookup() and retirement
Richard Jones <richard@users.sourceforge.net>
parents:
1195
diff
changeset
|
2312 if not row: |
| 6965 | 2313 raise KeyError('No key (%s) value "%s" for "%s"' % ( |
| 2314 self.key, keyvalue, self.classname)) | |
| 1165 | 2315 |
| 2316 # return the id | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2317 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2318 return str(row[0]) |
| 1165 | 2319 |
| 2320 def find(self, **propspec): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2321 """Get the ids of nodes in this class which link to the given nodes. |
| 1165 | 2322 |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2323 'propspec' consists of keyword args propname=nodeid or |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2324 propname={nodeid:1, } |
| 1165 | 2325 'propname' must be the name of a property in this class, or a |
|
1912
2b0ab61db194
fixes for [SF#818339]
Richard Jones <richard@users.sourceforge.net>
parents:
1911
diff
changeset
|
2326 KeyError is raised. That property must be a Link or |
|
2b0ab61db194
fixes for [SF#818339]
Richard Jones <richard@users.sourceforge.net>
parents:
1911
diff
changeset
|
2327 Multilink property, or a TypeError is raised. |
| 1165 | 2328 |
|
3239
440f0a6a2e3c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3224
diff
changeset
|
2329 Any node in this class whose 'propname' property links to any of |
|
440f0a6a2e3c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3224
diff
changeset
|
2330 the nodeids will be returned. Examples:: |
|
440f0a6a2e3c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3224
diff
changeset
|
2331 |
|
440f0a6a2e3c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3224
diff
changeset
|
2332 db.issue.find(messages='1') |
| 1165 | 2333 db.issue.find(messages={'1':1,'3':1}, files={'7':1}) |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2334 """ |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2335 # shortcut |
| 1165 | 2336 if not propspec: |
| 2337 return [] | |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2338 |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2339 # validate the args |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2340 props = self.getprops() |
|
7038
f524ddc27af8
replace for a,b in x.items() with for a in x.keys()
John Rouillard <rouilj@ieee.org>
parents:
7007
diff
changeset
|
2341 for propname in propspec.keys(): |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2342 # check the prop is OK |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2343 prop = props[propname] |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2344 if not isinstance(prop, Link) and not isinstance(prop, Multilink): |
| 6965 | 2345 raise TypeError("'%s' not a Link/Multilink property" % |
| 2346 propname) | |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2347 |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2348 # first, links |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2349 a = self.db.arg |
|
2733
ef396596a24e
more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents:
2731
diff
changeset
|
2350 allvalues = () |
|
ef396596a24e
more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents:
2731
diff
changeset
|
2351 sql = [] |
|
1988
5660b89f8903
more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents:
1986
diff
changeset
|
2352 where = [] |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
2353 for prop, values in propspec.items(): |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2354 if not isinstance(props[prop], hyperdb.Link): |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2355 continue |
| 6965 | 2356 if isinstance(values, dict) and len(values) == 1: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2357 values = list(values)[0] |
| 6965 | 2358 if isinstance(values, str): |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2359 allvalues += (values,) |
| 6965 | 2360 where.append('_%s = %s' % (prop, a)) |
|
1563
e2a8ce4d2317
Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents:
1557
diff
changeset
|
2361 elif values is None: |
| 6965 | 2362 where.append('_%s is NULL' % prop) |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2363 else: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2364 values = list(values) |
|
2494
ea7fb2f416db
fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents:
2472
diff
changeset
|
2365 s = '' |
|
ea7fb2f416db
fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents:
2472
diff
changeset
|
2366 if None in values: |
|
ea7fb2f416db
fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents:
2472
diff
changeset
|
2367 values.remove(None) |
| 6965 | 2368 s = '_%s is NULL or ' % prop |
|
2494
ea7fb2f416db
fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents:
2472
diff
changeset
|
2369 allvalues += tuple(values) |
| 6965 | 2370 s += '_%s in (%s)' % (prop, ','.join([a]*len(values))) |
| 2371 where.append('(' + s + ')') | |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2372 if where: |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2373 allvalues = (0, ) + allvalues |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2374 sql.append("""select id from _%s where __retired__=%s |
| 6965 | 2375 and %s""" % (self.classname, a, ' and '.join(where))) |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2376 |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2377 # now multilinks |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
2378 for prop, values in propspec.items(): |
|
6151
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2379 p = props[prop] |
|
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2380 if not isinstance(p, hyperdb.Multilink): |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2381 continue |
|
1988
5660b89f8903
more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents:
1986
diff
changeset
|
2382 if not values: |
|
5660b89f8903
more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents:
1986
diff
changeset
|
2383 continue |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2384 allvalues += (0, ) |
|
6151
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2385 tn = p.table_name |
|
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2386 ln = p.linkid_name |
|
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2387 nn = p.nodeid_name |
|
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2388 cn = '_' + self.classname |
|
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2389 ret = '' |
|
6155
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2390 dis = '' |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2391 ord = '' |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2392 if p.rev_property: |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2393 if isinstance(p.rev_property, Link): |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2394 ret = 'and %s.__retired__=%s ' % (tn, a) |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2395 allvalues += (0, ) |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2396 dis = 'distinct ' |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2397 ord = ' order by %s.id' % cn |
| 6965 | 2398 if isinstance(values, str): |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2399 allvalues += (values,) |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2400 s = a |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2401 else: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2402 allvalues += tuple(values) |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2403 s = ','.join([a]*len(values)) |
|
6155
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2404 sql.append("""select %s%s.id from %s, %s where %s.__retired__=%s |
| 6965 | 2405 %sand %s.id = %s.%s and %s.%s in (%s)%s""" % ( |
| 2406 dis, cn, cn, tn, cn, a, ret, cn, tn, nn, tn, ln, s, ord)) | |
|
2733
ef396596a24e
more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents:
2731
diff
changeset
|
2407 |
|
ef396596a24e
more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents:
2731
diff
changeset
|
2408 if not sql: |
|
1988
5660b89f8903
more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents:
1986
diff
changeset
|
2409 return [] |
|
2733
ef396596a24e
more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents:
2731
diff
changeset
|
2410 sql = ' union '.join(sql) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
2411 self.db.sql(sql, allvalues) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2412 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2413 l = [str(x[0]) for x in self.db.sql_fetchall()] |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2414 return l |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2415 |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2416 def stringFind(self, **requirements): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2417 """Locate a particular node by matching a set of its String |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2418 properties in a caseless search. |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2419 |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2420 If the property is not a String property, a TypeError is raised. |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2421 |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2422 The return is a list of the id of all nodes that match. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2423 """ |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2424 where = [] |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2425 args = [] |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2426 for propname in requirements: |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2427 prop = self.properties[propname] |
|
1951
767ff2a03eee
more unit tests to improve coverage (up from 85% to 88% for anydbm! :)
Richard Jones <richard@users.sourceforge.net>
parents:
1926
diff
changeset
|
2428 if not isinstance(prop, String): |
| 6965 | 2429 raise TypeError("'%s' not a String property" % propname) |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2430 where.append(propname) |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2431 args.append(requirements[propname].lower()) |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2432 |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2433 # generate the where clause |
| 6965 | 2434 s = ' and '.join(['lower(_%s)=%s' % (col, |
| 2435 self.db.arg) for col in where]) | |
| 2436 sql = 'select id from _%s where %s and __retired__=%s' % ( | |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
2437 self.classname, s, self.db.arg) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2438 args.append(0) |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2439 self.db.sql(sql, tuple(args)) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2440 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2441 l = [str(x[0]) for x in self.db.sql_fetchall()] |
| 1165 | 2442 return l |
| 2443 | |
| 2444 def list(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2445 """ Return a list of the ids of the active nodes in this class. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2446 """ |
|
1484
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2447 return self.getnodeids(retired=0) |
|
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2448 |
|
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2449 def getnodeids(self, retired=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2450 """ Retrieve all the ids of the nodes for a particular Class. |
|
1484
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2451 |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2452 Set retired=None to get all nodes. Otherwise it'll get all the |
|
1484
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2453 retired or non-retired nodes, depending on the flag. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2454 """ |
|
1707
3d627e34f18e
sqlite backend now passes all tests under 2.3.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1599
diff
changeset
|
2455 # flip the sense of the 'retired' flag if we don't want all of them |
|
1484
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2456 if retired is not None: |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2457 args = (0, ) |
|
1719
eeb167fb8faf
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
1707
diff
changeset
|
2458 if retired: |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2459 compare = '>' |
|
1719
eeb167fb8faf
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
1707
diff
changeset
|
2460 else: |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2461 compare = '=' |
| 6965 | 2462 sql = 'select id from _%s where __retired__%s%s' % ( |
| 2463 self.classname, compare, self.db.arg) | |
|
1539
800b5896e14a
fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents:
1528
diff
changeset
|
2464 else: |
|
800b5896e14a
fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents:
1528
diff
changeset
|
2465 args = () |
| 6965 | 2466 sql = 'select id from _%s' % self.classname |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
2467 self.db.sql(sql, args) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2468 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2469 ids = [str(x[0]) for x in self.db.cursor.fetchall()] |
|
1539
800b5896e14a
fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents:
1528
diff
changeset
|
2470 return ids |
| 1165 | 2471 |
|
6399
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2472 def _subselect(self, proptree, parentname=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2473 """Create a subselect. This is factored out because some |
|
3633
a292054b4393
The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3612
diff
changeset
|
2474 databases (hmm only one, so far) doesn't support subselects |
|
a292054b4393
The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3612
diff
changeset
|
2475 look for "I can't believe it's not a toy RDBMS" in the mysql |
|
a292054b4393
The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3612
diff
changeset
|
2476 backend. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2477 """ |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2478 multilink_table = proptree.propclass.table_name |
| 6965 | 2479 nodeid_name = proptree.propclass.nodeid_name |
| 2480 linkid_name = proptree.propclass.linkid_name | |
|
6399
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2481 if parentname is None: |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2482 parentname = '_' + proptree.parent.classname |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2483 |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2484 w = '' |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2485 if proptree.need_retired: |
| 6965 | 2486 w = ' where %s.__retired__=0' % (multilink_table) |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2487 if proptree.need_child_retired: |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2488 tn1 = multilink_table |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2489 tn2 = '_' + proptree.classname |
| 6965 | 2490 w = ', %s where %s.%s=%s.id and %s.__retired__=0' % ( |
| 2491 tn2, tn1, linkid_name, tn2, tn2) | |
| 2492 return '%s.id not in (select %s from %s%s)' % ( | |
| 2493 parentname, nodeid_name, multilink_table, w) | |
|
3633
a292054b4393
The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3612
diff
changeset
|
2494 |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2495 def _filter_multilink_expression_fallback(self, proptree, expr): |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2496 '''This is a fallback for database that do not support |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2497 subselects.''' |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2498 classname = proptree.parent.uniqname |
|
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2499 multilink_table = proptree.propclass.table_name |
|
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2500 nid = proptree.propclass.nodeid_name |
|
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2501 lid = proptree.propclass.linkid_name |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2502 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2503 is_valid = expr.evaluate |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2504 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2505 last_id, kws = None, [] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2506 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2507 ids = IdListOptimizer() |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2508 append = ids.append |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2509 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2510 # This join and the evaluation in program space |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2511 # can be expensive for larger databases! |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2512 # TODO: Find a faster way to collect the data needed |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2513 # to evalute the expression. |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2514 # Moving the expression evaluation into the database |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2515 # would be nice but this tricky: Think about the cases |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2516 # where the multilink table does not have join values |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2517 # needed in evaluation. |
|
6402
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2518 w = j = '' |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2519 s = 'm.%s' % lid |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2520 if proptree.need_retired: |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2521 w = ' and m.__retired__=0' |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2522 elif proptree.need_child_retired: |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2523 tn2 = '_' + proptree.classname |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2524 j = ' LEFT OUTER JOIN %s ON %s.id = m.%s' % (tn2, tn2, lid) |
| 6965 | 2525 w = ' and %s.__retired__=0' % (tn2) |
|
6402
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2526 s = '%s.id' % tn2 |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2527 |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2528 stmnt = "SELECT c.id, %s FROM _%s as c " \ |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2529 "LEFT OUTER JOIN %s as m " \ |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2530 "ON c.id = m.%s%s%s ORDER BY c.id" % ( |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2531 s, classname, multilink_table, nid, j, w) |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2532 self.db.sql(stmnt) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2533 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2534 # collect all multilink items for a class item |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2535 for nodeid, kw in self.db.sql_fetchiter(): |
|
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2536 if nodeid != last_id: |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2537 if last_id is None: |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2538 last_id = nodeid |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2539 else: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2540 # we have all multilink items -> evaluate! |
| 6965 | 2541 if is_valid(kws): append(last_id) # noqa: E701 |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2542 last_id, kws = nodeid, [] |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2543 if kw is not None: |
|
6395
8baf81d1cfc1
Make multilink expressions work with mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6392
diff
changeset
|
2544 kws.append(int(kw)) |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2545 |
| 6965 | 2546 if last_id is not None and is_valid(kws): |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2547 append(last_id) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2548 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2549 # we have ids of the classname table |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2550 return ids.where("_%s.id" % classname, self.db.arg) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2551 |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2552 def _filter_link_expression(self, proptree, v): |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2553 """ Filter elements in the table that match the given expression |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2554 """ |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2555 pln = proptree.parent.uniqname |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2556 prp = proptree.name |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2557 try: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2558 opcodes = [int(x) for x in v] |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2559 if min(opcodes) >= -1: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2560 raise ValueError() |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2561 expr = compile_expression(opcodes) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2562 # NULL doesn't compare to NULL in SQL |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2563 # So not (x = '1') will *not* include NULL values for x |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2564 # That's why we need that and clause: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2565 atom = "_%s._%s = %s and _%s._%s is not NULL" % ( |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2566 pln, prp, self.db.arg, pln, prp) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2567 atom_nil = "_%s._%s is NULL" % (pln, prp) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2568 lambda_atom = lambda n: atom if n.x >= 0 else atom_nil |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2569 values = [] |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2570 w = expr.generate(lambda_atom) |
| 6965 | 2571 |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2572 def collect_values(n): |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2573 if n.x >= 0: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2574 values.append(n.x) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2575 expr.visit(collect_values) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2576 return w, values |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2577 except: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2578 pass |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2579 # Fallback to original code |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2580 args = [] |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2581 where = None |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2582 d = {} |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2583 for entry in v: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2584 if entry == '-1': |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2585 entry = None |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2586 d[entry] = entry |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2587 l = [] |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2588 if None in d or not d: |
| 6965 | 2589 if None in d: del d[None] # noqa: E701 |
| 2590 l.append('_%s._%s is NULL' % (pln, prp)) | |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2591 if d: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2592 v = list(d) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2593 s = ','.join([self.db.arg for x in v]) |
| 6965 | 2594 l.append('(_%s._%s in (%s))' % (pln, prp, s)) |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2595 args = v |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2596 if l: |
| 6965 | 2597 where = '(' + ' or '.join(l) + ')' |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2598 return where, args |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2599 |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2600 def _filter_multilink_expression(self, proptree, v): |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2601 """ Filters out elements of the classname table that do not |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2602 match the given expression. |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2603 Returns tuple of 'WHERE' introns for the overall filter. |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2604 """ |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2605 classname = proptree.parent.uniqname |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2606 multilink_table = proptree.propclass.table_name |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2607 nid = proptree.propclass.nodeid_name |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2608 lid = proptree.propclass.linkid_name |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2609 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2610 try: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2611 opcodes = [int(x) for x in v] |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2612 if min(opcodes) >= -1: |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2613 raise ValueError() |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2614 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2615 expr = compile_expression(opcodes) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2616 |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2617 if not self.supports_subselects: |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2618 # We heavily rely on subselects. If there is |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2619 # no decent support fall back to slower variant. |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2620 return self._filter_multilink_expression_fallback( |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2621 proptree, expr) |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2622 |
|
6402
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2623 w = j = '' |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2624 if proptree.need_retired: |
| 6965 | 2625 w = ' and %s.__retired__=0' % (multilink_table) |
|
6402
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2626 elif proptree.need_child_retired: |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2627 tn1 = multilink_table |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2628 tn2 = '_' + proptree.classname |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2629 j = ', %s' % tn2 |
| 6965 | 2630 w = ' and %s.%s=%s.id and %s.__retired__=0' % ( |
| 2631 tn1, lid, tn2, tn2) | |
|
6402
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2632 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2633 atom = \ |
|
6402
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2634 "%s IN(SELECT %s FROM %s%s WHERE %s=a.id%s)" % ( |
| 6965 | 2635 self.db.arg, lid, multilink_table, j, nid, w) |
|
6399
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2636 atom_nil = self._subselect(proptree, 'a') |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2637 |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2638 lambda_atom = lambda n: atom if n.x >= 0 else atom_nil |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2639 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2640 intron = \ |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2641 "_%(classname)s.id in (SELECT id " \ |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2642 "FROM _%(classname)s AS a WHERE %(condition)s) " % { |
| 6965 | 2643 'classname': classname, |
| 2644 'condition': expr.generate(lambda_atom)} | |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2645 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2646 values = [] |
| 6965 | 2647 |
|
6399
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2648 def collect_values(n): |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2649 if n.x >= 0: |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2650 values.append(n.x) |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2651 expr.visit(collect_values) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2652 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2653 return intron, values |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2654 except: |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2655 # fallback behavior when expression parsing above fails |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2656 orclause = '' |
| 6965 | 2657 if '-1' in v: |
| 2658 v = [x for x in v if int(x) > 0] | |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2659 orclause = self._subselect(proptree) |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2660 where = [] |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2661 where.append("%s.%s in (%s)" % (multilink_table, lid, |
| 6965 | 2662 ','.join([self.db.arg] * len(v)))) |
| 2663 where.append('_%s.id=%s.%s' % (classname, multilink_table, nid)) | |
| 2664 where = ' and '.join(where) | |
| 2665 if orclause: | |
| 2666 where = '((' + ' or '.join((where + ')', orclause)) + ')' | |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2667 |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2668 return where, v |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2669 |
| 6965 | 2670 def _filter_sql(self, search_matches, filterspec, srt=[], grp=[], retr=0, |
| 2671 retired=False, exact_match_spec={}, limit=None, | |
| 2672 offset=None): | |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2673 """ Compute the proptree and the SQL/ARGS for a filter. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2674 For argument description see filter below. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2675 We return a 3-tuple, the proptree, the sql and the sql-args |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2676 or None if no SQL is necessary. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2677 The flag retr serves to retrieve *all* non-Multilink properties |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2678 (for filling the cache during a filter_iter) |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2679 """ |
|
2578
7f25486ff85e
fixed RDBMS filter() for no matches from full-text search [SF#990778]
Richard Jones <richard@users.sourceforge.net>
parents:
2546
diff
changeset
|
2680 # we can't match anything if search_matches is empty |
| 4044 | 2681 if not search_matches and search_matches is not None: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2682 return None |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
2683 |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2684 icn = self.classname |
| 1165 | 2685 |
|
2379
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2686 # vars to hold the components of the SQL statement |
|
2586
3c67f4bfa225
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2578
diff
changeset
|
2687 frum = [] # FROM clauses |
|
2379
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2688 loj = [] # LEFT OUTER JOIN clauses |
|
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2689 where = [] # WHERE clauses |
|
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2690 args = [] # *any* positional arguments |
|
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2691 a = self.db.arg |
|
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2692 |
| 1165 | 2693 # figure the WHERE clause from the filterspec |
|
6413
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2694 use_distinct = False # Do we need a distinct clause? |
| 6965 | 2695 sortattr = self._sortattr(group=grp, sort=srt) |
|
5871
acc4a128ab9b
Backwards-compatible signature of _proptree method
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5869
diff
changeset
|
2696 proptree = self._proptree(filterspec, exact_match_spec, sortattr, retr) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2697 mlseen = 0 |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2698 for pt in reversed(proptree.sortattr): |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2699 p = pt |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2700 while p.parent: |
| 6965 | 2701 if isinstance(p.propclass, Multilink): |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2702 mlseen = True |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2703 if mlseen: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2704 p.sort_ids_needed = True |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2705 p.tree_sort_done = False |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2706 p = p.parent |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2707 if not mlseen: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2708 pt.attr_sort_done = pt.tree_sort_done = True |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2709 proptree.compute_sort_done() |
|
3693
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2710 |
| 6965 | 2711 cols = ['_%s.id' % icn] |
|
3635
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
2712 for p in proptree: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2713 rc = ac = oc = None |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2714 cn = p.classname |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2715 ln = p.uniqname |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2716 pln = p.parent.uniqname |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2717 pcn = p.parent.classname |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2718 k = p.name |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2719 v = p.val |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2720 propclass = p.propclass |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2721 if p.parent == proptree and p.name == 'id' \ |
| 6965 | 2722 and 'retrieve' in p.need_for: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2723 p.sql_idx = 0 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2724 if 'sort' in p.need_for or 'retrieve' in p.need_for: |
| 6965 | 2725 rc = oc = ac = '_%s._%s' % (pln, k) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2726 if isinstance(propclass, Multilink): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2727 if 'search' in p.need_for: |
|
6413
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2728 # if we joining with Multilink tables we need distinct |
|
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2729 use_distinct = True |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
2730 tn = propclass.table_name |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
2731 nid = propclass.nodeid_name |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
2732 lid = propclass.linkid_name |
|
6403
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2733 frum.append(tn) |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2734 if p.children or p.need_child_retired: |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2735 frum.append('_%s as _%s' % (cn, ln)) |
| 6965 | 2736 where.append('%s.%s=_%s.id' % (tn, lid, ln)) |
|
6403
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2737 if p.need_child_retired: |
| 6965 | 2738 where.append('_%s.__retired__=0' % (ln)) |
|
6403
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2739 # Note: need the where-clause if p has |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2740 # children that compute additional restrictions |
| 6965 | 2741 if (not p.has_values or |
| 2742 (not isinstance(v, type([])) and v != '-1') or | |
| 2743 p.children): | |
| 2744 where.append('_%s.id=%s.%s' % (pln, tn, nid)) | |
|
4061
52431f2ad76b
Correct handling of a Multilink filterspec with an empty list of linked-to items.
Stefan Seefeld <stefan@seefeld.name>
parents:
4060
diff
changeset
|
2745 if v in ('-1', ['-1'], []): |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2746 # only match rows that have count(linkid)=0 in the |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2747 # corresponding multilink table) |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2748 where.append(self._subselect(p)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2749 else: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2750 if p.has_values: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2751 if isinstance(v, type([])): |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2752 # The where-clause above is conditionally |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2753 # created in _filter_multilink_expression |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2754 w, arg = self._filter_multilink_expression(p, v) |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2755 where.append(w) |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2756 args += arg |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2757 else: |
| 6965 | 2758 where.append('%s.%s=%s' % (tn, lid, a)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2759 args.append(v) |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2760 # Don't match retired nodes if rev_multilink |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2761 if p.need_retired: |
| 6965 | 2762 where.append('%s.__retired__=0' % (tn)) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2763 if 'sort' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2764 assert not p.attr_sort_done and not p.sort_ids_needed |
|
1365
4884fb0860f9
fixed rdbms searching by ID [SF#666615]
Richard Jones <richard@users.sourceforge.net>
parents:
1351
diff
changeset
|
2765 elif k == 'id': |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2766 if 'search' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2767 if isinstance(v, type([])): |
|
4462
0f4a8fcb9a1d
fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents:
4448
diff
changeset
|
2768 # If there are no permitted values, then the |
|
0f4a8fcb9a1d
fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents:
4448
diff
changeset
|
2769 # where clause will always be false, and we |
|
0f4a8fcb9a1d
fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents:
4448
diff
changeset
|
2770 # can optimize the query away. |
|
0f4a8fcb9a1d
fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents:
4448
diff
changeset
|
2771 if not v: |
|
4803
5e679e364f9a
Bug-fix: need to return None for empty query
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4787
diff
changeset
|
2772 return None |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2773 s = ','.join([a for x in v]) |
| 6965 | 2774 where.append('_%s.%s in (%s)' % (pln, k, s)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2775 args = args + v |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2776 else: |
| 6965 | 2777 where.append('_%s.%s=%s' % (pln, k, a)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2778 args.append(v) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2779 if 'sort' in p.need_for or 'retrieve' in p.need_for: |
| 6965 | 2780 rc = oc = ac = '_%s.id' % pln |
|
3683
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2781 elif isinstance(propclass, String): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2782 if 'search' in p.need_for: |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2783 exact = [] |
|
3683
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2784 if not isinstance(v, type([])): |
|
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2785 v = [v] |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2786 new_v = [] |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2787 for x in v: |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2788 if isinstance(x, hyperdb.Exact_Match): |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2789 exact.append(True) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2790 new_v.append(x.value) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2791 else: |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2792 exact.append(False) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2793 # Quote special search characters '%' and '_' for |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2794 # correct matching with LIKE/ILIKE |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2795 # Note that we now pass the elements of v as query |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2796 # arguments and don't interpolate the quoted string |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2797 # into the sql statement. Should be safer. |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2798 new_v.append(self.db.search_stringquote(x)) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2799 v = new_v |
|
3683
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2800 |
|
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2801 # now add to the where clause |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2802 w = [] |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2803 for vv, ex in zip(v, exact): |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2804 if ex: |
| 6965 | 2805 w.append("_%s._%s %s %s" % ( |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2806 pln, k, self.case_sensitive_equal, a)) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2807 args.append(vv) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2808 else: |
| 6965 | 2809 w.append("_%s._%s %s %s ESCAPE %s" % ( |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2810 pln, k, self.case_insensitive_like, a, a)) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2811 args.extend((vv, '\\')) |
| 6965 | 2812 where.append('(' + ' and '.join(w) + ')') |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2813 if 'sort' in p.need_for: |
| 6965 | 2814 oc = ac = 'lower(_%s._%s)' % (pln, k) |
|
1170
af104fa52746
Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1168
diff
changeset
|
2815 elif isinstance(propclass, Link): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2816 if 'search' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2817 if p.children: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2818 if 'sort' not in p.need_for: |
|
3694
595041d78104
Fix a peculiarity in sql generation.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3693
diff
changeset
|
2819 frum.append('_%s as _%s' % (cn, ln)) |
|
6413
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2820 c = [x for x in p.children if 'search' in x.need_for] |
|
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2821 if c: |
| 6965 | 2822 where.append('_%s._%s=_%s.id' % (pln, k, ln)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2823 if p.has_values: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2824 if isinstance(v, type([])): |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2825 w, arg = self._filter_link_expression(p, v) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2826 if w: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2827 where.append(w) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2828 args += arg |
|
3636
fa7becc62534
Of course for Links there is the same bug as I fixed before for multilinks.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3635
diff
changeset
|
2829 else: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2830 if v in ('-1', None): |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2831 v = None |
| 6965 | 2832 where.append('_%s._%s is NULL' % (pln, k)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2833 else: |
| 6965 | 2834 where.append('_%s._%s=%s' % (pln, k, a)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2835 args.append(v) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2836 if 'sort' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2837 lp = p.cls.labelprop() |
| 6965 | 2838 oc = ac = '_%s._%s' % (pln, k) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2839 if lp != 'id': |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2840 if p.tree_sort_done: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2841 loj.append( |
| 6965 | 2842 'LEFT OUTER JOIN _%s as _%s on _%s._%s=_%s.id' % ( |
| 2843 cn, ln, pln, k, ln)) | |
| 2844 oc = '_%s._%s' % (ln, lp) | |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2845 if 'retrieve' in p.need_for: |
| 6965 | 2846 rc = '_%s._%s' % (pln, k) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2847 elif isinstance(propclass, Date) and 'search' in p.need_for: |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
2848 dc = self.db.to_sql_value(hyperdb.Date) |
|
1351
d1bfb479e527
fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents:
1345
diff
changeset
|
2849 if isinstance(v, type([])): |
|
d1bfb479e527
fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents:
1345
diff
changeset
|
2850 s = ','.join([a for x in v]) |
| 6965 | 2851 where.append('_%s._%s in (%s)' % (pln, k, s)) |
|
3633
a292054b4393
The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3612
diff
changeset
|
2852 args = args + [dc(date.Date(x)) for x in v] |
|
1351
d1bfb479e527
fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents:
1345
diff
changeset
|
2853 else: |
|
1499
8ee69708da0c
added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1496
diff
changeset
|
2854 try: |
| 6118 | 2855 wh = [] |
| 2856 ar = [] | |
| 2857 for d in v.split(','): | |
| 2858 w1 = [] | |
| 2859 if d == '-': | |
| 6965 | 2860 wh.append('_%s._%s is NULL' % (pln, k)) |
| 6118 | 2861 continue |
| 2862 # Try to filter on range of dates | |
| 2863 date_rng = propclass.range_from_raw(d, self.db) | |
| 2864 if date_rng.from_value: | |
| 6965 | 2865 w1.append('_%s._%s >= %s' % (pln, k, a)) |
| 6118 | 2866 ar.append(dc(date_rng.from_value)) |
| 2867 if date_rng.to_value: | |
| 6965 | 2868 w1.append('_%s._%s <= %s' % (pln, k, a)) |
| 6118 | 2869 ar.append(dc(date_rng.to_value)) |
| 6965 | 2870 wh.append(' and '.join(w1)) |
| 2871 where.append('(' + ' or '.join(wh) + ')') | |
| 2872 args.extend(ar) | |
|
1499
8ee69708da0c
added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1496
diff
changeset
|
2873 except ValueError: |
|
8ee69708da0c
added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1496
diff
changeset
|
2874 # If range creation fails - ignore that search parameter |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2875 pass |
|
1351
d1bfb479e527
fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents:
1345
diff
changeset
|
2876 elif isinstance(propclass, Interval): |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2877 # filter/sort using the __<prop>_int__ column |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2878 if 'search' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2879 if isinstance(v, type([])): |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2880 s = ','.join([a for x in v]) |
| 6965 | 2881 where.append('_%s.__%s_int__ in (%s)' % (pln, k, s)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2882 args = args + [date.Interval(x).as_seconds() for x in v] |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2883 else: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2884 try: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2885 # Try to filter on range of intervals |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2886 date_rng = Range(v, date.Interval) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2887 if date_rng.from_value: |
| 6965 | 2888 where.append('_%s.__%s_int__ >= %s' % ( |
| 2889 pln, k, a)) | |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2890 args.append(date_rng.from_value.as_seconds()) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2891 if date_rng.to_value: |
| 6965 | 2892 where.append('_%s.__%s_int__ <= %s' % ( |
| 2893 pln, k, a)) | |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2894 args.append(date_rng.to_value.as_seconds()) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2895 except ValueError: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2896 # If range creation fails - ignore search parameter |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2897 pass |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2898 if 'sort' in p.need_for: |
| 6965 | 2899 oc = ac = '_%s.__%s_int__' % (pln, k) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2900 if 'retrieve' in p.need_for: |
| 6965 | 2901 rc = '_%s._%s' % (pln, k) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2902 elif isinstance(propclass, Boolean) and 'search' in p.need_for: |
| 6965 | 2903 if isinstance(v, str): |
|
4365
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2904 v = v.split(',') |
| 6965 | 2905 if not isinstance(v, list): |
|
4365
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2906 v = [v] |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2907 bv = [] |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2908 for val in v: |
| 6965 | 2909 if isinstance(val, str): |
| 2910 bv.append(propclass.from_raw(val)) | |
|
4365
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2911 else: |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2912 bv.append(bool(val)) |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2913 if len(bv) == 1: |
| 6965 | 2914 where.append('_%s._%s=%s' % (pln, k, a)) |
|
4365
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2915 args = args + bv |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2916 else: |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2917 s = ','.join([a for x in v]) |
| 6965 | 2918 where.append('_%s._%s in (%s)' % (pln, k, s)) |
|
4365
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2919 args = args + bv |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2920 elif 'search' in p.need_for: |
| 1165 | 2921 if isinstance(v, type([])): |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
2922 s = ','.join([a for x in v]) |
| 6965 | 2923 where.append('_%s._%s in (%s)' % (pln, k, s)) |
| 1165 | 2924 args = args + v |
| 2925 else: | |
| 6965 | 2926 where.append('_%s._%s=%s' % (pln, k, a)) |
| 1165 | 2927 args.append(v) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2928 if oc: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2929 if p.sort_ids_needed: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2930 if rc == ac: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2931 p.sql_idx = len(cols) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2932 p.auxcol = len(cols) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2933 cols.append(ac) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2934 if p.tree_sort_done and p.sort_direction: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2935 # Don't select top-level id or multilink twice |
| 6965 | 2936 if (not p.sort_ids_needed or ac != oc) and ( |
| 2937 p.name != 'id' or p.parent != proptree): | |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2938 if rc == oc: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2939 p.sql_idx = len(cols) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2940 cols.append(oc) |
|
3685
4d9adb8bc3b1
Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3683
diff
changeset
|
2941 desc = ['', ' desc'][p.sort_direction == '-'] |
|
4d9adb8bc3b1
Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3683
diff
changeset
|
2942 # Some SQL dbs sort NULL values last -- we want them first. |
|
4d9adb8bc3b1
Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3683
diff
changeset
|
2943 if (self.order_by_null_values and p.name != 'id'): |
|
4d9adb8bc3b1
Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3683
diff
changeset
|
2944 nv = self.order_by_null_values % oc |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2945 cols.append(nv) |
|
3693
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2946 p.orderby.append(nv + desc) |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2947 p.orderby.append(oc + desc) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2948 if 'retrieve' in p.need_for and p.sql_idx is None: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2949 assert(rc) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2950 p.sql_idx = len(cols) |
| 6965 | 2951 cols.append(rc) |
| 1165 | 2952 |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2953 props = self.getprops() |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2954 |
|
1740
5ca448ff8052
don't have RDBMS backends list retired nodes [SF#767319]
Richard Jones <richard@users.sourceforge.net>
parents:
1719
diff
changeset
|
2955 # don't match retired nodes |
|
5318
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2956 if retired is not None: |
|
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2957 op = '=' |
|
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2958 if retired: |
|
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2959 op = '!=' |
| 6965 | 2960 where.append('_%s.__retired__%s0' % (icn, op)) |
|
1740
5ca448ff8052
don't have RDBMS backends list retired nodes [SF#767319]
Richard Jones <richard@users.sourceforge.net>
parents:
1719
diff
changeset
|
2961 |
| 1165 | 2962 # add results of full text search |
| 2963 if search_matches is not None: | |
| 4044 | 2964 s = ','.join([a for x in search_matches]) |
| 6965 | 2965 where.append('_%s.id in (%s)' % (icn, s)) |
| 4113 | 2966 args = args + [x for x in search_matches] |
| 1165 | 2967 |
| 2968 # construct the SQL | |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2969 frum.append('_'+icn) |
| 1165 | 2970 frum = ','.join(frum) |
|
1170
af104fa52746
Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1168
diff
changeset
|
2971 if where: |
|
af104fa52746
Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1168
diff
changeset
|
2972 where = ' where ' + (' and '.join(where)) |
|
af104fa52746
Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1168
diff
changeset
|
2973 else: |
|
af104fa52746
Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1168
diff
changeset
|
2974 where = '' |
|
6413
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2975 if use_distinct: |
|
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2976 # Avoid dupes |
| 6965 | 2977 cols[0] = 'distinct(_%s.id)' % icn |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2978 |
|
3693
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2979 order = [] |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2980 # keep correct sequence of order attributes. |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2981 for sa in proptree.sortattr: |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2982 if not sa.attr_sort_done: |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2983 continue |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2984 order.extend(sa.orderby) |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2985 if order: |
| 6965 | 2986 order = ' order by %s' % (','.join(order)) |
| 1165 | 2987 else: |
| 2988 order = '' | |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2989 |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2990 if limit is not None: |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2991 limit = ' LIMIT %s' % limit |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2992 else: |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2993 limit = '' |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2994 if offset is not None: |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2995 offset = ' OFFSET %s' % offset |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2996 else: |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2997 offset = '' |
| 1165 | 2998 cols = ','.join(cols) |
|
2379
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2999 loj = ' '.join(loj) |
| 6965 | 3000 sql = 'select %s from %s %s %s%s%s%s' % ( |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3001 cols, frum, loj, where, order, limit, offset) |
| 1165 | 3002 args = tuple(args) |
|
2319
7cf7e3bd1b31
more column uniqueness fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2318
diff
changeset
|
3003 __traceback_info__ = (sql, args) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3004 return proptree, sql, args |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3005 |
|
5318
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
3006 def filter(self, search_matches, filterspec, sort=[], group=[], |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3007 retired=False, exact_match_spec={}, limit=None, offset=None): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3008 """Return a list of the ids of the active nodes in this class that |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3009 match the 'filter' spec, sorted by the group spec and then the |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3010 sort spec |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3011 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3012 "filterspec" is {propname: value(s)} |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3013 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3014 "sort" and "group" are [(dir, prop), ...] where dir is '+', '-' |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3015 or None and prop is a prop name or None. Note that for |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3016 backward-compatibility reasons a single (dir, prop) tuple is |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3017 also allowed. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3018 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3019 "search_matches" is a container type or None |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3020 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3021 The filter must match all properties specificed. If the property |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3022 value to match is a list: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3023 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3024 1. String properties must match all elements in the list, and |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3025 2. Other properties must match any of the elements in the list. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3026 """ |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3027 if __debug__: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3028 start_t = time.time() |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3029 |
| 6965 | 3030 sq = self._filter_sql(search_matches, filterspec, sort, group, |
| 3031 retired=retired, | |
| 3032 exact_match_spec=exact_match_spec, | |
| 3033 limit=limit, offset=offset) | |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3034 # nothing to match? |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3035 if sq is None: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3036 return [] |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3037 proptree, sql, args = sq |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3038 |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3039 cursor = self.db.sql_new_cursor(name='filter') |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3040 self.db.sql(sql, args, cursor) |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3041 # Reduce this to only the first row (the ID), this can save a |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3042 # lot of space for large query results (not using fetchall) |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3043 # We cannot do this if sorting by multilink |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3044 if proptree.tree_sort_done: |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3045 l = [str(row[0]) for row in cursor] |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3046 else: |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3047 l = cursor.fetchall() |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3048 cursor.close() |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3049 |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3050 # Multilink sorting |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
3051 # Compute values needed for sorting in proptree.sort |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3052 if not proptree.tree_sort_done: |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3053 for p in proptree: |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3054 if hasattr(p, 'auxcol'): |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3055 p.sort_ids = [row[p.auxcol] for row in l] |
| 6965 | 3056 p.sort_result = p._sort_repr( |
| 3057 p.propclass.sort_repr, p.sort_ids) | |
| 3058 l = proptree.sort([str(row[0]) for row in l]) | |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
3059 |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
3060 if __debug__: |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
3061 self.db.stats['filtering'] += (time.time() - start_t) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
3062 return l |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
3063 |
|
5318
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
3064 def filter_iter(self, search_matches, filterspec, sort=[], group=[], |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3065 retired=False, exact_match_spec={}, limit=None, |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3066 offset=None): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3067 """Iterator similar to filter above with same args. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3068 Limitation: We don't sort on multilinks. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3069 This uses an optimisation: We put all nodes that are in the |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3070 current row into the node cache. Then we return the node id. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3071 That way a fetch of a node won't create another sql-fetch (with |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3072 a join) from the database because the nodes are already in the |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3073 cache. We're using our own temporary cursor. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3074 """ |
|
5318
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
3075 sq = self._filter_sql(search_matches, filterspec, sort, group, retr=1, |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
3076 retired=retired, |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3077 exact_match_spec=exact_match_spec, |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3078 limit=limit, offset=offset) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3079 # nothing to match? |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3080 if sq is None: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3081 return |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3082 proptree, sql, args = sq |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3083 cursor = self.db.sql_new_cursor(name='filter_iter') |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3084 self.db.sql(sql, args, cursor) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3085 classes = {} |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3086 for p in proptree: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3087 if 'retrieve' in p.need_for: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3088 cn = p.parent.classname |
| 6965 | 3089 ptid = p.parent.id # not the nodeid! |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3090 key = (cn, ptid) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3091 if key not in classes: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3092 classes[key] = {} |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3093 name = p.name |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3094 assert (name) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3095 classes[key][name] = p |
|
4473
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
3096 p.to_hyperdb = self.db.to_hyperdb_value(p.propclass.__class__) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3097 while True: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3098 row = cursor.fetchone() |
| 6965 | 3099 if not row: break # noqa: E701 |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3100 # populate cache with current items |
| 6965 | 3101 for (classname, _ptid), pt in classes.items(): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3102 nodeid = str(row[pt['id'].sql_idx]) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3103 key = (classname, nodeid) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3104 if key in self.db.cache: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3105 self.db._cache_refresh(key) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3106 continue |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3107 node = {} |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
3108 for propname, p in pt.items(): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3109 value = row[p.sql_idx] |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3110 if value is not None: |
|
4473
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
3111 value = p.to_hyperdb(value) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3112 node[propname] = value |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3113 self.db._cache_save(key, node) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3114 yield str(row[0]) |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3115 cursor.close() |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3116 |
|
3688
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3117 def filter_sql(self, sql): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3118 """Return a list of the ids of the items in this class that match |
|
3688
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3119 the SQL provided. The SQL is a complete "select" statement. |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3120 |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3121 The SQL select must include the item id as the first column. |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3122 |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3123 This function DOES NOT filter out retired items, add on a where |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
3124 clause "__retired__=0" if you don't want retired nodes. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3125 """ |
|
3688
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3126 if __debug__: |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3127 start_t = time.time() |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3128 |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3129 self.db.sql(sql) |
| 6965 | 3130 results = self.db.sql_fetchall() |
|
3688
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3131 |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3132 if __debug__: |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3133 self.db.stats['filtering'] += (time.time() - start_t) |
| 6965 | 3134 return results |
|
3688
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3135 |
| 1165 | 3136 def count(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3137 """Get the number of nodes in this class. |
| 1165 | 3138 |
| 3139 If the returned integer is 'numnodes', the ids of all the nodes | |
| 3140 in this class run from 1 to numnodes, and numnodes+1 will be the | |
| 3141 id of the next node to be created in this class. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3142 """ |
| 1165 | 3143 return self.db.countnodes(self.classname) |
| 3144 | |
| 3145 # Manipulating properties: | |
| 3146 def getprops(self, protected=1): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3147 """Return a dictionary mapping property names to property objects. |
| 1165 | 3148 If the "protected" flag is true, we include protected properties - |
| 3149 those which may not be modified. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3150 """ |
| 1165 | 3151 d = self.properties.copy() |
| 3152 if protected: | |
| 3153 d['id'] = String() | |
| 3154 d['creation'] = hyperdb.Date() | |
| 3155 d['activity'] = hyperdb.Date() | |
|
1176
bd3b57859c37
On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents:
1175
diff
changeset
|
3156 d['creator'] = hyperdb.Link('user') |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
3157 d['actor'] = hyperdb.Link('user') |
| 1165 | 3158 return d |
| 3159 | |
| 3160 def addprop(self, **properties): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3161 """Add properties to this class. |
| 1165 | 3162 |
| 3163 The keyword arguments in 'properties' must map names to property | |
| 3164 objects, or a TypeError is raised. None of the keys in 'properties' | |
| 3165 may collide with the names of existing properties, or a ValueError | |
| 3166 is raised before any properties have been added. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3167 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3168 for key in properties: |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3169 if key in self.properties: |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3170 raise ValueError(key) |
| 1165 | 3171 self.properties.update(properties) |
| 3172 | |
| 3173 def index(self, nodeid): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3174 """Add (or refresh) the node to search indexes |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3175 """ |
| 1165 | 3176 # find all the String properties that have indexme |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
3177 for prop, propclass in self.getprops().items(): |
| 1165 | 3178 if isinstance(propclass, String) and propclass.indexme: |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3179 self.db.indexer.add_text((self.classname, nodeid, prop), |
| 6965 | 3180 str(self.get(nodeid, prop))) |
| 1165 | 3181 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3182 # |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3183 # import / export support |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3184 # |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3185 def export_list(self, propnames, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3186 """ Export a node - generate a list of CSV-able data in the order |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3187 specified by propnames for the given node. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3188 """ |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3189 properties = self.getprops() |
| 6965 | 3190 return_list = [] |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3191 for prop in propnames: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3192 proptype = properties[prop] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3193 value = self.get(nodeid, prop) |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3194 # "marshal" data where needed |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3195 if value is None: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3196 pass |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3197 elif isinstance(proptype, hyperdb.Date): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3198 value = value.get_tuple() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3199 elif isinstance(proptype, hyperdb.Interval): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3200 value = value.get_tuple() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3201 elif isinstance(proptype, hyperdb.Password): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3202 value = str(value) |
| 6965 | 3203 return_list.append(repr_export(value)) |
| 3204 return_list.append(repr_export(self.is_retired(nodeid))) | |
| 3205 return return_list | |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3206 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3207 def import_list(self, propnames, proplist): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3208 """ Import a node - all information including "id" is present and |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3209 should not be sanity checked. Triggers are not triggered. The |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3210 journal should be initialised using the "creator" and "created" |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3211 information. |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3212 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3213 Return the nodeid of the node imported. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3214 """ |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3215 |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3216 logger = logging.getLogger('roundup.hyperdb.backend') |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3217 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3218 if self.db.journaltag is None: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3219 raise DatabaseError(_('Database open read-only')) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3220 properties = self.getprops() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3221 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3222 # make the new node's property map |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3223 d = {} |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3224 retire = 0 |
| 6965 | 3225 if "id" not in propnames: |
|
2505
bdd112cf61ba
rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
3226 newid = self.db.newid(self.classname) |
|
bdd112cf61ba
rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
3227 else: |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3228 newid = eval_import(proplist[propnames.index("id")]) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3229 for i in range(len(propnames)): |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3230 # Use eval_import to reverse the repr_export() used to |
|
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3231 # output the CSV |
|
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3232 value = eval_import(proplist[i]) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3233 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3234 # Figure the property for this column |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3235 propname = propnames[i] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3236 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3237 # "unmarshal" where necessary |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3238 if propname == 'id': |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3239 continue |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3240 elif propname == 'is retired': |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3241 # is the item retired? |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3242 if int(value): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3243 retire = 1 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3244 continue |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3245 elif value is None: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3246 d[propname] = None |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3247 continue |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3248 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3249 prop = properties[propname] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3250 if value is None: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3251 # don't set Nones |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3252 continue |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3253 elif isinstance(prop, hyperdb.Date): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3254 value = date.Date(value) |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3255 elif isinstance(prop, hyperdb.Interval): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3256 value = date.Interval(value) |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3257 elif isinstance(prop, hyperdb.Password): |
|
7211
506c86823abb
Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents:
7038
diff
changeset
|
3258 value = password.Password(encrypted=value, |
|
506c86823abb
Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents:
7038
diff
changeset
|
3259 config=self.db.config) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
3260 elif isinstance(prop, String): |
|
5416
56c9bcdea47f
Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5395
diff
changeset
|
3261 value = us2s(value) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
3262 if not isinstance(value, str): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3263 raise TypeError('new property "%(propname)s" not a ' |
| 6965 | 3264 'string: %(value)r' % locals()) |
|
2892
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
3265 if prop.indexme: |
|
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
3266 self.db.indexer.add_text((self.classname, newid, propname), |
| 6965 | 3267 value) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
3268 d[propname] = value |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3269 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3270 # get a new id if necessary |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
3271 if newid is None: |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3272 newid = self.db.newid(self.classname) |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
3273 |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3274 activeid = None |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3275 has_node = False |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3276 |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3277 # use the arg for __retired__ to cope with any odd database type |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3278 # conversion (hello, sqlite) |
| 6965 | 3279 retired_sql = 'update _%s set __retired__=%s where id=%s' % ( |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3280 self.classname, self.db.arg, self.db.arg) |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3281 |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
3282 # insert new node or update existing? |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3283 # if integrity error raised try to recover |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3284 try: |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3285 has_node = self.hasnode(newid) |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3286 if not has_node: |
| 6965 | 3287 self.db.addnode(self.classname, newid, d) # insert |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3288 else: |
| 6965 | 3289 self.db.setnode(self.classname, newid, d) # update |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
3290 self.db.checkpoint_data() |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3291 # Blech, different db's return different exceptions |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3292 # so I can't list them here as some might not be defined |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3293 # on a given system. So capture all exceptions from the |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3294 # code above and try to correct it. If it's correctable its |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3295 # some form of Uniqueness Failure/Integrity Error otherwise |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3296 # undo the fixup and pass on the error. |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
3297 except Exception as e: # nosec |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3298 logger.info('Attempting to handle import exception ' |
| 6965 | 3299 'for id %s: %s' % (newid, e)) |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3300 |
|
7665
6f841a55eabf
fix: database error handling during import of a non-user item
John Rouillard <rouilj@ieee.org>
parents:
7211
diff
changeset
|
3301 classdb = self.db.getclass(self.classname) |
|
6f841a55eabf
fix: database error handling during import of a non-user item
John Rouillard <rouilj@ieee.org>
parents:
7211
diff
changeset
|
3302 keyname = classdb.getkey() |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3303 if has_node or not keyname: # Not an integrity error |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3304 raise |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
3305 self.db.restore_connection_on_error() |
|
7665
6f841a55eabf
fix: database error handling during import of a non-user item
John Rouillard <rouilj@ieee.org>
parents:
7211
diff
changeset
|
3306 activeid = classdb.lookup(d[keyname]) |
| 6965 | 3307 self.db.sql(retired_sql, (-1, activeid)) # clear the active node |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3308 # this can only happen on an addnode, so retry |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3309 try: |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3310 # if this raises an error, let it propagate upward |
| 6965 | 3311 self.db.addnode(self.classname, newid, d) # insert |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3312 except Exception: |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3313 # undo the database change |
| 6965 | 3314 self.db.sql(retired_sql, (0, activeid)) # clear active node |
| 3315 raise # propagate | |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3316 logger.info('Successfully handled import exception ' |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3317 'for id %s which conflicted with %s' % ( |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3318 newid, activeid)) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3319 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3320 # retire? |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3321 if retire: |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3322 self.db.sql(retired_sql, (newid, newid)) |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3323 |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3324 if activeid: |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3325 # unretire the active node |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3326 self.db.sql(retired_sql, ('0', activeid)) |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3327 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3328 return newid |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3329 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3330 def export_journals(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3331 """Export a class's journal - generate a list of lists of |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3332 CSV-able data: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3333 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3334 nodeid, date, user, action, params |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3335 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3336 No heading here - the columns are fixed. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3337 """ |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3338 properties = self.getprops() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3339 r = [] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3340 for nodeid in self.getnodeids(): |
| 6965 | 3341 for nodeid, date_, user, action, params in self.history( |
| 3342 nodeid, enforceperm=False, skipquiet=False): | |
| 3343 date_ = date_.get_tuple() | |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3344 if action == 'set': |
|
3365
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3345 export_data = {} |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
3346 for propname, value in params.items(): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3347 if propname not in properties: |
|
3365
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3348 # property no longer in the schema |
|
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3349 continue |
|
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3350 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3351 prop = properties[propname] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3352 # make sure the params are eval()'able |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3353 if value is None: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3354 pass |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3355 elif isinstance(prop, Date): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3356 value = value.get_tuple() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3357 elif isinstance(prop, Interval): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3358 value = value.get_tuple() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3359 elif isinstance(prop, Password): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3360 value = str(value) |
|
3365
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3361 export_data[propname] = value |
|
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3362 params = export_data |
|
3933
9095a4da67f9
Handle export and import of old trackers that have data attached to...
Richard Jones <richard@users.sourceforge.net>
parents:
3925
diff
changeset
|
3363 elif action == 'create' and params: |
|
9095a4da67f9
Handle export and import of old trackers that have data attached to...
Richard Jones <richard@users.sourceforge.net>
parents:
3925
diff
changeset
|
3364 # old tracker with data stored in the create! |
|
9095a4da67f9
Handle export and import of old trackers that have data attached to...
Richard Jones <richard@users.sourceforge.net>
parents:
3925
diff
changeset
|
3365 params = {} |
| 6965 | 3366 l = [nodeid, date_, user, action, params] |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3367 r.append(list(map(repr_export, l))) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3368 return r |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3369 |
| 6965 | 3370 |
|
2597
c86b2179085b
fix journal export of files to remove content from CSV files
Richard Jones <richard@users.sourceforge.net>
parents:
2586
diff
changeset
|
3371 class FileClass(hyperdb.FileClass, Class): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3372 """This class defines a large chunk of data. To support this, it has a |
| 1165 | 3373 mandatory String property "content" which is typically saved off |
| 3374 externally to the hyperdb. | |
| 3375 | |
| 3376 The default MIME type of this data is defined by the | |
| 3377 "default_mime_type" class attribute, which may be overridden by each | |
| 3378 node if the class defines a "type" String property. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3379 """ |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3380 def __init__(self, db, classname, **properties): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3381 """The newly-created class automatically includes the "content" |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3382 and "type" properties. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3383 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3384 if 'content' not in properties: |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3385 properties['content'] = hyperdb.String(indexme='yes') |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3386 if 'type' not in properties: |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3387 properties['type'] = hyperdb.String() |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3388 Class.__init__(self, db, classname, **properties) |
| 1165 | 3389 |
| 3390 def create(self, **propvalues): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3391 """ snaffle the file propvalue and store in a file |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3392 """ |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3393 # we need to fire the auditors now, or the content property won't |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3394 # be in propvalues for the auditors to play with |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3395 self.fireAuditors('create', None, propvalues) |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3396 |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3397 # now remove the content property so it's not stored in the db |
| 1165 | 3398 content = propvalues['content'] |
| 3399 del propvalues['content'] | |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3400 |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3401 # do the database create |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3402 newid = self.create_inner(**propvalues) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3403 |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3404 # figure the mime type |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3405 mime_type = propvalues.get('type', self.default_mime_type) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3406 |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3407 # and index! |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3408 if self.properties['content'].indexme: |
|
5544
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3409 index_content = content |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3410 if bytes != str and isinstance(content, bytes): |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3411 index_content = content.decode('utf-8', errors='ignore') |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3412 self.db.indexer.add_text((self.classname, newid, 'content'), |
| 6965 | 3413 index_content, mime_type) |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3414 |
|
3979
954971d612ee
Fire reactors after file storage is all done (patch [SF#2001243])
Richard Jones <richard@users.sourceforge.net>
parents:
3977
diff
changeset
|
3415 # store off the content as a file |
|
5492
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3416 self.db.storefile(self.classname, newid, None, bs2b(content)) |
|
3979
954971d612ee
Fire reactors after file storage is all done (patch [SF#2001243])
Richard Jones <richard@users.sourceforge.net>
parents:
3977
diff
changeset
|
3417 |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3418 # fire reactors |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3419 self.fireReactors('create', newid, None) |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3420 |
| 1165 | 3421 return newid |
| 3422 | |
| 3423 def get(self, nodeid, propname, default=_marker, cache=1): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3424 """ Trap the content propname and get it from the file |
|
1780
d2801a2b0a77
Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents:
1751
diff
changeset
|
3425 |
|
d2801a2b0a77
Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents:
1751
diff
changeset
|
3426 'cache' exists for backwards compatibility, and is not used. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3427 """ |
| 1165 | 3428 poss_msg = 'Possibly a access right configuration problem.' |
| 3429 if propname == 'content': | |
| 3430 try: | |
|
5492
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3431 return b2s(self.db.getfile(self.classname, nodeid, None)) |
|
5248
198b6e810c67
Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents:
5236
diff
changeset
|
3432 except IOError as strerror: |
| 1165 | 3433 # BUG: by catching this we donot see an error in the log. |
| 6965 | 3434 return 'ERROR reading file: %s%s\n%s\n%s' % ( |
| 1165 | 3435 self.classname, nodeid, poss_msg, strerror) |
| 6965 | 3436 except UnicodeDecodeError: |
|
5725
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3437 # if content is not text (e.g. jpeg file) we get |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3438 # unicode error trying to convert to string in python 3. |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3439 # trap it and supply an error message. Include md5sum |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3440 # of content as this string is included in the etag |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3441 # calculation of the object. |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3442 return ('%s%s is not text, retrieve using ' |
| 6965 | 3443 'binary_content property. mdsum: %s') % ( |
| 3444 self.classname, nodeid, | |
| 3445 md5(self.db.getfile( | |
| 3446 self.classname, | |
| 3447 nodeid, | |
| 3448 None)).hexdigest()) # nosec - bandit md5 use ok | |
|
5492
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3449 elif propname == 'binary_content': |
|
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3450 return self.db.getfile(self.classname, nodeid, None) |
|
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3451 |
|
2692
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
3452 if default is not _marker: |
|
1780
d2801a2b0a77
Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents:
1751
diff
changeset
|
3453 return Class.get(self, nodeid, propname, default) |
| 1165 | 3454 else: |
|
1780
d2801a2b0a77
Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents:
1751
diff
changeset
|
3455 return Class.get(self, nodeid, propname) |
| 1165 | 3456 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3457 def set(self, itemid, **propvalues): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3458 """ Snarf the "content" propvalue and update it in a file |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3459 """ |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3460 self.fireAuditors('set', itemid, propvalues) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3461 oldvalues = copy.deepcopy(self.db.getnode(self.classname, itemid)) |
| 1165 | 3462 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3463 # now remove the content property so it's not stored in the db |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3464 content = None |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3465 if 'content' in propvalues: |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3466 content = propvalues['content'] |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3467 del propvalues['content'] |
| 1165 | 3468 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3469 # do the database create |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3470 propvalues = self.set_inner(itemid, **propvalues) |
| 1165 | 3471 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3472 # do content? |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3473 if content: |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3474 # store and possibly index |
|
5492
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3475 self.db.storefile(self.classname, itemid, None, bs2b(content)) |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3476 if self.properties['content'].indexme: |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3477 mime_type = self.get(itemid, 'type', self.default_mime_type) |
|
5544
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3478 index_content = content |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3479 if bytes != str and isinstance(content, bytes): |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3480 index_content = content.decode('utf-8', errors='ignore') |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3481 self.db.indexer.add_text((self.classname, itemid, 'content'), |
| 6965 | 3482 index_content, mime_type) |
|
2640
b01eca163779
The "type" parameter is supposed to be optional
Richard Jones <richard@users.sourceforge.net>
parents:
2634
diff
changeset
|
3483 propvalues['content'] = content |
|
b01eca163779
The "type" parameter is supposed to be optional
Richard Jones <richard@users.sourceforge.net>
parents:
2634
diff
changeset
|
3484 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3485 # fire reactors |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3486 self.fireReactors('set', itemid, oldvalues) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3487 return propvalues |
| 1165 | 3488 |
|
2505
bdd112cf61ba
rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
3489 def index(self, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3490 """ Add (or refresh) the node to search indexes. |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3491 |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3492 Use the content-type property for the content property. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3493 """ |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3494 # find all the String properties that have indexme |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
3495 for prop, propclass in self.getprops().items(): |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3496 if prop == 'content' and propclass.indexme: |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3497 mime_type = self.get(nodeid, 'type', self.default_mime_type) |
|
5544
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3498 index_content = self.get(nodeid, 'binary_content') |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3499 if bytes != str and isinstance(index_content, bytes): |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3500 index_content = index_content.decode('utf-8', |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3501 errors='ignore') |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3502 self.db.indexer.add_text((self.classname, nodeid, 'content'), |
| 6965 | 3503 index_content, mime_type) |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3504 elif isinstance(propclass, hyperdb.String) and propclass.indexme: |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3505 # index them under (classname, nodeid, property) |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3506 try: |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3507 value = str(self.get(nodeid, prop)) |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3508 except IndexError: |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3509 # node has been destroyed |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3510 continue |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3511 self.db.indexer.add_text((self.classname, nodeid, prop), value) |
|
2505
bdd112cf61ba
rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
3512 |
| 6965 | 3513 |
| 1165 | 3514 # XXX deviation from spec - was called ItemClass |
| 3515 class IssueClass(Class, roundupdb.IssueClass): | |
| 3516 # Overridden methods: | |
| 3517 def __init__(self, db, classname, **properties): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3518 """The newly-created class automatically includes the "messages", |
| 1165 | 3519 "files", "nosy", and "superseder" properties. If the 'properties' |
| 3520 dictionary attempts to specify any of these properties or a | |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
3521 "creation", "creator", "activity" or "actor" property, a ValueError |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
3522 is raised. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3523 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3524 if 'title' not in properties: |
| 1165 | 3525 properties['title'] = hyperdb.String(indexme='yes') |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3526 if 'messages' not in properties: |
| 1165 | 3527 properties['messages'] = hyperdb.Multilink("msg") |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3528 if 'files' not in properties: |
| 1165 | 3529 properties['files'] = hyperdb.Multilink("file") |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3530 if 'nosy' not in properties: |
| 1165 | 3531 # note: journalling is turned off as it really just wastes |
| 3532 # space. this behaviour may be overridden in an instance | |
| 3533 properties['nosy'] = hyperdb.Multilink("user", do_journal="no") | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3534 if 'superseder' not in properties: |
| 1165 | 3535 properties['superseder'] = hyperdb.Multilink(classname) |
| 3536 Class.__init__(self, db, classname, **properties) | |
| 3537 | |
|
2692
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
3538 # vim: set et sts=4 sw=4 : |
