Mercurial > p > roundup > code
annotate roundup/backends/rdbms_common.py @ 8241:741ea8a86012
fix: issue2551374. Error handling for filter expressions.
Errors in filter expressions are now reported. The UI needs some work
but even the current code is helpful when debugging filter
expressions.
mlink_expr:
defines/raises ExpressionError(error string template,
context=dict())
raises ExpressionError when it detects errors when popping arguments
off stack
raises ExpressionError when more than one element left on the stack
before returning
also ruff fix to group boolean expression with parens
back_anydbm.py, rdbms_common.py:
catches ExpressionError, augments context with class and
attribute being searched. raises the exception
for both link and multilink relations
client.py
catches ExpressionError returning a basic error page. The page is a
dead end. There are no links or anything for the user to move
forward. The user has to go back, possibly refresh the page (because
the submit button may be disalbled) re-enter the query and try
again.
This needs to be improved.
test_liveserver.py
test the error page generated by client.py
db_test_base
unit tests for filter with too few arguments, too many arguments,
check all repr and str formats.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Mon, 30 Dec 2024 20:22:55 -0500 |
| parents | a9b136565838 |
| children | 82a26ea1afdf |
| rev | line source |
|---|---|
|
3554
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
1 # |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/) |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
3 # This module is free software, and you may redistribute it and/or modify |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
4 # under the same terms as Python, so long as this copyright message and |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
5 # disclaimer are retained in their original form. |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
6 # |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
10 # POSSIBILITY OF SUCH DAMAGE. |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
11 # |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS" |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
17 # |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
18 """ Relational database (SQL) backend common code. |
|
1244
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
19 |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
20 Basics: |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
21 |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
22 - map roundup classes to relational tables |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
23 - automatically detect schema changes and modify the table schemas |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
24 appropriately (we store the "database version" of the schema in the |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
25 database itself as the only row of the "schema" table) |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
26 - multilinks (which represent a many-to-many relationship) are handled through |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
27 intermediate tables |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
28 - journals are stored adjunct to the per-class tables |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
29 - table names and columns have "_" prepended so the names can't clash with |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
30 restricted names (like "order") |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
31 - retirement is determined by the __retired__ column being > 0 |
|
1244
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
32 |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
33 Database-specific changes may generally be pushed out to the overridable |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
34 sql_* methods, since everything else should be fairly generic. There's |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
35 probably a bit of work to be done if a database is used that actually |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
36 honors column typing, since the initial databases don't (sqlite stores |
|
1528
96cd422532ef
bye bye gadfly - you served your purpose well [SF#701127]
Richard Jones <richard@users.sourceforge.net>
parents:
1523
diff
changeset
|
37 everything as a string.) |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
38 |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
39 The schema of the hyperdb being mapped to the database is stored in the |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
40 database itself as a repr()'ed dictionary of information about each Class |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
41 that maps to a table. If that information differs from the hyperdb schema, |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
42 then we update it. We also store in the schema dict a version which |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
43 allows us to upgrade the database schema when necessary. See upgrade_db(). |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
44 |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
45 To force a unqiueness constraint on the key properties we put the item |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
46 id into the __retired__ column duing retirement (so it's 0 for "active" |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
47 items) and place a unqiueness constraint on key + __retired__. This is |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
48 particularly important for the users class where multiple users may |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
49 try to have the same username, with potentially many retired users with |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
50 the same name. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
51 """ |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
52 __docformat__ = 'restructuredtext' |
| 1165 | 53 |
| 54 # standard python modules | |
| 6965 | 55 import copy |
| 56 import datetime | |
| 57 import logging | |
| 58 import os | |
| 59 import re | |
| 60 import time | |
| 61 | |
| 62 from hashlib import md5 | |
| 1165 | 63 |
| 64 # roundup modules | |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
65 from roundup import hyperdb, date, password, roundupdb, security, support |
| 6965 | 66 from roundup.anypy.strings import b2s, bs2b, us2s, repr_export, eval_import |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
67 from roundup.backends.blobfiles import FileStorage |
|
5096
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
5086
diff
changeset
|
68 from roundup.backends.indexer_common import get_indexer |
|
6915
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
69 from roundup.backends.indexer_common import Indexer as CommonIndexer |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
70 from roundup.backends.sessions_rdbms import Sessions, OneTimeKeys |
|
1499
8ee69708da0c
added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1496
diff
changeset
|
71 from roundup.date import Range |
| 6965 | 72 from roundup.hyperdb import String, Password, Date, Interval, Link, \ |
| 73 Multilink, DatabaseError, Boolean, Number, Integer | |
| 74 from roundup.i18n import _ | |
|
8241
741ea8a86012
fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents:
7936
diff
changeset
|
75 from roundup.mlink_expr import compile_expression, ExpressionError |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
76 |
|
2692
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
77 # dummy value meaning "argument not passed" |
|
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
78 _marker = [] |
|
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
79 |
| 6965 | 80 # python 3 doesn't have a unicode type |
| 81 try: | |
| 82 unicode # noqa: F821 | |
| 83 except NameError: | |
| 84 unicode = str | |
| 85 | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
86 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
87 def _num_cvt(num): |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
88 num = str(num) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
89 try: |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
90 return int(num) |
| 6002 | 91 except ValueError: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
92 return float(num) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
93 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
94 |
|
2472
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
95 def _bool_cvt(value): |
|
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
96 if value in ('TRUE', 'FALSE'): |
|
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
97 return {'TRUE': 1, 'FALSE': 0}[value] |
|
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
98 # assume it's a number returned from the db API |
|
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
99 return int(value) |
|
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
100 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
101 |
|
4473
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
102 def date_to_hyperdb_value(d): |
|
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
103 """ convert date d to a roundup date """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
104 if isinstance(d, datetime.datetime): |
|
4473
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
105 return date.Date(d) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
106 return date.Date(str(d).replace(' ', '.')) |
|
4473
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
107 |
|
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
108 |
|
2634
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
109 def connection_dict(config, dbnamestr=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
110 """ Used by Postgresql and MySQL to detemine the keyword args for |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
111 opening the database connection.""" |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
112 d = {} |
|
2634
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
113 if dbnamestr: |
|
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
114 d[dbnamestr] = config.RDBMS_NAME |
|
7696
4af0d235b570
feat(db): support using postgresql service connection file
John Rouillard <rouilj@ieee.org>
parents:
7665
diff
changeset
|
115 for name in ('host', 'port', 'password', 'user', 'service', |
|
4af0d235b570
feat(db): support using postgresql service connection file
John Rouillard <rouilj@ieee.org>
parents:
7665
diff
changeset
|
116 'read_default_group', 'read_default_file'): |
|
2634
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
117 cvar = 'RDBMS_'+name.upper() |
|
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
118 if config[cvar] is not None: |
|
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
119 d[name] = config[cvar] |
|
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
120 return d |
|
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
121 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
122 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
123 class IdListOptimizer: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
124 """ To prevent flooding the SQL parser of the underlaying |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
125 db engine with "x IN (1, 2, 3, ..., <large number>)" collapses |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
126 these cases to "x BETWEEN 1 AND <large number>". |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
127 """ |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
128 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
129 def __init__(self): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
130 self.ranges = [] |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
131 self.singles = [] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
132 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
133 def append(self, nid): |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
134 """ Invariant: nid are ordered ascending """ |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
135 if self.ranges: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
136 last = self.ranges[-1] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
137 if last[1] == nid-1: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
138 last[1] = nid |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
139 return |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
140 if self.singles: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
141 last = self.singles[-1] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
142 if last == nid-1: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
143 self.singles.pop() |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
144 self.ranges.append([last, nid]) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
145 return |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
146 self.singles.append(nid) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
147 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
148 def where(self, field, placeholder): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
149 ranges = self.ranges |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
150 singles = self.singles |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
151 |
| 6965 | 152 if not singles and not ranges: return "(1=0)", [] # noqa: E701 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
153 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
154 if ranges: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
155 between = '%s BETWEEN %s AND %s' % ( |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
156 field, placeholder, placeholder) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
157 stmnt = [between] * len(ranges) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
158 else: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
159 stmnt = [] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
160 if singles: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
161 stmnt.append('%s in (%s)' % ( |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
162 field, ','.join([placeholder]*len(singles)))) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
163 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
164 return '(%s)' % ' OR '.join(stmnt), sum(ranges, []) + singles |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
165 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
166 def __str__(self): |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
167 return "ranges: %r / singles: %r" % (self.ranges, self.singles) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
168 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
169 |
| 1165 | 170 class Database(FileStorage, hyperdb.Database, roundupdb.Database): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
171 """ Wrapper around an SQL database that presents a hyperdb interface. |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
172 |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
173 - some functionality is specific to the actual SQL database, hence |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
174 the sql_* methods that are NotImplemented |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
175 - we keep a cache of the latest N row fetches (where N is |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
176 configurable). |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
177 """ |
| 1165 | 178 def __init__(self, config, journaltag=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
179 """ Open the database and load the schema from it. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
180 """ |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3872
diff
changeset
|
181 FileStorage.__init__(self, config.UMASK) |
|
6658
408fd477761f
Add i18n object to roundupdb.Database
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6599
diff
changeset
|
182 roundupdb.Database.__init__(self) |
| 1165 | 183 self.config, self.journaltag = config, journaltag |
| 184 self.dir = config.DATABASE | |
| 185 self.classes = {} | |
|
6915
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
186 # Assign the indexer base class here. During schema |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
187 # generation in open_connection, the min/max size for FTS |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
188 # tokens is used when creating the database tables for |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
189 # indexer=native full text search. These tables are always |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
190 # created as part of the schema so that the admin can choose |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
191 # indexer=native at some later date and "things will just |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
192 # work" (TM). |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
193 # |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
194 # We would like to use get_indexer() to return the real |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
195 # indexer class. However indexer=native-fts for postgres |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
196 # requires a database connection (conn) to be defined when |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
197 # calling get_indexer. The call to open_connection creates the |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
198 # conn but also creates the schema if it is missing. |
|
6918
cb2ed1e8c852
Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents:
6915
diff
changeset
|
199 if self.config['INDEXER'] != 'native-fts': |
|
cb2ed1e8c852
Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents:
6915
diff
changeset
|
200 self.indexer = get_indexer(config, self) |
|
cb2ed1e8c852
Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents:
6915
diff
changeset
|
201 else: |
|
cb2ed1e8c852
Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents:
6915
diff
changeset
|
202 self.indexer = CommonIndexer(self) |
| 1165 | 203 self.security = security.Security(self) |
| 204 | |
| 205 # additional transaction support for external files and the like | |
| 206 self.transactions = [] | |
| 207 | |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
208 # keep a cache of the N most recently retrieved rows of any kind |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
209 # (classname, nodeid) = row |
|
4074
e039f3cbbb96
Make RDBMS cache-size configurable.
Stefan Seefeld <stefan@seefeld.name>
parents:
4062
diff
changeset
|
210 self.cache_size = config.RDBMS_CACHE_SIZE |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
211 self.clearCache() |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
212 self.stats = {'cache_hits': 0, 'cache_misses': 0, 'get_items': 0, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
213 'filtering': 0} |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
214 |
|
5041
5251e97b1de0
Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents:
4995
diff
changeset
|
215 # make sure the database directory exists |
|
5251e97b1de0
Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents:
4995
diff
changeset
|
216 if not os.path.isdir(self.config.DATABASE): |
|
5251e97b1de0
Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents:
4995
diff
changeset
|
217 os.makedirs(self.config.DATABASE) |
|
5251e97b1de0
Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents:
4995
diff
changeset
|
218 |
|
1333
80d27b7d6db5
implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents:
1304
diff
changeset
|
219 # database lock |
|
80d27b7d6db5
implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents:
1304
diff
changeset
|
220 self.lockfile = None |
|
80d27b7d6db5
implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents:
1304
diff
changeset
|
221 |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
222 # Uppercase to not collide with Class names |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
223 self.Session = None |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
224 self.Otk = None |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
225 |
| 1165 | 226 # open a connection to the database, creating the "conn" attribute |
|
2082
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
227 self.open_connection() |
| 1165 | 228 |
|
6915
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
229 # If indexer is native-fts, conn to db must be available. |
|
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
230 # so we set the real self.indexer value here, after db is open. |
|
6918
cb2ed1e8c852
Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents:
6915
diff
changeset
|
231 if self.config['INDEXER'] == 'native-fts': |
|
cb2ed1e8c852
Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents:
6915
diff
changeset
|
232 self.indexer = get_indexer(config, self) |
|
6915
9ff091537f43
postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
233 |
|
1181
49aebf5a8691
some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents:
1176
diff
changeset
|
234 def clearCache(self): |
|
49aebf5a8691
some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents:
1176
diff
changeset
|
235 self.cache = {} |
|
49aebf5a8691
some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents:
1176
diff
changeset
|
236 self.cache_lru = [] |
|
4652
dfbc0cfa9811
Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4517
diff
changeset
|
237 # upcall is necessary! |
|
dfbc0cfa9811
Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4517
diff
changeset
|
238 roundupdb.Database.clearCache(self) |
|
1181
49aebf5a8691
some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents:
1176
diff
changeset
|
239 |
|
2082
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
240 def getSessionManager(self): |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
241 if not self.Session: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
242 self.Session = Sessions(self) |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
243 return self.Session |
|
2082
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
244 |
|
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
245 def getOTKManager(self): |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
246 if not self.Otk: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
247 self.Otk = OneTimeKeys(self) |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
248 return self.Otk |
|
2082
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
249 |
|
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
250 def open_connection(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
251 """ Open a connection to the database, creating it if necessary. |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
252 |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
253 Must call self.load_dbschema() |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
254 """ |
|
5236
0a20e1965f4b
Pylint checks bark about:
John Rouillard <rouilj@ieee.org>
parents:
5232
diff
changeset
|
255 raise NotImplementedError |
| 1165 | 256 |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
257 def sql(self, sql, args=None, cursor=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
258 """ Execute the sql with the optional args. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
259 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
260 self.log_debug('SQL %r %r' % (sql, args)) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
261 if not cursor: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
262 cursor = self.cursor |
| 1165 | 263 if args: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
264 cursor.execute(sql, args) |
| 1165 | 265 else: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
266 cursor.execute(sql) |
| 1165 | 267 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
268 def sql_fetchone(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
269 """ Fetch a single row. If there's nothing to fetch, return None. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
270 """ |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
271 return self.cursor.fetchone() |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
272 |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
273 def sql_fetchall(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
274 """ Fetch all rows. If there's nothing to fetch, return []. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
275 """ |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
276 return self.cursor.fetchall() |
| 1165 | 277 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
278 def sql_fetchiter(self): |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
279 """ Fetch all row as a generator |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
280 """ |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
281 while True: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
282 row = self.cursor.fetchone() |
| 6965 | 283 if not row: break # noqa: E701 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
284 yield row |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
285 |
|
4849
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
286 def search_stringquote(self, value): |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
287 """ Quote a search string to escape magic search characters |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
288 '%' and '_', also need to quote '\' (first) |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
289 Then put '%' around resulting string for LIKE (or ILIKE) operator |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
290 """ |
|
4849
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
291 v = value.replace('\\', '\\\\') |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
292 v = v.replace('%', '\\%') |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
293 v = v.replace('_', '\\_') |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
294 return '%' + v + '%' |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
295 |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
296 def init_dbschema(self): |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
297 self.database_schema = { |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
298 'version': self.current_db_version, |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
299 'tables': {} |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
300 } |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
301 |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
302 def load_dbschema(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
303 """ Load the schema definition that the database currently implements |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
304 """ |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
305 self.cursor.execute('select schema from schema') |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
306 schema = self.cursor.fetchone() |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
307 if schema: |
| 6002 | 308 # bandit - schema is trusted |
| 309 self.database_schema = eval(schema[0]) # nosec | |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
310 else: |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
311 self.database_schema = {} |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
312 |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
313 def save_dbschema(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
314 """ Save the schema definition that the database currently implements |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
315 """ |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
316 s = repr(self.database_schema) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
317 self.sql('delete from schema') |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
318 self.sql('insert into schema values (%s)' % self.arg, (s,)) |
| 1165 | 319 |
| 320 def post_init(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
321 """ Called once the schema initialisation has finished. |
| 1165 | 322 |
| 323 We should now confirm that the schema defined by our "classes" | |
| 324 attribute actually matches the schema in the database. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
325 """ |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
326 super(Database, self).post_init() |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
327 |
|
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
328 # upgrade the database for column type changes, new internal |
|
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
329 # tables, etc. |
|
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
330 save = self.upgrade_db() |
| 3553 | 331 |
| 332 # handle changes in the schema | |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
333 tables = self.database_schema['tables'] |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
334 for classname, spec in self.classes.items(): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
335 if classname in tables: |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
336 dbspec = tables[classname] |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
337 if self.update_class(spec, dbspec): |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
338 tables[classname] = spec.schema() |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
339 save = 1 |
| 1165 | 340 else: |
| 341 self.create_class(spec) | |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
342 tables[classname] = spec.schema() |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
343 save = 1 |
| 1165 | 344 |
|
7038
f524ddc27af8
replace for a,b in x.items() with for a in x.keys()
John Rouillard <rouilj@ieee.org>
parents:
7007
diff
changeset
|
345 for classname in list(tables.keys()): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
346 if classname not in self.classes: |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
347 self.drop_class(classname, tables[classname]) |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
348 del tables[classname] |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
349 save = 1 |
| 1165 | 350 |
| 351 # update the database version of the schema | |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
352 if save: |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
353 self.save_dbschema() |
| 1165 | 354 |
| 355 # reindex the db if necessary | |
| 356 if self.indexer.should_reindex(): | |
| 357 self.reindex() | |
| 358 | |
| 359 # commit | |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
2089
diff
changeset
|
360 self.sql_commit() |
| 1165 | 361 |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
362 # update this number when we need to make changes to the SQL structure |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
363 # of the backend database |
|
6806
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
364 current_db_version = 8 |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
365 db_version_updated = False |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
366 |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
367 def upgrade_db(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
368 """ Update the SQL database to reflect changes in the backend code. |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
369 |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
370 Return boolean whether we need to save the schema. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
371 """ |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
372 version = self.database_schema.get('version', 1) |
|
3993
1eba65a4b0aa
Don't run old code on newer database
Richard Jones <richard@users.sourceforge.net>
parents:
3979
diff
changeset
|
373 if version > self.current_db_version: |
|
1eba65a4b0aa
Don't run old code on newer database
Richard Jones <richard@users.sourceforge.net>
parents:
3979
diff
changeset
|
374 raise DatabaseError('attempting to run rev %d DATABASE with rev ' |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
375 '%d CODE!' % (version, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
376 self.current_db_version)) |
|
2081
fb4bf55b94d7
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2077
diff
changeset
|
377 if version == self.current_db_version: |
|
fb4bf55b94d7
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2077
diff
changeset
|
378 # nothing to do |
|
fb4bf55b94d7
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2077
diff
changeset
|
379 return 0 |
|
fb4bf55b94d7
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2077
diff
changeset
|
380 |
|
2429
e28a5d5d95a7
better upgrading
Richard Jones <richard@users.sourceforge.net>
parents:
2424
diff
changeset
|
381 if version < 2: |
| 4062 | 382 self.log_info('upgrade to version 2') |
|
2100
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
383 # change the schema structure |
|
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
384 self.database_schema = {'tables': self.database_schema} |
|
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
385 |
|
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
386 # version 1 didn't have the actor column (note that in |
|
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
387 # MySQL this will also transition the tables to typed columns) |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
388 self.add_new_columns_v2() |
|
2100
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
389 |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
390 # version 1 doesn't have the OTK, session and indexing in the |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
391 # database |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
392 self.create_version_2_tables() |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
393 |
|
2429
e28a5d5d95a7
better upgrading
Richard Jones <richard@users.sourceforge.net>
parents:
2424
diff
changeset
|
394 if version < 3: |
| 4062 | 395 self.log_info('upgrade to version 3') |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
396 self.fix_version_2_tables() |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
397 |
|
2721
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
398 if version < 4: |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
399 self.log_info('upgrade to version 4') |
|
2721
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
400 self.fix_version_3_tables() |
|
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
401 |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
402 if version < 5: |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
403 self.log_info('upgrade to version 5') |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
404 self.fix_version_4_tables() |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
405 |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
406 if version < 6: |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
407 self.log_info('upgrade to version 6') |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
408 self.fix_version_5_tables() |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
409 |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
410 if version < 7: |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
411 self.log_info('upgrade to version 7') |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
412 self.fix_version_6_tables() |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
413 |
|
6806
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
414 if version < 8: |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
415 self.log_info('upgrade to version 8') |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
416 self.fix_version_7_tables() |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
417 |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
418 self.database_schema['version'] = self.current_db_version |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
419 self.db_version_updated = True |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
420 return 1 |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
421 |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
422 def fix_version_2_tables(self): |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
423 # Default (used by sqlite): NOOP |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
424 pass |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
425 |
|
2721
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
426 def fix_version_3_tables(self): |
|
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
427 # drop the shorter VARCHAR OTK column and add a new TEXT one |
|
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
428 for name in ('otk', 'session'): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
429 self.sql('DELETE FROM %ss' % name) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
430 self.sql('ALTER TABLE %ss DROP %s_value' % (name, name)) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
431 self.sql('ALTER TABLE %ss ADD %s_value TEXT' % (name, name)) |
|
2721
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
432 |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
433 def fix_version_4_tables(self): |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
434 # note this is an explicit call now |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
435 c = self.cursor |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
436 for cn, klass in self.classes.items(): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
437 c.execute('select id from _%s where __retired__<>0' % (cn,)) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
438 for (id,) in c.fetchall(): |
| 6965 | 439 c.execute('update _%s set __retired__=%s where id=%s' % ( |
| 440 cn, | |
| 441 self.arg, self.arg), (id, id)) | |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
442 |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
443 if klass.key: |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
444 self.add_class_key_required_unique_constraint(cn, klass.key) |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
445 |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
446 def fix_version_5_tables(self): |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
447 # Default (used by sqlite, postgres): NOOP |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
448 # mysql overrides this because it is missing |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
449 # _<class>_key_retired_idx index used to make |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
450 # sure that the key is unique if it was created |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
451 # as version 5. |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
452 pass |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
453 |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
454 def fix_version_6_tables(self): |
|
6599
39189dd94f2c
issue2551189 - increase size of words in full text index.
John Rouillard <rouilj@ieee.org>
parents:
6588
diff
changeset
|
455 # Default (used by nobody): NOOP |
|
39189dd94f2c
issue2551189 - increase size of words in full text index.
John Rouillard <rouilj@ieee.org>
parents:
6588
diff
changeset
|
456 # Each backend mysql, postgres, sqlite overrides this |
|
39189dd94f2c
issue2551189 - increase size of words in full text index.
John Rouillard <rouilj@ieee.org>
parents:
6588
diff
changeset
|
457 # You would think ALTER commands would be the same but nooo. |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
458 pass |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
459 |
|
6806
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
460 def fix_version_7_tables(self): |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
461 # Default (used by sqlite): NOOP |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
462 # Each backend mysql, postgres overrides this |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
463 # You would think ALTER commands would be the same but nooo. |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
464 pass |
|
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6658
diff
changeset
|
465 |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
466 def _convert_journal_tables(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
467 """Get current journal table contents, drop the table and re-create""" |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
468 c = self.cursor |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
469 cols = ','.join('nodeid date tag action params'.split()) |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
470 for klass in self.classes.values(): |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
471 # slurp and drop |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
472 sql = 'select %s from %s__journal order by date' % \ |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
473 (cols, klass.classname) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
474 c.execute(sql) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
475 contents = c.fetchall() |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
476 self.drop_journal_table_indexes(klass.classname) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
477 c.execute('drop table %s__journal' % klass.classname) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
478 |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
479 # re-create and re-populate |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
480 self.create_journal_table(klass) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
481 a = self.arg |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
482 sql = 'insert into %s__journal (%s) values (%s,%s,%s,%s,%s)' % ( |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
483 klass.classname, cols, a, a, a, a, a) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
484 for row in contents: |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
485 # no data conversion needed |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
486 self.cursor.execute(sql, row) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
487 |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
488 def _convert_string_properties(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
489 """Get current Class tables that contain String properties, and |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
490 convert the VARCHAR columns to TEXT""" |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
491 c = self.cursor |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
492 for klass in self.classes.values(): |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
493 # slurp and drop |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
494 cols, mls = self.determine_columns(list(klass.properties.items())) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
495 scols = ','.join([i[0] for i in cols]) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
496 sql = 'select id,%s from _%s' % (scols, klass.classname) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
497 c.execute(sql) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
498 contents = c.fetchall() |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
499 self.drop_class_table_indexes(klass.classname, klass.getkey()) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
500 c.execute('drop table _%s' % klass.classname) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
501 |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
502 # re-create and re-populate |
|
2424
74474ec41050
argh! backwards compat
Richard Jones <richard@users.sourceforge.net>
parents:
2416
diff
changeset
|
503 self.create_class_table(klass, create_sequence=0) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
504 a = ','.join([self.arg for i in range(len(cols)+1)]) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
505 sql = 'insert into _%s (id,%s) values (%s)' % (klass.classname, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
506 scols, a) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
507 for row in contents: |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
508 l = [] |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
509 for entry in row: |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
510 # mysql will already be a string - psql needs "help" |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
511 if entry is not None and not isinstance(entry, type('')): |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
512 entry = str(entry) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
513 l.append(entry) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
514 self.cursor.execute(sql, l) |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
515 |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
516 def refresh_database(self): |
|
1951
767ff2a03eee
more unit tests to improve coverage (up from 85% to 88% for anydbm! :)
Richard Jones <richard@users.sourceforge.net>
parents:
1926
diff
changeset
|
517 self.post_init() |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
518 |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
519 def reindex(self, classname=None, show_progress=False): |
|
2650
d68a444fcce3
roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents:
2649
diff
changeset
|
520 if classname: |
|
d68a444fcce3
roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents:
2649
diff
changeset
|
521 classes = [self.getclass(classname)] |
|
d68a444fcce3
roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents:
2649
diff
changeset
|
522 else: |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
523 classes = list(self.classes.values()) |
|
2650
d68a444fcce3
roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents:
2649
diff
changeset
|
524 for klass in classes: |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
525 if show_progress: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
526 for nodeid in support.Progress('Reindex %s' % klass.classname, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
527 klass.list()): |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
528 klass.index(nodeid) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
529 else: |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
530 for nodeid in klass.list(): |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
531 klass.index(nodeid) |
|
3295
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3239
diff
changeset
|
532 self.indexer.save_index() |
| 1165 | 533 |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
534 def checkpoint_data(self): |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
535 """Call if you need to commit the state of the database |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
536 so you can try to fix the error rather than rolling back |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
537 |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
538 Needed for postgres when importing data. |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
539 """ |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
540 pass |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
541 |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
542 def restore_connection_on_error(self): |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
543 """on a database error/exception recover the db connection |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
544 if left in an unusable state (e.g. postgres requires |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
545 a rollback). |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
546 """ |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
547 pass |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
548 |
|
5175
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
549 # Used here in the generic backend to determine if the database |
|
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
550 # supports 'DOUBLE PRECISION' for floating point numbers. |
|
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
551 implements_double_precision = True |
|
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
552 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
553 hyperdb_to_sql_datatypes = { |
| 6965 | 554 hyperdb.String : 'TEXT', # noqa: E203 |
| 555 hyperdb.Date : 'TIMESTAMP', # noqa: E203 | |
| 556 hyperdb.Link : 'INTEGER', # noqa: E203 | |
| 557 hyperdb.Interval : 'VARCHAR(255)', # noqa: E203 | |
| 558 hyperdb.Password : 'VARCHAR(255)', # noqa: E203 | |
| 559 hyperdb.Boolean : 'BOOLEAN', # noqa: E203 | |
| 560 hyperdb.Number : 'REAL', # noqa: E203 | |
| 561 hyperdb.Integer : 'INTEGER', # noqa: E203 | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
562 } |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
563 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
564 def hyperdb_to_sql_datatype(self, propclass, prop=None): |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
565 |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
566 datatype = self.hyperdb_to_sql_datatypes.get(propclass) |
|
5175
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
567 if self.implements_double_precision and prop and \ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
568 isinstance(prop, Number) and prop.use_double: |
|
5175
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
569 datatype = 'DOUBLE PRECISION' |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
570 if datatype: |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
571 return datatype |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
572 |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
573 for k, v in self.hyperdb_to_sql_datatypes.items(): |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
574 if issubclass(propclass, k): |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
575 return v |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
576 |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
577 raise ValueError('%r is not a hyperdb property class' % propclass) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
578 |
| 1165 | 579 def determine_columns(self, properties): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
580 """ Figure the column names and multilink properties from the spec |
| 1165 | 581 |
| 582 "properties" is a list of (name, prop) where prop may be an | |
| 583 instance of a hyperdb "type" _or_ a string repr of that type. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
584 """ |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
585 cols = [ |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
586 ('_actor', self.hyperdb_to_sql_datatype(hyperdb.Link)), |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
587 ('_activity', self.hyperdb_to_sql_datatype(hyperdb.Date)), |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
588 ('_creator', self.hyperdb_to_sql_datatype(hyperdb.Link)), |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
589 ('_creation', self.hyperdb_to_sql_datatype(hyperdb.Date)), |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
590 ] |
| 1165 | 591 mls = [] |
| 592 # add the multilinks separately | |
| 593 for col, prop in properties: | |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
594 # Computed props are not in the db |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
595 if prop.computed: |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
596 continue |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
597 |
| 1165 | 598 if isinstance(prop, Multilink): |
| 599 mls.append(col) | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
600 continue |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
601 |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
602 if isinstance(prop, type('')): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
603 raise ValueError("string property spec!") |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
604 # and prop.find('Multilink') != -1: |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
605 # mls.append(col) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
606 |
|
5175
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
607 datatype = self.hyperdb_to_sql_datatype(prop.__class__, prop) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
608 cols.append(('_'+col, datatype)) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
609 |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
610 # Intervals stored as two columns |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
611 if isinstance(prop, Interval): |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
612 cols.append(('__'+col+'_int__', 'BIGINT')) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
613 |
| 1165 | 614 cols.sort() |
| 615 return cols, mls | |
| 616 | |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
617 def update_class(self, spec, old_spec, force=0): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
618 """ Determine the differences between the current spec and the |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
619 database version of the spec, and update where necessary. |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
620 |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
621 If 'force' is true, update the database anyway. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
622 """ |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
623 new_spec = spec.schema() |
|
1515
a516bbb9896b
fixed rdbms table update detection logic [SF#703297]
Richard Jones <richard@users.sourceforge.net>
parents:
1500
diff
changeset
|
624 new_spec[1].sort() |
|
a516bbb9896b
fixed rdbms table update detection logic [SF#703297]
Richard Jones <richard@users.sourceforge.net>
parents:
1500
diff
changeset
|
625 old_spec[1].sort() |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
626 if not force and new_spec == old_spec: |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
627 # no changes |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
628 return 0 |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
629 |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
630 if not self.config.RDBMS_ALLOW_ALTER: |
|
6133
e7cb0147e6fe
Make format strings use named placeholders rather than %s/%r
John Rouillard <rouilj@ieee.org>
parents:
6118
diff
changeset
|
631 raise DatabaseError(_( |
|
7750
216662fbaaee
fix(i18n): fix incorrect lookup of some translations
John Rouillard <rouilj@ieee.org>
parents:
7696
diff
changeset
|
632 'ALTER operation disallowed: %(old)r -> %(new)r.') % { |
|
216662fbaaee
fix(i18n): fix incorrect lookup of some translations
John Rouillard <rouilj@ieee.org>
parents:
7696
diff
changeset
|
633 'old': old_spec, 'new': new_spec}) |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
634 |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
635 logger = logging.getLogger('roundup.hyperdb.backend') |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
636 logger.info('update_class %s' % spec.classname) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
637 |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
638 logger.debug('old_spec %r' % (old_spec,)) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
639 logger.debug('new_spec %r' % (new_spec,)) |
| 1165 | 640 |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
641 # detect key prop change for potential index change |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
642 keyprop_changes = {} |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
643 if new_spec[0] != old_spec[0]: |
|
3554
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
644 if old_spec[0]: |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
645 keyprop_changes['remove'] = old_spec[0] |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
646 if new_spec[0]: |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
647 keyprop_changes['add'] = new_spec[0] |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
648 |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
649 # detect multilinks that have been removed, and drop their table |
|
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
650 old_has = {} |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
651 for name, prop in old_spec[1]: |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
652 old_has[name] = 1 |
|
6392
99455aeec1ae
Fix dropping of multilinks on schema change
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6332
diff
changeset
|
653 if name in spec.properties and not spec.properties[name].computed: |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
654 continue |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
655 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
656 if prop.find('Multilink to') != -1: |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
657 # first drop indexes. |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
658 self.drop_multilink_table_indexes(spec.classname, name) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
659 |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
660 # now the multilink table itself |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
661 sql = 'drop table %s_%s' % (spec.classname, name) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
662 else: |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
663 # if this is the key prop, drop the index first |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
664 if old_spec[0] == prop: |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
665 self.drop_class_table_key_index(spec.classname, name) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
666 del keyprop_changes['remove'] |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
667 |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
668 # drop the column |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
669 sql = 'alter table _%s drop column _%s' % (spec.classname, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
670 name) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
671 |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
672 self.sql(sql) |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
673 |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
674 # if we didn't remove the key prop just then, but the key prop has |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
675 # changed, we still need to remove the old index |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
676 if 'remove' in keyprop_changes: |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
677 self.drop_class_table_key_index(spec.classname, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
678 keyprop_changes['remove']) |
| 1165 | 679 |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
680 # add new columns |
|
2196
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
681 for propname, prop in new_spec[1]: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
682 if propname in old_has: |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
683 continue |
|
2196
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
684 prop = spec.properties[propname] |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
685 if isinstance(prop, Multilink): |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
686 self.create_multilink_table(spec, propname) |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
687 else: |
|
2727
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
688 # add the column |
|
5175
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
689 coltype = self.hyperdb_to_sql_datatype(prop.__class__, prop) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
690 sql = 'alter table _%s add column _%s %s' % ( |
|
2727
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
691 spec.classname, propname, coltype) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
692 self.sql(sql) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
693 |
|
2727
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
694 # extra Interval column |
|
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
695 if isinstance(prop, Interval): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
696 sql = 'alter table _%s add column __%s_int__ BIGINT' % ( |
|
2727
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
697 spec.classname, propname) |
|
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
698 self.sql(sql) |
|
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
699 |
|
2196
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
700 # if the new column is a key prop, we need an index! |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
701 if new_spec[0] == propname: |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
702 self.create_class_table_key_index(spec.classname, propname) |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
703 del keyprop_changes['add'] |
| 1165 | 704 |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
705 # if we didn't add the key prop just then, but the key prop has |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
706 # changed, we still need to add the new index |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
707 if 'add' in keyprop_changes: |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
708 self.create_class_table_key_index(spec.classname, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
709 keyprop_changes['add']) |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
710 |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
711 return 1 |
| 1165 | 712 |
|
3016
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
713 def determine_all_columns(self, spec): |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
714 """Figure out the columns from the spec and also add internal columns |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
715 |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
716 """ |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
717 cols, mls = self.determine_columns(list(spec.properties.items())) |
|
3082
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
718 |
|
3016
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
719 # add on our special columns |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
720 cols.append(('id', 'INTEGER PRIMARY KEY')) |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
721 cols.append(('__retired__', 'INTEGER DEFAULT 0')) |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
722 return cols, mls |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
723 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
724 def create_class_table(self, spec): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
725 """Create the class table for the given Class "spec". Creates the |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
726 indexes too.""" |
|
3016
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
727 cols, mls = self.determine_all_columns(spec) |
| 1165 | 728 |
| 729 # create the base table | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
730 scols = ','.join(['%s %s' % x for x in cols]) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
731 sql = 'create table _%s (%s)' % (spec.classname, scols) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
732 self.sql(sql) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
733 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
734 self.create_class_table_indexes(spec) |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
735 |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
736 return cols, mls |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
737 |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
738 def create_class_table_indexes(self, spec): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
739 """ create the class table for the given spec |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
740 """ |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
741 # create __retired__ index |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
742 index_sql2 = 'create index _%s_retired_idx on _%s(__retired__)' % ( |
|
1836
94e430ad4fdb
make the RDBMS common backend and the SQLite and MYsql backend create...
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1800
diff
changeset
|
743 spec.classname, spec.classname) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
744 self.sql(index_sql2) |
| 1165 | 745 |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
746 # create index for key property |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
747 if spec.key: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
748 index_sql3 = 'create index _%s_%s_idx on _%s(_%s)' % ( |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
749 spec.classname, spec.key, |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
750 spec.classname, spec.key) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
751 self.sql(index_sql3) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
752 |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
753 # and the unique index for key / retired(id) |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
754 self.add_class_key_required_unique_constraint(spec.classname, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
755 spec.key) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
756 |
|
2228
1d1362c54c94
Some doc / comment fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
2217
diff
changeset
|
757 # TODO: create indexes on (selected?) Link property columns, as |
|
1d1362c54c94
Some doc / comment fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
2217
diff
changeset
|
758 # they're more likely to be used for lookup |
|
1d1362c54c94
Some doc / comment fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
2217
diff
changeset
|
759 |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
760 def add_class_key_required_unique_constraint(self, cn, key): |
|
3969
905faf52a51f
fix mysql breakage in 1.4.2
Richard Jones <richard@users.sourceforge.net>
parents:
3967
diff
changeset
|
761 sql = '''create unique index _%s_key_retired_idx |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
762 on _%s(__retired__, _%s)''' % (cn, cn, key) |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
763 try: |
|
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
764 self.sql(sql) |
| 6002 | 765 except Exception: # nosec |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
766 # XXX catch e.g.: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
767 # _sqlite.DatabaseError: index _status_key_retired_idx |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
768 # already exists |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
769 pass |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
770 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
771 def drop_class_table_indexes(self, cn, key): |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
772 # drop the old table indexes first |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
773 l = ['_%s_id_idx' % cn, '_%s_retired_idx' % cn] |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
774 if key: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
775 l.append('_%s_%s_idx' % (cn, key)) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
776 |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
777 table_name = '_%s' % cn |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
778 for index_name in l: |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
779 if not self.sql_index_exists(table_name, index_name): |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
780 continue |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
781 index_sql = 'drop index '+index_name |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
782 self.sql(index_sql) |
| 1165 | 783 |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
784 def create_class_table_key_index(self, cn, key): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
785 """ create the class table for the given spec |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
786 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
787 sql = 'create index _%s_%s_idx on _%s(_%s)' % (cn, key, cn, key) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
788 self.sql(sql) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
789 |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
790 def drop_class_table_key_index(self, cn, key): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
791 table_name = '_%s' % cn |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
792 index_name = '_%s_%s_idx' % (cn, key) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
793 if self.sql_index_exists(table_name, index_name): |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
794 sql = 'drop index '+index_name |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
795 self.sql(sql) |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
796 |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
797 # and now the retired unique index too |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
798 index_name = '_%s_key_retired_idx' % cn |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
799 if self.sql_index_exists(table_name, index_name): |
|
3967
b1e81ad3fa6a
don't recompute the index name
Richard Jones <richard@users.sourceforge.net>
parents:
3963
diff
changeset
|
800 sql = 'drop index '+index_name |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
801 self.sql(sql) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
802 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
803 def create_journal_table(self, spec): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
804 """ create the journal table for a class given the spec and |
| 1165 | 805 already-determined cols |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
806 """ |
| 1165 | 807 # journal table |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
808 sql = """create table %s__journal ( |
|
3224
1fd11a9803bb
use backend datatype for journal timestamps.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3155
diff
changeset
|
809 nodeid integer, date %s, tag varchar(255), |
| 6965 | 810 action varchar(255), params text)""" % ( |
| 811 spec.classname, | |
| 812 self.hyperdb_to_sql_datatype(hyperdb.Date)) | |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
813 self.sql(sql) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
814 self.create_journal_table_indexes(spec) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
815 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
816 def create_journal_table_indexes(self, spec): |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
817 # index on nodeid |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
818 sql = 'create index %s_journ_idx on %s__journal(nodeid)' % ( |
|
1836
94e430ad4fdb
make the RDBMS common backend and the SQLite and MYsql backend create...
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1800
diff
changeset
|
819 spec.classname, spec.classname) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
820 self.sql(sql) |
| 1165 | 821 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
822 def drop_journal_table_indexes(self, classname): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
823 index_name = '%s_journ_idx' % classname |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
824 if not self.sql_index_exists('%s__journal' % classname, index_name): |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
825 return |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
826 index_sql = 'drop index '+index_name |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
827 self.sql(index_sql) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
828 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
829 def create_multilink_table(self, spec, ml): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
830 """ Create a multilink table for the "ml" property of the class |
| 1165 | 831 given by the spec |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
832 """ |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
833 # create the table |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
834 sql = 'create table %s_%s (linkid INTEGER, nodeid INTEGER)' % ( |
| 1165 | 835 spec.classname, ml) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
836 self.sql(sql) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
837 self.create_multilink_table_indexes(spec, ml) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
838 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
839 def create_multilink_table_indexes(self, spec, ml): |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
840 # create index on linkid |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
841 index_sql = 'create index %s_%s_l_idx on %s_%s(linkid)' % ( |
|
2100
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
842 spec.classname, ml, spec.classname, ml) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
843 self.sql(index_sql) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
844 |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
845 # create index on nodeid |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
846 index_sql = 'create index %s_%s_n_idx on %s_%s(nodeid)' % ( |
|
2100
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
847 spec.classname, ml, spec.classname, ml) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
848 self.sql(index_sql) |
| 1165 | 849 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
850 def drop_multilink_table_indexes(self, classname, ml): |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
851 l = [ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
852 '%s_%s_l_idx' % (classname, ml), |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
853 '%s_%s_n_idx' % (classname, ml) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
854 ] |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
855 table_name = '%s_%s' % (classname, ml) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
856 for index_name in l: |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
857 if not self.sql_index_exists(table_name, index_name): |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
858 continue |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
859 index_sql = 'drop index %s' % index_name |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
860 self.sql(index_sql) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
861 |
| 1165 | 862 def create_class(self, spec): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
863 """ Create a database table according to the given spec. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
864 """ |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
865 |
|
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
866 if not self.config.RDBMS_ALLOW_CREATE: |
|
7750
216662fbaaee
fix(i18n): fix incorrect lookup of some translations
John Rouillard <rouilj@ieee.org>
parents:
7696
diff
changeset
|
867 raise DatabaseError(_('CREATE operation disallowed: "%s".') % |
|
216662fbaaee
fix(i18n): fix incorrect lookup of some translations
John Rouillard <rouilj@ieee.org>
parents:
7696
diff
changeset
|
868 spec.classname) |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
869 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
870 cols, mls = self.create_class_table(spec) |
|
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
871 self.create_journal_table(spec) |
| 1165 | 872 |
| 873 # now create the multilink tables | |
| 874 for ml in mls: | |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
875 self.create_multilink_table(spec, ml) |
| 1165 | 876 |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
877 def drop_class(self, cn, spec): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
878 """ Drop the given table from the database. |
| 1165 | 879 |
| 880 Drop the journal and multilink tables too. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
881 """ |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
882 |
|
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
883 if not self.config.RDBMS_ALLOW_DROP: |
|
7750
216662fbaaee
fix(i18n): fix incorrect lookup of some translations
John Rouillard <rouilj@ieee.org>
parents:
7696
diff
changeset
|
884 raise DatabaseError(_('DROP operation disallowed: "%s".') % cn) |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
885 |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
886 properties = spec[1] |
| 1165 | 887 # figure the multilinks |
| 888 mls = [] | |
|
3924
21d3d7eeea8c
assorted pyflakes fixes
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3906
diff
changeset
|
889 for propname, prop in properties: |
| 1165 | 890 if isinstance(prop, Multilink): |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
891 mls.append(propname) |
| 1165 | 892 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
893 # drop class table and indexes |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
894 self.drop_class_table_indexes(cn, spec[0]) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
895 |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
896 self.drop_class_table(cn) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
897 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
898 # drop journal table and indexes |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
899 self.drop_journal_table_indexes(cn) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
900 sql = 'drop table %s__journal' % cn |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
901 self.sql(sql) |
| 1165 | 902 |
| 903 for ml in mls: | |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
904 # drop multilink table and indexes |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
905 self.drop_multilink_table_indexes(cn, ml) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
906 sql = 'drop table %s_%s' % (spec.classname, ml) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
907 self.sql(sql) |
| 1165 | 908 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
909 def drop_class_table(self, cn): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
910 sql = 'drop table _%s' % cn |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
911 self.sql(sql) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
912 |
| 1165 | 913 # |
| 914 # Classes | |
| 915 # | |
| 916 def __getattr__(self, classname): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
917 """ A convenient way of calling self.getclass(classname). |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
918 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
919 if classname in self.classes: |
| 1165 | 920 return self.classes[classname] |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
921 raise AttributeError(classname) |
| 1165 | 922 |
| 923 def addclass(self, cl): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
924 """ Add a Class to the hyperdatabase. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
925 """ |
| 1165 | 926 cn = cl.classname |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
927 if cn in self.classes: |
|
7750
216662fbaaee
fix(i18n): fix incorrect lookup of some translations
John Rouillard <rouilj@ieee.org>
parents:
7696
diff
changeset
|
928 raise ValueError(_('Class "%s" already defined.') % cn) |
| 1165 | 929 self.classes[cn] = cl |
| 930 | |
|
2076
2a4309450202
security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents:
2075
diff
changeset
|
931 # add default Edit and View permissions |
| 6965 | 932 self.security.addPermission( |
| 933 name="Create", klass=cn, | |
|
2649
1df7d4a41da4
Buncha stuff (sorry about the large checkin):
Richard Jones <richard@users.sourceforge.net>
parents:
2640
diff
changeset
|
934 description="User is allowed to create "+cn) |
| 6965 | 935 self.security.addPermission( |
| 936 name="Edit", klass=cn, | |
|
2076
2a4309450202
security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents:
2075
diff
changeset
|
937 description="User is allowed to edit "+cn) |
| 6965 | 938 self.security.addPermission( |
| 939 name="View", klass=cn, | |
|
2076
2a4309450202
security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents:
2075
diff
changeset
|
940 description="User is allowed to access "+cn) |
| 6965 | 941 self.security.addPermission( |
| 942 name="Retire", klass=cn, | |
|
4517
f8e85cf5f0fe
how odd, the Retire permission wasn't being registered;
Richard Jones <richard@users.sourceforge.net>
parents:
4490
diff
changeset
|
943 description="User is allowed to retire "+cn) |
|
2076
2a4309450202
security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents:
2075
diff
changeset
|
944 |
| 1165 | 945 def getclasses(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
946 """ Return a list of the names of all existing classes. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
947 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
948 return sorted(self.classes) |
| 1165 | 949 |
| 950 def getclass(self, classname): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
951 """Get the Class object representing a particular class. |
| 1165 | 952 |
| 953 If 'classname' is not a valid class name, a KeyError is raised. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
954 """ |
| 1165 | 955 try: |
| 956 return self.classes[classname] | |
| 957 except KeyError: | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
958 raise KeyError('There is no class called "%s"' % classname) |
| 1165 | 959 |
| 960 def clear(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
961 """Delete all database contents. |
| 1165 | 962 |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
963 Note: I don't commit here, which is different behaviour to the |
|
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
964 "nuke from orbit" behaviour in the dbs. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
965 """ |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
966 logging.getLogger('roundup.hyperdb.backend').info('clear') |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
967 for cn in self.classes: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
968 sql = 'delete from _%s' % cn |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
969 self.sql(sql) |
|
7007
05d81de0d92d
in clear() clear journal tables as well.
John Rouillard <rouilj@ieee.org>
parents:
6965
diff
changeset
|
970 sql = 'delete from %s__journal' % cn |
|
05d81de0d92d
in clear() clear journal tables as well.
John Rouillard <rouilj@ieee.org>
parents:
6965
diff
changeset
|
971 self.sql(sql) |
| 1165 | 972 |
| 973 # | |
| 974 # Nodes | |
| 975 # | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
976 |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
977 hyperdb_to_sql_value = { |
| 6965 | 978 hyperdb.String : str, # noqa: E203 |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
979 # fractional seconds by default |
| 6965 | 980 hyperdb.Date : lambda x: x.formal(sep=' ', # noqa: E203 |
| 981 sec='%06.3f'), | |
| 982 hyperdb.Link : int, # noqa: E203 | |
| 983 hyperdb.Interval : str, # noqa: E203 | |
| 984 hyperdb.Password : str, # noqa: E203 | |
| 985 hyperdb.Boolean : lambda x: x and 'TRUE' or 'FALSE', # noqa: E203 | |
| 986 hyperdb.Number : lambda x: x, # noqa: E203 | |
| 987 hyperdb.Integer : lambda x: x, # noqa: E203 | |
| 988 # used in journal marshalling | |
| 989 hyperdb.Multilink : lambda x: x, # noqa: E203 | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
990 } |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
991 |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
992 def to_sql_value(self, propklass): |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
993 |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
994 fn = self.hyperdb_to_sql_value.get(propklass) |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
995 if fn: |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
996 return fn |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
997 |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
998 for k, v in self.hyperdb_to_sql_value.items(): |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
999 if issubclass(propklass, k): |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1000 return v |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1001 |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1002 raise ValueError('%r is not a hyperdb property class' % propklass) |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1003 |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1004 def _cache_del(self, key): |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1005 del self.cache[key] |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1006 self.cache_lru.remove(key) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1007 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1008 def _cache_refresh(self, key): |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1009 self.cache_lru.remove(key) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1010 self.cache_lru.insert(0, key) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1011 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1012 def _cache_save(self, key, node): |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1013 self.cache[key] = node |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1014 # update the LRU |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1015 self.cache_lru.insert(0, key) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1016 if len(self.cache_lru) > self.cache_size: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1017 del self.cache[self.cache_lru.pop()] |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1018 |
| 1165 | 1019 def addnode(self, classname, nodeid, node): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1020 """ Add the specified node to its class's db. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1021 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1022 self.log_debug('addnode %s%s %r' % (classname, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1023 nodeid, node)) |
|
1528
96cd422532ef
bye bye gadfly - you served your purpose well [SF#701127]
Richard Jones <richard@users.sourceforge.net>
parents:
1523
diff
changeset
|
1024 |
|
96cd422532ef
bye bye gadfly - you served your purpose well [SF#701127]
Richard Jones <richard@users.sourceforge.net>
parents:
1523
diff
changeset
|
1025 # determine the column definitions and multilink tables |
| 1165 | 1026 cl = self.classes[classname] |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1027 cols, mls = self.determine_columns(list(cl.properties.items())) |
| 1165 | 1028 |
|
1189
23b8d1e87fe3
import fixes
Richard Jones <richard@users.sourceforge.net>
parents:
1187
diff
changeset
|
1029 # we'll be supplied these props if we're doing an import |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1030 values = node.copy() |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1031 if 'creator' not in values: |
|
1189
23b8d1e87fe3
import fixes
Richard Jones <richard@users.sourceforge.net>
parents:
1187
diff
changeset
|
1032 # add in the "calculated" properties (dupe so we don't affect |
|
23b8d1e87fe3
import fixes
Richard Jones <richard@users.sourceforge.net>
parents:
1187
diff
changeset
|
1033 # calling code's node assumptions) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1034 values['creation'] = values['activity'] = date.Date() |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1035 values['actor'] = values['creator'] = self.getuid() |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1036 |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1037 cl = self.classes[classname] |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1038 props = cl.getprops(protected=1) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1039 del props['id'] |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1040 |
| 1165 | 1041 # default the non-multilink columns |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1042 for col, prop in props.items(): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1043 if col not in values: |
|
1496
e6ac4e074acb
relaxed CVS importing (feature [SF#693277])
Richard Jones <richard@users.sourceforge.net>
parents:
1492
diff
changeset
|
1044 if isinstance(prop, Multilink): |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1045 values[col] = [] |
|
1496
e6ac4e074acb
relaxed CVS importing (feature [SF#693277])
Richard Jones <richard@users.sourceforge.net>
parents:
1492
diff
changeset
|
1046 else: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1047 values[col] = None |
| 1165 | 1048 |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1049 # clear this node out of the cache if it's in there |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1050 key = (classname, nodeid) |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1051 if key in self.cache: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1052 self._cache_del(key) |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1053 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1054 # figure the values to insert |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1055 vals = [] |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1056 for col, _dt in cols: |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1057 # this is somewhat dodgy.... |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1058 if col.endswith('_int__'): |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1059 # XXX eugh, this test suxxors |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1060 value = values[col[2:-6]] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1061 # this is an Interval special "int" column |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1062 if value is not None: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1063 vals.append(value.as_seconds()) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1064 else: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1065 vals.append(value) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1066 continue |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1067 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1068 prop = props[col[1:]] |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1069 value = values[col[1:]] |
|
2472
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
1070 if value is not None: |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1071 value = self.to_sql_value(prop.__class__)(value) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1072 vals.append(value) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1073 vals.append(nodeid) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1074 vals = tuple(vals) |
| 1165 | 1075 |
| 1076 # make sure the ordering is correct for column name -> column value | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1077 s = ','.join([self.arg for x in cols]) + ',%s' % self.arg |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1078 cols = ','.join([col for col, dt in cols]) + ',id' |
| 1165 | 1079 |
| 1080 # perform the inserts | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1081 sql = 'insert into _%s (%s) values (%s)' % (classname, cols, s) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1082 self.sql(sql, vals) |
| 1165 | 1083 |
| 1084 # insert the multilink rows | |
| 1085 for col in mls: | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1086 t = '%s_%s' % (classname, col) |
| 1165 | 1087 for entry in node[col]: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1088 sql = 'insert into %s (linkid, nodeid) values (%s,%s)' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1089 t, self.arg, self.arg) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1090 self.sql(sql, (entry, nodeid)) |
| 1165 | 1091 |
| 6965 | 1092 def setnode(self, classname, nodeid, values, multilink_changes=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1093 """ Change the specified node. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1094 """ |
| 6965 | 1095 if multilink_changes is None: multilink_changes = {} # noqa: E701 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1096 self.log_debug('setnode %s%s %r' % (classname, nodeid, values)) |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1097 |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1098 # clear this node out of the cache if it's in there |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1099 key = (classname, nodeid) |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1100 if key in self.cache: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1101 self._cache_del(key) |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1102 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1103 cl = self.classes[classname] |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1104 props = cl.getprops() |
| 1165 | 1105 |
| 1106 cols = [] | |
| 1107 mls = [] | |
| 1108 # add the multilinks separately | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1109 for col in values: |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1110 prop = props[col] |
| 1165 | 1111 if isinstance(prop, Multilink): |
| 1112 mls.append(col) | |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1113 elif isinstance(prop, Interval): |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1114 # Intervals store the seconds value too |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1115 cols.append(col) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1116 # extra leading '_' added by code below |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1117 cols.append('_' + col + '_int__') |
| 1165 | 1118 else: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1119 cols.append(col) |
| 1165 | 1120 cols.sort() |
| 1121 | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1122 # figure the values to insert |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1123 vals = [] |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1124 for col in cols: |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1125 if col.endswith('_int__'): |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1126 # XXX eugh, this test suxxors |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1127 # Intervals store the seconds value too |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1128 col = col[1:-6] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1129 prop = props[col] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1130 value = values[col] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1131 if value is None: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1132 vals.append(None) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1133 else: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1134 vals.append(value.as_seconds()) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1135 else: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1136 prop = props[col] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1137 value = values[col] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1138 if value is None: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1139 e = None |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1140 else: |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1141 e = self.to_sql_value(prop.__class__)(value) |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1142 vals.append(e) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1143 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1144 vals.append(int(nodeid)) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1145 vals = tuple(vals) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1146 |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1147 # if there's any updates to regular columns, do them |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1148 if cols: |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1149 # make sure the ordering is correct for column name -> column value |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1150 s = ','.join(['_%s=%s' % (x, self.arg) for x in cols]) |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1151 cols = ','.join(cols) |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1152 |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1153 # perform the update |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1154 sql = 'update _%s set %s where id=%s' % (classname, s, self.arg) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1155 self.sql(sql, vals) |
| 1165 | 1156 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1157 # we're probably coming from an import, not a change |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1158 if not multilink_changes: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1159 for name in mls: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1160 prop = props[name] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1161 value = values[name] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1162 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1163 t = '%s_%s' % (classname, name) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1164 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1165 # clear out previous values for this node |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1166 # XXX numeric ids |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1167 self.sql('delete from %s where nodeid=%s' % (t, self.arg), |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1168 (nodeid,)) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1169 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1170 # insert the values for this node |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1171 for entry in values[name]: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1172 sql = 'insert into %s (linkid, nodeid) values (%s,%s)' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1173 t, self.arg, self.arg) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1174 # XXX numeric ids |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1175 self.sql(sql, (entry, nodeid)) |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1176 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1177 # we have multilink changes to apply |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1178 for col, (add, remove) in multilink_changes.items(): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1179 tn = '%s_%s' % (classname, col) |
| 1165 | 1180 if add: |
| 6965 | 1181 sql = 'insert into %s (nodeid, linkid) values (%s,%s)' % ( |
| 1182 tn, self.arg, self.arg) | |
| 1165 | 1183 for addid in add: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1184 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1185 self.sql(sql, (int(nodeid), int(addid))) |
| 1165 | 1186 if remove: |
|
4039
f44ec45cedb4
Fuse multiple DELETE calls into one for multilinks.
Stefan Seefeld <stefan@seefeld.name>
parents:
4038
diff
changeset
|
1187 s = ','.join([self.arg]*len(remove)) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1188 sql = 'delete from %s where nodeid=%s and linkid in (%s)' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1189 tn, self.arg, s) |
|
4039
f44ec45cedb4
Fuse multiple DELETE calls into one for multilinks.
Stefan Seefeld <stefan@seefeld.name>
parents:
4038
diff
changeset
|
1190 # XXX numeric ids |
|
f44ec45cedb4
Fuse multiple DELETE calls into one for multilinks.
Stefan Seefeld <stefan@seefeld.name>
parents:
4038
diff
changeset
|
1191 self.sql(sql, [int(nodeid)] + remove) |
| 1165 | 1192 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1193 sql_to_hyperdb_value = { |
| 6965 | 1194 hyperdb.String : us2s, # noqa: E203 |
| 1195 hyperdb.Date : date_to_hyperdb_value, # noqa: E203 | |
| 1196 # hyperdb.Link : int, # XXX numeric ids | |
| 1197 hyperdb.Link : str, # noqa: E203 | |
| 1198 hyperdb.Interval : date.Interval, # noqa: E203 | |
| 1199 hyperdb.Password : lambda x: password.Password(encrypted=x), # noqa: E203 | |
| 1200 hyperdb.Boolean : _bool_cvt, # noqa: E203 | |
| 1201 hyperdb.Number : _num_cvt, # noqa: E203 | |
| 1202 hyperdb.Integer : int, # noqa: E203 | |
| 1203 # used in journal marshalling | |
| 1204 hyperdb.Multilink : lambda x: x, # noqa: E203 | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1205 } |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1206 |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1207 def to_hyperdb_value(self, propklass): |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1208 |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1209 fn = self.sql_to_hyperdb_value.get(propklass) |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1210 if fn: |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1211 return fn |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1212 |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1213 for k, v in self.sql_to_hyperdb_value.items(): |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1214 if issubclass(propklass, k): |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1215 return v |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1216 |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1217 raise ValueError('%r is not a hyperdb property class' % propklass) |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1218 |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1219 def _materialize_multilink(self, classname, nodeid, node, propname): |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1220 """ evaluation of single Multilink (lazy eval may have skipped this) |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1221 """ |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1222 if propname not in node: |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1223 prop = self.getclass(classname).properties[propname] |
| 6965 | 1224 tn = prop.table_name |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1225 lid = prop.linkid_name |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1226 nid = prop.nodeid_name |
| 6965 | 1227 w = '' |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1228 joi = '' |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1229 if prop.computed: |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1230 if isinstance(prop.rev_property, Link): |
| 6965 | 1231 w = ' and %s.__retired__=0' % tn |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1232 else: |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1233 tn2 = '_' + prop.classname |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1234 joi = ', %s' % tn2 |
| 6965 | 1235 w = ' and %s.%s=%s.id and %s.__retired__=0' % ( |
| 1236 tn, lid, tn2, tn2) | |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1237 cursor = self.sql_new_cursor(name='_materialize_multilink') |
| 6965 | 1238 sql = 'select %s from %s%s where %s=%s%s' % (lid, tn, joi, nid, |
| 1239 self.arg, w) | |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1240 self.sql(sql, (nodeid,), cursor) |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1241 # Reduce this to only the first row (the ID), this can save a |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1242 # lot of space for large query results (not using fetchall) |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1243 node[propname] = [str(x) for x in sorted(int(r[0]) for r in cursor)] |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1244 cursor.close() |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1245 |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1246 def _materialize_multilinks(self, classname, nodeid, node, props=None): |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1247 """ get all Multilinks of a node (lazy eval may have skipped this) |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1248 """ |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1249 cl = self.classes[classname] |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1250 props = props or [pn for (pn, p) in cl.properties.items() |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1251 if isinstance(p, Multilink)] |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1252 for propname in props: |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1253 if propname not in node: |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1254 self._materialize_multilink(classname, nodeid, node, propname) |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1255 |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1256 def getnode(self, classname, nodeid, fetch_multilinks=True): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1257 """ Get a node from the database. |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1258 For optimisation optionally we don't fetch multilinks |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1259 (lazy Multilinks). |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1260 But for internal database operations we need them. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1261 """ |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1262 # see if we have this node cached |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1263 key = (classname, nodeid) |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1264 if key in self.cache: |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1265 # push us back to the top of the LRU |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1266 self._cache_refresh(key) |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1267 if __debug__: |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1268 self.stats['cache_hits'] += 1 |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1269 # return the cached information |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1270 if fetch_multilinks: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1271 self._materialize_multilinks(classname, nodeid, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1272 self.cache[key]) |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1273 return self.cache[key] |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1274 |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1275 if __debug__: |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1276 self.stats['cache_misses'] += 1 |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1277 start_t = time.time() |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1278 |
| 1165 | 1279 # figure the columns we're fetching |
| 1280 cl = self.classes[classname] | |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1281 cols, mls = self.determine_columns(list(cl.properties.items())) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1282 scols = ','.join([col for col, dt in cols]) |
| 1165 | 1283 |
| 1284 # perform the basic property fetch | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1285 sql = 'select %s from _%s where id=%s' % (scols, classname, self.arg) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1286 self.sql(sql, (nodeid,)) |
| 1165 | 1287 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1288 values = self.sql_fetchone() |
| 1165 | 1289 if values is None: |
|
6197
2492e2e17371
Change text from no such <class> node <node id> to drop "node"
John Rouillard <rouilj@ieee.org>
parents:
6179
diff
changeset
|
1290 raise IndexError('no such %s %s' % (classname, nodeid)) |
| 1165 | 1291 |
| 1292 # make up the node | |
| 1293 node = {} | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1294 props = cl.getprops(protected=1) |
| 1165 | 1295 for col in range(len(cols)): |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1296 name = cols[col][0][1:] |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1297 if name.endswith('_int__'): |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1298 # XXX eugh, this test suxxors |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1299 # ignore the special Interval-as-seconds column |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1300 continue |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1301 value = values[col] |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1302 if value is not None: |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1303 value = self.to_hyperdb_value(props[name].__class__)(value) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1304 node[name] = value |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1305 |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1306 if fetch_multilinks and mls: |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1307 self._materialize_multilinks(classname, nodeid, node, mls) |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1308 |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1309 # save off in the cache |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1310 key = (classname, nodeid) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1311 self._cache_save(key, node) |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1312 |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1313 if __debug__: |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1314 self.stats['get_items'] += (time.time() - start_t) |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1315 |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1316 return node |
| 1165 | 1317 |
| 1318 def destroynode(self, classname, nodeid): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1319 """Remove a node from the database. Called exclusively by the |
| 1165 | 1320 destroy() method on Class. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1321 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1322 message = 'destroynode %s%s' % (classname, nodeid) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1323 logging.getLogger('roundup.hyperdb.backend').info(message) |
| 1165 | 1324 |
| 1325 # make sure the node exists | |
| 1326 if not self.hasnode(classname, nodeid): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1327 raise IndexError('%s has no node %s' % (classname, nodeid)) |
| 1165 | 1328 |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1329 # see if we have this node cached |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1330 if (classname, nodeid) in self.cache: |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1331 del self.cache[(classname, nodeid)] |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1332 |
| 1165 | 1333 # see if there's any obvious commit actions that we should get rid of |
| 1334 for entry in self.transactions[:]: | |
| 1335 if entry[1][:2] == (classname, nodeid): | |
| 1336 self.transactions.remove(entry) | |
| 1337 | |
| 1338 # now do the SQL | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1339 sql = 'delete from _%s where id=%s' % (classname, self.arg) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1340 self.sql(sql, (nodeid,)) |
| 1165 | 1341 |
| 1342 # remove from multilnks | |
| 1343 cl = self.getclass(classname) | |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1344 x, mls = self.determine_columns(list(cl.properties.items())) |
| 1165 | 1345 for col in mls: |
| 1346 # get the link ids | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1347 sql = 'delete from %s_%s where nodeid=%s' % (classname, col, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1348 self.arg) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
1349 self.sql(sql, (nodeid,)) |
| 1165 | 1350 |
| 1351 # remove journal entries | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1352 sql = 'delete from %s__journal where nodeid=%s' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1353 classname, self.arg) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1354 self.sql(sql, (nodeid,)) |
| 1165 | 1355 |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
1356 # cleanup any blob filestorage when we commit |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1357 self.transactions.append((FileStorage.destroy, (self, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1358 classname, nodeid))) |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
1359 |
| 1165 | 1360 def hasnode(self, classname, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1361 """ Determine if the database has a given node. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1362 """ |
|
5086
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1363 # nodeid (aka link) is sql type INTEGER. max positive value |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1364 # for INTEGER is 2^31-1 for Postgres and MySQL. The max |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1365 # positive value for SqLite is 2^63 -1, so arguably this check |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1366 # needs to adapt for the type of the RDBMS. For right now, |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1367 # choose lowest common denominator. |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1368 if int(nodeid) >= 2**31: |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1369 # value out of range return false |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1370 return 0 |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1371 |
| 4031 | 1372 # If this node is in the cache, then we do not need to go to |
| 1373 # the database. (We don't consider this an LRU hit, though.) | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1374 if (classname, nodeid) in self.cache: |
| 4031 | 1375 # Return 1, not True, to match the type of the result of |
| 1376 # the SQL operation below. | |
| 1377 return 1 | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1378 sql = 'select count(*) from _%s where id=%s' % (classname, self.arg) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1379 self.sql(sql, (nodeid,)) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1380 return int(self.cursor.fetchone()[0]) |
| 1165 | 1381 |
| 1382 def countnodes(self, classname): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1383 """ Count the number of nodes that exist for a particular Class. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1384 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1385 sql = 'select count(*) from _%s' % classname |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1386 self.sql(sql) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1387 return self.cursor.fetchone()[0] |
| 1165 | 1388 |
| 1389 def addjournal(self, classname, nodeid, action, params, creator=None, | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1390 creation=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1391 """ Journal the Action |
| 1165 | 1392 'action' may be: |
| 1393 | |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1394 'set' -- 'params' is a dictionary of property values |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1395 'create' -- 'params' is an empty dictionary as of |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1396 Wed Nov 06 11:38:43 2002 +0000 |
| 1165 | 1397 'link' or 'unlink' -- 'params' is (classname, nodeid, propname) |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1398 'retired' or 'restored' -- 'params' is None |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1399 |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1400 'creator' -- the user performing the action, which defaults to |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1401 the current user. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1402 """ |
| 1165 | 1403 # handle supply of the special journalling parameters (usually |
| 1404 # supplied on importing an existing database) | |
| 1405 if creator: | |
| 1406 journaltag = creator | |
| 1407 else: | |
|
1800
a3b1b1dcf639
Use getuid(), not figure_curuserid()
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
1794
diff
changeset
|
1408 journaltag = self.getuid() |
| 1165 | 1409 if creation: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1410 journaldate = creation |
| 1165 | 1411 else: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1412 journaldate = date.Date() |
| 1165 | 1413 |
| 1414 # create the journal entry | |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1415 cols = 'nodeid,date,tag,action,params' |
| 1165 | 1416 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1417 self.log_debug('addjournal %s%s %r %s %s %r' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1418 classname, nodeid, journaldate, journaltag, action, params)) |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1419 |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1420 # make the journalled data marshallable |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1421 if isinstance(params, type({})): |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1422 self._journal_marshal(params, classname) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1423 |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
1424 params = repr_export(params) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1425 |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1426 dc = self.to_sql_value(hyperdb.Date) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1427 journaldate = dc(journaldate) |
| 1165 | 1428 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1429 self.save_journal(classname, cols, nodeid, journaldate, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1430 journaltag, action, params) |
| 1165 | 1431 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1432 def setjournal(self, classname, nodeid, journal): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1433 """Set the journal to the "journal" list.""" |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1434 # clear out any existing entries |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1435 self.sql('delete from %s__journal where nodeid=%s' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1436 classname, self.arg), (nodeid,)) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1437 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1438 # create the journal entry |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1439 cols = 'nodeid,date,tag,action,params' |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1440 |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1441 dc = self.to_sql_value(hyperdb.Date) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1442 for nodeid, journaldate, journaltag, action, params in journal: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1443 self.log_debug('addjournal %s%s %r %s %s %r' % ( |
|
4059
ef0b4396888a
Enhance and simplify logging.
Stefan Seefeld <stefan@seefeld.name>
parents:
4044
diff
changeset
|
1444 classname, nodeid, journaldate, journaltag, action, |
|
ef0b4396888a
Enhance and simplify logging.
Stefan Seefeld <stefan@seefeld.name>
parents:
4044
diff
changeset
|
1445 params)) |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1446 |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1447 # make the journalled data marshallable |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1448 if isinstance(params, type({})): |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1449 self._journal_marshal(params, classname) |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
1450 params = repr_export(params) |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1451 |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1452 self.save_journal(classname, cols, nodeid, dc(journaldate), |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1453 journaltag, action, params) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1454 |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1455 def _journal_marshal(self, params, classname): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1456 """Convert the journal params values into safely repr'able and |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1457 eval'able values.""" |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1458 properties = self.getclass(classname).getprops() |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1459 for param, value in params.items(): |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1460 if not value: |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1461 continue |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1462 property = properties[param] |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1463 cvt = self.to_sql_value(property.__class__) |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1464 if isinstance(property, Password): |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1465 params[param] = cvt(value) |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1466 elif isinstance(property, Date): |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1467 params[param] = cvt(value) |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1468 elif isinstance(property, Interval): |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1469 params[param] = cvt(value) |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1470 elif isinstance(property, Boolean): |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1471 params[param] = cvt(value) |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1472 |
| 1165 | 1473 def getjournal(self, classname, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1474 """ get the journal for id |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1475 """ |
| 1165 | 1476 # make sure the node exists |
| 1477 if not self.hasnode(classname, nodeid): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1478 raise IndexError('%s has no node %s' % (classname, nodeid)) |
| 1165 | 1479 |
| 1480 cols = ','.join('nodeid date tag action params'.split()) | |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1481 journal = self.load_journal(classname, cols, nodeid) |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1482 |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1483 # now unmarshal the data |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1484 dc = self.to_hyperdb_value(hyperdb.Date) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1485 res = [] |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1486 properties = self.getclass(classname).getprops() |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1487 for nodeid, date_stamp, user, action, params in journal: |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
1488 params = eval_import(params) |
|
2248
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1489 if isinstance(params, type({})): |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1490 for param, value in params.items(): |
|
2248
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1491 if not value: |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1492 continue |
|
2731
7c472ed6babf
handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents:
2727
diff
changeset
|
1493 property = properties.get(param, None) |
|
7c472ed6babf
handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents:
2727
diff
changeset
|
1494 if property is None: |
|
7c472ed6babf
handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents:
2727
diff
changeset
|
1495 # deleted property |
|
7c472ed6babf
handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents:
2727
diff
changeset
|
1496 continue |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1497 cvt = self.to_hyperdb_value(property.__class__) |
|
2248
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1498 if isinstance(property, Password): |
|
4483
22bc0426e348
Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4480
diff
changeset
|
1499 params[param] = password.JournalPassword(value) |
|
2248
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1500 elif isinstance(property, Date): |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1501 params[param] = cvt(value) |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1502 elif isinstance(property, Interval): |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1503 params[param] = cvt(value) |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1504 elif isinstance(property, Boolean): |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1505 params[param] = cvt(value) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1506 # XXX numeric ids |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1507 res.append((str(nodeid), dc(date_stamp), user, action, params)) |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1508 return res |
| 1165 | 1509 |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1510 def save_journal(self, classname, cols, nodeid, journaldate, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1511 journaltag, action, params): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1512 """ Save the journal entry to the database |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1513 """ |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1514 entry = (nodeid, journaldate, journaltag, action, params) |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1515 |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1516 # do the insert |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1517 a = self.arg |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1518 sql = 'insert into %s__journal (%s) values (%s,%s,%s,%s,%s)' % ( |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1519 classname, cols, a, a, a, a, a) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1520 self.sql(sql, entry) |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1521 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1522 def load_journal(self, classname, cols, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1523 """ Load the journal from the database |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1524 """ |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1525 # now get the journal entries |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1526 sql = 'select %s from %s__journal where nodeid=%s order by date' % ( |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1527 cols, classname, self.arg) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1528 self.sql(sql, (nodeid,)) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1529 return self.cursor.fetchall() |
| 1165 | 1530 |
| 1531 def pack(self, pack_before): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1532 """ Delete all journal entries except "create" before 'pack_before'. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1533 """ |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1534 date_stamp = self.to_sql_value(Date)(pack_before) |
| 1165 | 1535 |
| 1536 # do the delete | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1537 for classname in self.classes: |
| 1165 | 1538 sql = "delete from %s__journal where date<%s and "\ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1539 "action<>'create'" % (classname, self.arg) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1540 self.sql(sql, (date_stamp,)) |
| 1165 | 1541 |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1542 def sql_commit(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1543 """ Actually commit to the database. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1544 """ |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1545 logging.getLogger('roundup.hyperdb.backend').info('commit') |
|
3687
ff9f4ca42454
Postgres backend allows transaction collisions to be ignored when...
Richard Jones <richard@users.sourceforge.net>
parents:
3685
diff
changeset
|
1546 |
| 1165 | 1547 self.conn.commit() |
| 1548 | |
|
2374
31cb1014300c
Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents:
2362
diff
changeset
|
1549 # open a new cursor for subsequent work |
|
31cb1014300c
Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents:
2362
diff
changeset
|
1550 self.cursor = self.conn.cursor() |
|
31cb1014300c
Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents:
2362
diff
changeset
|
1551 |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1552 def sql_new_cursor(self, conn=None, *args, **kw): |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1553 """ Create new cursor, this may need additional parameters for |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1554 performance optimization for different backends. |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1555 """ |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1556 if conn is None: |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1557 conn = self.conn |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1558 return conn.cursor() |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1559 |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1560 def commit(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1561 """ Commit the current transactions. |
| 1165 | 1562 |
| 1563 Save all data changed since the database was opened or since the | |
| 1564 last commit() or rollback(). | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1565 """ |
| 1165 | 1566 # commit the database |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1567 self.sql_commit() |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1568 |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1569 # session and otk are committed with the db but not the other |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1570 # way round |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1571 if self.Session: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1572 self.Session.commit() |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1573 if self.Otk: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1574 self.Otk.commit() |
| 1165 | 1575 |
| 1576 # now, do all the other transaction stuff | |
| 1577 for method, args in self.transactions: | |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1578 method(*args) |
| 1165 | 1579 |
|
3295
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3239
diff
changeset
|
1580 # save the indexer |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3239
diff
changeset
|
1581 self.indexer.save_index() |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3239
diff
changeset
|
1582 |
| 1165 | 1583 # clear out the transactions |
| 1584 self.transactions = [] | |
| 1585 | |
|
4448
2784c239e6c8
clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4430
diff
changeset
|
1586 # clear the cache: Don't carry over cached values from one |
|
2784c239e6c8
clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4430
diff
changeset
|
1587 # transaction to the next (there may be other changes from other |
|
2784c239e6c8
clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4430
diff
changeset
|
1588 # transactions) |
|
2784c239e6c8
clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4430
diff
changeset
|
1589 self.clearCache() |
|
2784c239e6c8
clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4430
diff
changeset
|
1590 |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1591 def sql_rollback(self): |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1592 self.conn.rollback() |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1593 |
| 1165 | 1594 def rollback(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1595 """ Reverse all actions from the current transaction. |
| 1165 | 1596 |
| 1597 Undo all the changes made since the database was opened or the last | |
| 1598 commit() or rollback() was performed. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1599 """ |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1600 logging.getLogger('roundup.hyperdb.backend').info('rollback') |
| 1165 | 1601 |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1602 self.sql_rollback() |
| 1165 | 1603 |
| 1604 # roll back "other" transaction stuff | |
| 1605 for method, args in self.transactions: | |
| 1606 # delete temporary files | |
| 1607 if method == self.doStoreFile: | |
| 1608 self.rollbackStoreFile(*args) | |
| 1609 self.transactions = [] | |
| 1610 | |
|
1492
2fc7d4a8c9e7
fixed sqlite rollback/caching bug [SF#689383]
Richard Jones <richard@users.sourceforge.net>
parents:
1484
diff
changeset
|
1611 # clear the cache |
|
2fc7d4a8c9e7
fixed sqlite rollback/caching bug [SF#689383]
Richard Jones <richard@users.sourceforge.net>
parents:
1484
diff
changeset
|
1612 self.clearCache() |
|
2fc7d4a8c9e7
fixed sqlite rollback/caching bug [SF#689383]
Richard Jones <richard@users.sourceforge.net>
parents:
1484
diff
changeset
|
1613 |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1614 def sql_close(self): |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1615 logging.getLogger('roundup.hyperdb.backend').info('close') |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1616 self.conn.close() |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1617 |
| 1165 | 1618 def close(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1619 """ Close off the connection. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1620 """ |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
2089
diff
changeset
|
1621 self.indexer.close() |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1622 self.sql_close() |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1623 if self.Session: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1624 self.Session.close() |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1625 self.Session = None |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1626 if self.Otk: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1627 self.Otk.close() |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1628 self.Otk = None |
| 1165 | 1629 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1630 |
| 1165 | 1631 # |
| 1632 # The base Class class | |
| 1633 # | |
| 1634 class Class(hyperdb.Class): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1635 """ The handle to a particular class of nodes in a hyperdatabase. |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1636 |
| 1165 | 1637 All methods except __repr__ and getnode must be implemented by a |
| 1638 concrete backend Class. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1639 """ |
|
4849
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
1640 # For many databases the LIKE operator ignores case. |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
1641 # Postgres and Oracle have an ILIKE operator to support this. |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
1642 # We define the default here, can be changed in derivative class |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
1643 case_insensitive_like = 'LIKE' |
| 1165 | 1644 |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
1645 # For some databases (mysql) the = operator for strings ignores case. |
|
7860
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
1646 # We define the default here, can be changed in derivative class. |
|
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
1647 # If set to any false value, self.get_case_sensitive_equal() is |
|
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
1648 # called to set its value. |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
1649 case_sensitive_equal = '=' |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
1650 |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1651 # Some DBs order NULL values last. Set this variable in the backend |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1652 # for prepending an order by clause for each attribute that causes |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1653 # correct sort order for NULLs. Examples: |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1654 # order_by_null_values = '(%s is not NULL)' |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1655 # order_by_null_values = 'notnull(%s)' |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1656 # The format parameter is replaced with the attribute. |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1657 order_by_null_values = None |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1658 |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1659 # Assuming DBs can do subselects, overwrite if they cannot. |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1660 supports_subselects = True |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1661 |
| 1165 | 1662 def schema(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1663 """ A dumpable version of the schema that we can store in the |
| 1165 | 1664 database |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1665 """ |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1666 return (self.key, |
| 6965 | 1667 [(x, repr(y)) for x, y in |
| 1668 self.properties.items() if not y.computed]) | |
| 1165 | 1669 |
| 1670 def enableJournalling(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1671 """Turn journalling on for this class |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1672 """ |
| 1165 | 1673 self.do_journal = 1 |
| 1674 | |
| 1675 def disableJournalling(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1676 """Turn journalling off for this class |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1677 """ |
| 1165 | 1678 self.do_journal = 0 |
| 1679 | |
|
7860
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
1680 def get_case_sensitive_equal(self): |
|
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
1681 """ For some databases (mysql) the = operator for strings ignores |
|
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
1682 case. We define the default here, can be changed in derivative class. |
|
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
1683 |
|
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
1684 It takes config as an argument because mysql has multiple collations. |
|
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
1685 The admin sets both the primary and case sensitive collation in |
|
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
1686 config.ini for mysql. |
|
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
1687 """ |
|
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
1688 raise ValueError("get_case_sensitive_equal called in error") |
|
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
1689 |
| 1165 | 1690 # Editing nodes: |
| 1691 def create(self, **propvalues): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1692 """ Create a new node of this class and return its id. |
| 1165 | 1693 |
| 1694 The keyword arguments in 'propvalues' map property names to values. | |
| 1695 | |
| 1696 The values of arguments must be acceptable for the types of their | |
| 1697 corresponding properties or a TypeError is raised. | |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1698 |
| 1165 | 1699 If this class has a key property, it must be present and its value |
| 1700 must not collide with other key strings or a ValueError is raised. | |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1701 |
| 1165 | 1702 Any other properties on this class that are missing from the |
| 1703 'propvalues' dictionary are set to None. | |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1704 |
| 1165 | 1705 If an id in a link or multilink property does not refer to a valid |
| 1706 node, an IndexError is raised. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1707 """ |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
1708 self.fireAuditors('create', None, propvalues) |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
1709 newid = self.create_inner(**propvalues) |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
1710 self.fireReactors('create', newid, None) |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
1711 return newid |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1712 |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
1713 def create_inner(self, **propvalues): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1714 """ Called by create, in-between the audit and react calls. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1715 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1716 if 'id' in propvalues: |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1717 raise KeyError('"id" is reserved') |
| 1165 | 1718 |
| 1719 if self.db.journaltag is None: | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1720 raise DatabaseError(_('Database open read-only')) |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1721 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1722 if ('creator' in propvalues or 'actor' in propvalues or |
| 6965 | 1723 'creation' in propvalues or 'activity' in propvalues): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1724 raise KeyError('"creator", "actor", "creation" and ' |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1725 '"activity" are reserved') |
| 1165 | 1726 |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1727 for p in propvalues: |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1728 prop = self.properties[p] |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1729 if prop.computed: |
| 6965 | 1730 raise KeyError('"%s" is a computed property' % p) |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1731 |
| 1165 | 1732 # new node's id |
| 1733 newid = self.db.newid(self.classname) | |
| 1734 | |
| 1735 # validate propvalues | |
| 5809 | 1736 num_re = re.compile(r'^\d+$') |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1737 for key, value in propvalues.items(): |
| 1165 | 1738 if key == self.key: |
| 1739 try: | |
| 1740 self.lookup(value) | |
| 1741 except KeyError: | |
| 1742 pass | |
| 1743 else: | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1744 raise ValueError('node with key "%s" exists' % value) |
| 1165 | 1745 |
| 1746 # try to handle this property | |
| 1747 try: | |
| 1748 prop = self.properties[key] | |
| 1749 except KeyError: | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1750 raise KeyError('"%s" has no property "%s"' % (self.classname, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1751 key)) |
| 1165 | 1752 |
| 1753 if value is not None and isinstance(prop, Link): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1754 if not isinstance(value, type('')): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1755 raise ValueError('link value must be String') |
| 1165 | 1756 link_class = self.properties[key].classname |
| 1757 # if it isn't a number, it's a key | |
| 1758 if not num_re.match(value): | |
| 1759 try: | |
| 1760 value = self.db.classes[link_class].lookup(value) | |
| 1761 except (TypeError, KeyError): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1762 raise IndexError('new property "%s": %s not a %s' % ( |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1763 key, value, link_class)) |
| 1165 | 1764 elif not self.db.getclass(link_class).hasnode(value): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1765 raise IndexError('%s has no node %s' % (link_class, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1766 value)) |
| 1165 | 1767 |
| 1768 # save off the value | |
| 1769 propvalues[key] = value | |
| 1770 | |
| 1771 # register the link with the newly linked node | |
| 1772 if self.do_journal and self.properties[key].do_journal: | |
| 1773 self.db.addjournal(link_class, value, 'link', | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1774 (self.classname, newid, key)) |
| 1165 | 1775 |
| 1776 elif isinstance(prop, Multilink): | |
|
3872
34128a809e22
Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3869
diff
changeset
|
1777 if value is None: |
|
34128a809e22
Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3869
diff
changeset
|
1778 value = [] |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1779 if not hasattr(value, '__iter__') or \ |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1780 isinstance(value, type('')): |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1781 raise TypeError('new property "%s" not an iterable of ids' |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1782 % key) |
| 1165 | 1783 # clean up and validate the list of links |
| 1784 link_class = self.properties[key].classname | |
| 1785 l = [] | |
| 1786 for entry in value: | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1787 if not isinstance(entry, type('')): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1788 raise ValueError('"%s" multilink value (%r) ' |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1789 'must contain Strings' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1790 key, value)) |
| 1165 | 1791 # if it isn't a number, it's a key |
| 1792 if not num_re.match(entry): | |
| 1793 try: | |
| 1794 entry = self.db.classes[link_class].lookup(entry) | |
| 1795 except (TypeError, KeyError): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1796 raise IndexError('new property "%s": %s not a %s' % ( |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1797 key, entry, self.properties[key].classname)) |
| 1165 | 1798 l.append(entry) |
| 1799 value = l | |
| 1800 propvalues[key] = value | |
| 1801 | |
| 1802 # handle additions | |
| 1803 for nodeid in value: | |
| 1804 if not self.db.getclass(link_class).hasnode(nodeid): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1805 raise IndexError('%s has no node %s' % (link_class, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1806 nodeid)) |
| 1165 | 1807 # register the link with the newly linked node |
| 1808 if self.do_journal and self.properties[key].do_journal: | |
| 1809 self.db.addjournal(link_class, nodeid, 'link', | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1810 (self.classname, newid, key)) |
| 1165 | 1811 |
| 1812 elif isinstance(prop, String): | |
| 6965 | 1813 if not isinstance(value, (str, unicode)): |
| 1814 raise TypeError('new property "%s" not a string' % key) | |
|
2892
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
1815 if prop.indexme: |
|
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
1816 self.db.indexer.add_text((self.classname, newid, key), |
| 6965 | 1817 value) |
| 1165 | 1818 |
| 1819 elif isinstance(prop, Password): | |
| 1820 if not isinstance(value, password.Password): | |
| 6965 | 1821 raise TypeError('new property "%s" not a Password' % key) |
| 1165 | 1822 |
| 1823 elif isinstance(prop, Date): | |
| 1824 if value is not None and not isinstance(value, date.Date): | |
| 6965 | 1825 raise TypeError('new property "%s" not a Date' % key) |
| 1165 | 1826 |
| 1827 elif isinstance(prop, Interval): | |
| 1828 if value is not None and not isinstance(value, date.Interval): | |
| 6965 | 1829 raise TypeError('new property "%s" not an Interval' % key) |
| 1165 | 1830 |
| 1831 elif value is not None and isinstance(prop, Number): | |
| 1832 try: | |
| 1833 float(value) | |
| 1834 except ValueError: | |
| 6965 | 1835 raise TypeError('new property "%s" not numeric' % key) |
| 1165 | 1836 |
|
5067
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1837 elif value is not None and isinstance(prop, Integer): |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1838 try: |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1839 int(value) |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1840 except ValueError: |
| 6965 | 1841 raise TypeError('new property "%s" not integer' % key) |
|
5067
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1842 |
| 1165 | 1843 elif value is not None and isinstance(prop, Boolean): |
| 1844 try: | |
| 1845 int(value) | |
| 1846 except ValueError: | |
| 6965 | 1847 raise TypeError('new property "%s" not boolean' % key) |
| 1165 | 1848 |
| 1849 # make sure there's data where there needs to be | |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1850 for key, prop in self.properties.items(): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1851 if key in propvalues: |
| 1165 | 1852 continue |
| 1853 if key == self.key: | |
| 6965 | 1854 raise ValueError('key property "%s" is required' % key) |
| 1165 | 1855 if isinstance(prop, Multilink): |
| 1856 propvalues[key] = [] | |
| 1857 else: | |
| 1858 propvalues[key] = None | |
| 1859 | |
| 1860 # done | |
| 1861 self.db.addnode(self.classname, newid, propvalues) | |
| 1862 if self.do_journal: | |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
1863 self.db.addjournal(self.classname, newid, ''"create", {}) |
| 1165 | 1864 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1865 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1866 return str(newid) |
| 1165 | 1867 |
| 1868 def get(self, nodeid, propname, default=_marker, cache=1): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1869 """Get the value of a property on an existing node of this class. |
| 1165 | 1870 |
| 1871 'nodeid' must be the id of an existing node of this class or an | |
| 1872 IndexError is raised. 'propname' must be the name of a property | |
| 1873 of this class or a KeyError is raised. | |
| 1874 | |
|
1780
d2801a2b0a77
Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents:
1751
diff
changeset
|
1875 'cache' exists for backwards compatibility, and is not used. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1876 """ |
| 1165 | 1877 if propname == 'id': |
| 1878 return nodeid | |
| 1879 | |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1880 # get the node's dict |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1881 d = self.db.getnode(self.classname, nodeid, fetch_multilinks=False) |
|
4476
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1882 # handle common case -- that property is in dict -- first |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1883 # if None and one of creator/creation actor/activity return None |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1884 if propname in d: |
| 6965 | 1885 r = d[propname] |
|
4476
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1886 # return copy of our list |
| 6965 | 1887 if isinstance(r, list): |
|
4476
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1888 return r[:] |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1889 if r is not None: |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1890 return r |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1891 elif propname in ('creation', 'activity', 'creator', 'actor'): |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1892 return r |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1893 |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1894 # propname not in d: |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1895 if propname == 'creation' or propname == 'activity': |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1896 return date.Date() |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1897 if propname == 'creator' or propname == 'actor': |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1898 return self.db.getuid() |
| 1165 | 1899 |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1900 # get the property (raises KeyError if invalid) |
| 1165 | 1901 prop = self.properties[propname] |
| 1902 | |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1903 # lazy evaluation of Multilink |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1904 if propname not in d and isinstance(prop, Multilink): |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1905 self.db._materialize_multilink(self.classname, nodeid, d, propname) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1906 |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1907 # handle there being no value in the table for the property |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1908 if propname not in d or d[propname] is None: |
|
2692
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
1909 if default is _marker: |
| 1165 | 1910 if isinstance(prop, Multilink): |
| 1911 return [] | |
| 1912 else: | |
| 1913 return None | |
| 1914 else: | |
| 1915 return default | |
| 1916 | |
| 1917 # don't pass our list to other code | |
| 1918 if isinstance(prop, Multilink): | |
| 1919 return d[propname][:] | |
| 1920 | |
| 1921 return d[propname] | |
| 1922 | |
| 1923 def set(self, nodeid, **propvalues): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1924 """Modify a property on an existing node of this class. |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1925 |
| 1165 | 1926 'nodeid' must be the id of an existing node of this class or an |
| 1927 IndexError is raised. | |
| 1928 | |
| 1929 Each key in 'propvalues' must be the name of a property of this | |
| 1930 class or a KeyError is raised. | |
| 1931 | |
| 1932 All values in 'propvalues' must be acceptable types for their | |
| 1933 corresponding properties or a TypeError is raised. | |
| 1934 | |
| 1935 If the value of the key property is set, it must not collide with | |
| 1936 other key strings or a ValueError is raised. | |
| 1937 | |
| 1938 If the value of a Link or Multilink property contains an invalid | |
| 1939 node id, a ValueError is raised. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1940 """ |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1941 self.fireAuditors('set', nodeid, propvalues) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1942 oldvalues = copy.deepcopy(self.db.getnode(self.classname, nodeid)) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1943 propvalues = self.set_inner(nodeid, **propvalues) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1944 self.fireReactors('set', nodeid, oldvalues) |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1945 return propvalues |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1946 |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1947 def set_inner(self, nodeid, **propvalues): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1948 """ Called by set, in-between the audit and react calls. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1949 """ |
| 1165 | 1950 if not propvalues: |
| 1951 return propvalues | |
| 1952 | |
| 6965 | 1953 if ('creator' in propvalues or 'actor' in propvalues or |
| 1954 'creation' in propvalues or 'activity' in propvalues): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1955 raise KeyError('"creator", "actor", "creation" and ' |
| 6965 | 1956 '"activity" are reserved') |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1957 |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1958 if 'id' in propvalues: |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1959 raise KeyError('"id" is reserved') |
| 1165 | 1960 |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1961 for p in propvalues: |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1962 prop = self.properties[p] |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1963 if prop.computed: |
| 6965 | 1964 raise KeyError('"%s" is a computed property' % p) |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1965 |
| 1165 | 1966 if self.db.journaltag is None: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1967 raise DatabaseError(_('Database open read-only')) |
| 1165 | 1968 |
| 1969 node = self.db.getnode(self.classname, nodeid) | |
| 1970 if self.is_retired(nodeid): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1971 raise IndexError('Requested item is retired') |
| 5809 | 1972 num_re = re.compile(r'^\d+$') |
| 1165 | 1973 |
|
3082
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
1974 # make a copy of the values dictionary - we'll modify the contents |
|
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
1975 propvalues = propvalues.copy() |
|
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
1976 |
| 1165 | 1977 # if the journal value is to be different, store it in here |
| 1978 journalvalues = {} | |
| 1979 | |
| 1980 # remember the add/remove stuff for multilinks, making it easier | |
| 1981 # for the Database layer to do its stuff | |
| 1982 multilink_changes = {} | |
| 1983 | |
|
5112
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
1984 # omit quiet properties from history/changelog |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
1985 quiet_props = [] |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
1986 |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1987 for propname, value in list(propvalues.items()): |
| 1165 | 1988 # check to make sure we're not duplicating an existing key |
| 1989 if propname == self.key and node[propname] != value: | |
| 1990 try: | |
| 1991 self.lookup(value) | |
| 1992 except KeyError: | |
| 1993 pass | |
| 1994 else: | |
| 6965 | 1995 raise ValueError('node with key "%s" exists' % value) |
| 1165 | 1996 |
| 1997 # this will raise the KeyError if the property isn't valid | |
| 1998 # ... we don't use getprops() here because we only care about | |
| 1999 # the writeable properties. | |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
2000 try: |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
2001 prop = self.properties[propname] |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
2002 except KeyError: |
| 6965 | 2003 raise KeyError('"%s" has no property named "%s"' % ( |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2004 self.classname, propname)) |
| 1165 | 2005 |
| 2006 # if the value's the same as the existing value, no sense in | |
| 2007 # doing anything | |
|
1304
61ad556cfc8d
working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents:
1295
diff
changeset
|
2008 current = node.get(propname, None) |
|
61ad556cfc8d
working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents:
1295
diff
changeset
|
2009 if value == current: |
| 1165 | 2010 del propvalues[propname] |
| 2011 continue | |
|
1304
61ad556cfc8d
working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents:
1295
diff
changeset
|
2012 journalvalues[propname] = current |
| 1165 | 2013 |
| 2014 # do stuff based on the prop type | |
| 2015 if isinstance(prop, Link): | |
| 2016 link_class = prop.classname | |
| 2017 # if it isn't a number, it's a key | |
| 6965 | 2018 if value is not None and not isinstance(value, str): |
| 2019 raise ValueError('property "%s" link value be a string' % ( | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2020 propname)) |
| 6965 | 2021 if isinstance(value, str) and not num_re.match(value): |
| 1165 | 2022 try: |
| 2023 value = self.db.classes[link_class].lookup(value) | |
| 2024 except (TypeError, KeyError): | |
| 6965 | 2025 raise IndexError('new property "%s": %s not a %s' % ( |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2026 propname, value, prop.classname)) |
| 1165 | 2027 |
| 2028 if (value is not None and | |
| 2029 not self.db.getclass(link_class).hasnode(value)): | |
| 6965 | 2030 raise IndexError('%s has no node %s' % (link_class, |
| 2031 value)) | |
| 1165 | 2032 |
| 2033 if self.do_journal and prop.do_journal: | |
| 2034 # register the unlink with the old linked node | |
| 2035 if node[propname] is not None: | |
| 6965 | 2036 self.db.addjournal( |
| 2037 link_class, node[propname], | |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
2038 ''"unlink", (self.classname, nodeid, propname)) |
| 1165 | 2039 |
| 2040 # register the link with the newly linked node | |
| 2041 if value is not None: | |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
2042 self.db.addjournal(link_class, value, ''"link", |
| 6965 | 2043 (self.classname, nodeid, propname)) |
| 1165 | 2044 |
| 2045 elif isinstance(prop, Multilink): | |
|
3872
34128a809e22
Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3869
diff
changeset
|
2046 if value is None: |
|
34128a809e22
Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3869
diff
changeset
|
2047 value = [] |
| 6965 | 2048 if not hasattr(value, '__iter__') or isinstance(value, str): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2049 raise TypeError('new property "%s" not an iterable of' |
| 6965 | 2050 ' ids' % propname) |
| 1165 | 2051 link_class = self.properties[propname].classname |
| 2052 l = [] | |
| 2053 for entry in value: | |
| 2054 # if it isn't a number, it's a key | |
| 6965 | 2055 if not isinstance(entry, str): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2056 raise ValueError('new property "%s" link value ' |
| 6965 | 2057 'must be a string' % propname) |
| 1165 | 2058 if not num_re.match(entry): |
| 2059 try: | |
| 2060 entry = self.db.classes[link_class].lookup(entry) | |
| 2061 except (TypeError, KeyError): | |
| 6965 | 2062 raise IndexError( |
| 2063 'new property "%s": %s not a %s' % ( | |
| 2064 propname, entry, | |
| 2065 self.properties[propname].classname)) | |
| 1165 | 2066 l.append(entry) |
| 2067 value = l | |
| 2068 propvalues[propname] = value | |
| 2069 | |
| 2070 # figure the journal entry for this property | |
| 2071 add = [] | |
| 2072 remove = [] | |
| 2073 | |
| 2074 # handle removals | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2075 if propname in node: |
| 1165 | 2076 l = node[propname] |
| 2077 else: | |
| 2078 l = [] | |
| 2079 for id in l[:]: | |
| 2080 if id in value: | |
| 2081 continue | |
| 2082 # register the unlink with the old linked node | |
| 2083 if self.do_journal and self.properties[propname].do_journal: | |
| 2084 self.db.addjournal(link_class, id, 'unlink', | |
| 6965 | 2085 (self.classname, nodeid, propname)) |
| 1165 | 2086 l.remove(id) |
| 2087 remove.append(id) | |
| 2088 | |
| 2089 # handle additions | |
| 2090 for id in value: | |
|
4095
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2091 if id in l: |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2092 continue |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2093 # We can safely check this condition after |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2094 # checking that this is an addition to the |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2095 # multilink since the condition was checked for |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2096 # existing entries at the point they were added to |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2097 # the multilink. Since the hasnode call will |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2098 # result in a SQL query, it is more efficient to |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2099 # avoid the check if possible. |
| 1165 | 2100 if not self.db.getclass(link_class).hasnode(id): |
| 6965 | 2101 raise IndexError('%s has no node %s' % (link_class, |
| 2102 id)) | |
| 1165 | 2103 # register the link with the newly linked node |
| 2104 if self.do_journal and self.properties[propname].do_journal: | |
| 2105 self.db.addjournal(link_class, id, 'link', | |
| 6965 | 2106 (self.classname, nodeid, propname)) |
| 1165 | 2107 l.append(id) |
| 2108 add.append(id) | |
| 2109 | |
| 2110 # figure the journal entry | |
| 2111 l = [] | |
| 2112 if add: | |
| 2113 l.append(('+', add)) | |
| 2114 if remove: | |
| 2115 l.append(('-', remove)) | |
| 2116 multilink_changes[propname] = (add, remove) | |
| 2117 if l: | |
| 2118 journalvalues[propname] = tuple(l) | |
| 2119 | |
| 2120 elif isinstance(prop, String): | |
| 6965 | 2121 if value is not None and not isinstance(value, (str, unicode)): |
| 2122 raise TypeError( | |
| 2123 'new property "%s" not a string' % propname) | |
|
2892
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
2124 if prop.indexme: |
| 6965 | 2125 if value is None: value = '' # noqa: E701 |
| 2126 self.db.indexer.add_text( | |
| 2127 (self.classname, nodeid, propname), value) | |
| 1165 | 2128 |
| 2129 elif isinstance(prop, Password): | |
| 2130 if not isinstance(value, password.Password): | |
| 6965 | 2131 raise TypeError( |
| 2132 'new property "%s" not a Password' % propname) | |
| 1165 | 2133 propvalues[propname] = value |
|
4483
22bc0426e348
Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4480
diff
changeset
|
2134 journalvalues[propname] = \ |
|
22bc0426e348
Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4480
diff
changeset
|
2135 current and password.JournalPassword(current) |
| 1165 | 2136 |
| 2137 elif value is not None and isinstance(prop, Date): | |
| 2138 if not isinstance(value, date.Date): | |
| 6965 | 2139 raise TypeError('new property "%s" not a Date' % propname) |
| 1165 | 2140 propvalues[propname] = value |
| 2141 | |
| 2142 elif value is not None and isinstance(prop, Interval): | |
| 2143 if not isinstance(value, date.Interval): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2144 raise TypeError('new property "%s" not an ' |
| 6965 | 2145 'Interval' % propname) |
| 1165 | 2146 propvalues[propname] = value |
| 2147 | |
| 2148 elif value is not None and isinstance(prop, Number): | |
| 2149 try: | |
| 2150 float(value) | |
| 2151 except ValueError: | |
| 6965 | 2152 raise TypeError('new property "%s" not numeric' % propname) |
| 1165 | 2153 |
|
5067
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2154 elif value is not None and isinstance(prop, Integer): |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2155 try: |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2156 int(value) |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2157 except ValueError: |
| 6965 | 2158 raise TypeError('new property "%s" not integer' % propname) |
|
5067
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2159 |
| 1165 | 2160 elif value is not None and isinstance(prop, Boolean): |
| 2161 try: | |
| 2162 int(value) | |
| 2163 except ValueError: | |
| 6965 | 2164 raise TypeError('new property "%s" not boolean' % propname) |
| 1165 | 2165 |
|
5112
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2166 # record quiet properties to omit from history/changelog |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2167 if prop.quiet: |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2168 quiet_props.append(propname) |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2169 |
| 1165 | 2170 # nothing to do? |
| 2171 if not propvalues: | |
| 2172 return propvalues | |
| 2173 | |
|
3082
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
2174 # update the activity time |
|
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
2175 propvalues['activity'] = date.Date() |
|
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
2176 propvalues['actor'] = self.db.getuid() |
|
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
2177 |
|
3328
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2178 # do the set |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
2179 self.db.setnode(self.classname, nodeid, propvalues, multilink_changes) |
| 1165 | 2180 |
|
3328
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2181 # remove the activity props now they're handled |
|
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2182 del propvalues['activity'] |
|
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2183 del propvalues['actor'] |
|
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2184 |
|
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2185 # journal the set |
| 1165 | 2186 if self.do_journal: |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
2187 self.db.addjournal(self.classname, nodeid, ''"set", journalvalues) |
| 1165 | 2188 |
|
5112
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2189 # remove quiet properties from output |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2190 for propname in quiet_props: |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2191 if propname in propvalues: |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2192 del propvalues[propname] |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2193 |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
2194 return propvalues |
| 1165 | 2195 |
| 2196 def retire(self, nodeid): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2197 """Retire a node. |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2198 |
| 1165 | 2199 The properties on the node remain available from the get() method, |
| 2200 and the node's id is never reused. | |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2201 |
| 1165 | 2202 Retired nodes are not returned by the find(), list(), or lookup() |
| 2203 methods, and other nodes may reuse the values of their key properties. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2204 """ |
| 1165 | 2205 if self.db.journaltag is None: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2206 raise DatabaseError(_('Database open read-only')) |
| 1165 | 2207 |
|
1345
618aa9c37d65
fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents:
1333
diff
changeset
|
2208 self.fireAuditors('retire', nodeid, None) |
|
618aa9c37d65
fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents:
1333
diff
changeset
|
2209 |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2210 # use the arg for __retired__ to cope with any odd database type |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2211 # conversion (hello, sqlite) |
| 6965 | 2212 sql = 'update _%s set __retired__=%s where id=%s' % ( |
| 2213 self.classname, self.db.arg, self.db.arg) | |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2214 self.db.sql(sql, (nodeid, nodeid)) |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2215 if self.do_journal: |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
2216 self.db.addjournal(self.classname, nodeid, ''"retired", None) |
| 1165 | 2217 |
|
1345
618aa9c37d65
fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents:
1333
diff
changeset
|
2218 self.fireReactors('retire', nodeid, None) |
|
618aa9c37d65
fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents:
1333
diff
changeset
|
2219 |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2220 def restore(self, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2221 """Restore a retired node. |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2222 |
|
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2223 Make node available for all operations like it was before retirement. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2224 """ |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2225 if self.db.journaltag is None: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2226 raise DatabaseError(_('Database open read-only')) |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2227 |
|
1523
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2228 node = self.db.getnode(self.classname, nodeid) |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2229 # check if key property was overrided |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2230 key = self.getkey() |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2231 try: |
| 6965 | 2232 self.lookup(node[key]) |
|
1523
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2233 except KeyError: |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2234 pass |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2235 else: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2236 raise KeyError("Key property (%s) of retired node clashes " |
| 6965 | 2237 "with existing one (%s)" % (key, node[key])) |
|
1523
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2238 |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2239 self.fireAuditors('restore', nodeid, None) |
|
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2240 # use the arg for __retired__ to cope with any odd database type |
|
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2241 # conversion (hello, sqlite) |
| 6965 | 2242 sql = 'update _%s set __retired__=%s where id=%s' % ( |
| 2243 self.classname, self.db.arg, self.db.arg) | |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
2244 self.db.sql(sql, (0, nodeid)) |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2245 if self.do_journal: |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
2246 self.db.addjournal(self.classname, nodeid, ''"restored", None) |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2247 |
|
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2248 self.fireReactors('restore', nodeid, None) |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2249 |
| 1165 | 2250 def is_retired(self, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2251 """Return true if the node is rerired |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2252 """ |
| 6965 | 2253 sql = 'select __retired__ from _%s where id=%s' % (self.classname, |
| 2254 self.db.arg) | |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
2255 self.db.sql(sql, (nodeid,)) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2256 return int(self.db.sql_fetchone()[0]) > 0 |
| 1165 | 2257 |
| 2258 def destroy(self, nodeid): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2259 """Destroy a node. |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2260 |
| 1165 | 2261 WARNING: this method should never be used except in extremely rare |
| 2262 situations where there could never be links to the node being | |
| 2263 deleted | |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
2264 |
| 1165 | 2265 WARNING: use retire() instead |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
2266 |
| 1165 | 2267 WARNING: the properties of this node will not be available ever again |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
2268 |
| 1165 | 2269 WARNING: really, use retire() instead |
| 2270 | |
| 2271 Well, I think that's enough warnings. This method exists mostly to | |
| 2272 support the session storage of the cgi interface. | |
| 2273 | |
| 2274 The node is completely removed from the hyperdb, including all journal | |
| 2275 entries. It will no longer be available, and will generally break code | |
| 2276 if there are any references to the node. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2277 """ |
| 1165 | 2278 if self.db.journaltag is None: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2279 raise DatabaseError(_('Database open read-only')) |
| 1165 | 2280 self.db.destroynode(self.classname, nodeid) |
| 2281 | |
| 2282 # Locating nodes: | |
| 2283 def hasnode(self, nodeid): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2284 """Determine if the given nodeid actually exists |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2285 """ |
| 1165 | 2286 return self.db.hasnode(self.classname, nodeid) |
| 2287 | |
| 2288 def setkey(self, propname): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2289 """Select a String property of this class to be the key property. |
| 1165 | 2290 |
| 2291 'propname' must be the name of a String property of this class or | |
| 2292 None, or a TypeError is raised. The values of the key property on | |
| 2293 all existing nodes must be unique or a ValueError is raised. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2294 """ |
| 1165 | 2295 prop = self.getprops()[propname] |
| 2296 if not isinstance(prop, String): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2297 raise TypeError('key properties must be String') |
| 1165 | 2298 self.key = propname |
| 2299 | |
| 2300 def getkey(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2301 """Return the name of the key property for this class or None.""" |
| 1165 | 2302 return self.key |
| 2303 | |
| 2304 def lookup(self, keyvalue): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2305 """Locate a particular node by its key property and return its id. |
| 1165 | 2306 |
| 2307 If this class has no key property, a TypeError is raised. If the | |
| 2308 'keyvalue' matches one of the values for the key property among | |
| 2309 the nodes in this class, the matching node's id is returned; | |
| 2310 otherwise a KeyError is raised. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2311 """ |
| 1165 | 2312 if not self.key: |
| 6965 | 2313 raise TypeError('No key property set for class %s' % |
| 2314 self.classname) | |
| 1165 | 2315 |
|
7936
a9b136565838
feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents:
7860
diff
changeset
|
2316 # special notation for looking up the current database user |
|
a9b136565838
feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents:
7860
diff
changeset
|
2317 if keyvalue == '@current_user' and self.classname == 'user': |
|
a9b136565838
feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents:
7860
diff
changeset
|
2318 keyvalue = self.db.user.get(self.db.getuid(), self.key) |
|
a9b136565838
feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents:
7860
diff
changeset
|
2319 |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2320 # use the arg to handle any odd database type conversion (hello, |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2321 # sqlite) |
| 6965 | 2322 sql = "select id from _%s where _%s=%s and __retired__=%s" % ( |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2323 self.classname, self.key, self.db.arg, self.db.arg) |
|
3977
732a37da3a10
Fix for postgres 8.3 compatibility (and bug) (patch [SF#2030479])
Richard Jones <richard@users.sourceforge.net>
parents:
3969
diff
changeset
|
2324 self.db.sql(sql, (str(keyvalue), 0)) |
| 1165 | 2325 |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
2326 # see if there was a result that's not retired |
|
1203
735adcbfc665
fix to SQL lookup() and retirement
Richard Jones <richard@users.sourceforge.net>
parents:
1195
diff
changeset
|
2327 row = self.db.sql_fetchone() |
|
735adcbfc665
fix to SQL lookup() and retirement
Richard Jones <richard@users.sourceforge.net>
parents:
1195
diff
changeset
|
2328 if not row: |
| 6965 | 2329 raise KeyError('No key (%s) value "%s" for "%s"' % ( |
| 2330 self.key, keyvalue, self.classname)) | |
| 1165 | 2331 |
| 2332 # return the id | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2333 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2334 return str(row[0]) |
| 1165 | 2335 |
| 2336 def find(self, **propspec): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2337 """Get the ids of nodes in this class which link to the given nodes. |
| 1165 | 2338 |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2339 'propspec' consists of keyword args propname=nodeid or |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2340 propname={nodeid:1, } |
| 1165 | 2341 'propname' must be the name of a property in this class, or a |
|
1912
2b0ab61db194
fixes for [SF#818339]
Richard Jones <richard@users.sourceforge.net>
parents:
1911
diff
changeset
|
2342 KeyError is raised. That property must be a Link or |
|
2b0ab61db194
fixes for [SF#818339]
Richard Jones <richard@users.sourceforge.net>
parents:
1911
diff
changeset
|
2343 Multilink property, or a TypeError is raised. |
| 1165 | 2344 |
|
3239
440f0a6a2e3c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3224
diff
changeset
|
2345 Any node in this class whose 'propname' property links to any of |
|
440f0a6a2e3c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3224
diff
changeset
|
2346 the nodeids will be returned. Examples:: |
|
440f0a6a2e3c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3224
diff
changeset
|
2347 |
|
440f0a6a2e3c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3224
diff
changeset
|
2348 db.issue.find(messages='1') |
| 1165 | 2349 db.issue.find(messages={'1':1,'3':1}, files={'7':1}) |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2350 """ |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2351 # shortcut |
| 1165 | 2352 if not propspec: |
| 2353 return [] | |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2354 |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2355 # validate the args |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2356 props = self.getprops() |
|
7038
f524ddc27af8
replace for a,b in x.items() with for a in x.keys()
John Rouillard <rouilj@ieee.org>
parents:
7007
diff
changeset
|
2357 for propname in propspec.keys(): |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2358 # check the prop is OK |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2359 prop = props[propname] |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2360 if not isinstance(prop, Link) and not isinstance(prop, Multilink): |
| 6965 | 2361 raise TypeError("'%s' not a Link/Multilink property" % |
| 2362 propname) | |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2363 |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2364 # first, links |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2365 a = self.db.arg |
|
2733
ef396596a24e
more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents:
2731
diff
changeset
|
2366 allvalues = () |
|
ef396596a24e
more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents:
2731
diff
changeset
|
2367 sql = [] |
|
1988
5660b89f8903
more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents:
1986
diff
changeset
|
2368 where = [] |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
2369 for prop, values in propspec.items(): |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2370 if not isinstance(props[prop], hyperdb.Link): |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2371 continue |
| 6965 | 2372 if isinstance(values, dict) and len(values) == 1: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2373 values = list(values)[0] |
| 6965 | 2374 if isinstance(values, str): |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2375 allvalues += (values,) |
| 6965 | 2376 where.append('_%s = %s' % (prop, a)) |
|
1563
e2a8ce4d2317
Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents:
1557
diff
changeset
|
2377 elif values is None: |
| 6965 | 2378 where.append('_%s is NULL' % prop) |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2379 else: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2380 values = list(values) |
|
2494
ea7fb2f416db
fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents:
2472
diff
changeset
|
2381 s = '' |
|
ea7fb2f416db
fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents:
2472
diff
changeset
|
2382 if None in values: |
|
ea7fb2f416db
fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents:
2472
diff
changeset
|
2383 values.remove(None) |
| 6965 | 2384 s = '_%s is NULL or ' % prop |
|
2494
ea7fb2f416db
fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents:
2472
diff
changeset
|
2385 allvalues += tuple(values) |
| 6965 | 2386 s += '_%s in (%s)' % (prop, ','.join([a]*len(values))) |
| 2387 where.append('(' + s + ')') | |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2388 if where: |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2389 allvalues = (0, ) + allvalues |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2390 sql.append("""select id from _%s where __retired__=%s |
| 6965 | 2391 and %s""" % (self.classname, a, ' and '.join(where))) |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2392 |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2393 # now multilinks |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
2394 for prop, values in propspec.items(): |
|
6151
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2395 p = props[prop] |
|
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2396 if not isinstance(p, hyperdb.Multilink): |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2397 continue |
|
1988
5660b89f8903
more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents:
1986
diff
changeset
|
2398 if not values: |
|
5660b89f8903
more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents:
1986
diff
changeset
|
2399 continue |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2400 allvalues += (0, ) |
|
6151
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2401 tn = p.table_name |
|
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2402 ln = p.linkid_name |
|
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2403 nn = p.nodeid_name |
|
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2404 cn = '_' + self.classname |
|
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2405 ret = '' |
|
6155
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2406 dis = '' |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2407 ord = '' |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2408 if p.rev_property: |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2409 if isinstance(p.rev_property, Link): |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2410 ret = 'and %s.__retired__=%s ' % (tn, a) |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2411 allvalues += (0, ) |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2412 dis = 'distinct ' |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2413 ord = ' order by %s.id' % cn |
| 6965 | 2414 if isinstance(values, str): |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2415 allvalues += (values,) |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2416 s = a |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2417 else: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2418 allvalues += tuple(values) |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2419 s = ','.join([a]*len(values)) |
|
6155
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2420 sql.append("""select %s%s.id from %s, %s where %s.__retired__=%s |
| 6965 | 2421 %sand %s.id = %s.%s and %s.%s in (%s)%s""" % ( |
| 2422 dis, cn, cn, tn, cn, a, ret, cn, tn, nn, tn, ln, s, ord)) | |
|
2733
ef396596a24e
more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents:
2731
diff
changeset
|
2423 |
|
ef396596a24e
more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents:
2731
diff
changeset
|
2424 if not sql: |
|
1988
5660b89f8903
more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents:
1986
diff
changeset
|
2425 return [] |
|
2733
ef396596a24e
more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents:
2731
diff
changeset
|
2426 sql = ' union '.join(sql) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
2427 self.db.sql(sql, allvalues) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2428 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2429 l = [str(x[0]) for x in self.db.sql_fetchall()] |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2430 return l |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2431 |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2432 def stringFind(self, **requirements): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2433 """Locate a particular node by matching a set of its String |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2434 properties in a caseless search. |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2435 |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2436 If the property is not a String property, a TypeError is raised. |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2437 |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2438 The return is a list of the id of all nodes that match. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2439 """ |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2440 where = [] |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2441 args = [] |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2442 for propname in requirements: |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2443 prop = self.properties[propname] |
|
1951
767ff2a03eee
more unit tests to improve coverage (up from 85% to 88% for anydbm! :)
Richard Jones <richard@users.sourceforge.net>
parents:
1926
diff
changeset
|
2444 if not isinstance(prop, String): |
| 6965 | 2445 raise TypeError("'%s' not a String property" % propname) |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2446 where.append(propname) |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2447 args.append(requirements[propname].lower()) |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2448 |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2449 # generate the where clause |
| 6965 | 2450 s = ' and '.join(['lower(_%s)=%s' % (col, |
| 2451 self.db.arg) for col in where]) | |
| 2452 sql = 'select id from _%s where %s and __retired__=%s' % ( | |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
2453 self.classname, s, self.db.arg) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2454 args.append(0) |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2455 self.db.sql(sql, tuple(args)) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2456 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2457 l = [str(x[0]) for x in self.db.sql_fetchall()] |
| 1165 | 2458 return l |
| 2459 | |
| 2460 def list(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2461 """ Return a list of the ids of the active nodes in this class. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2462 """ |
|
1484
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2463 return self.getnodeids(retired=0) |
|
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2464 |
|
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2465 def getnodeids(self, retired=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2466 """ Retrieve all the ids of the nodes for a particular Class. |
|
1484
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2467 |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2468 Set retired=None to get all nodes. Otherwise it'll get all the |
|
1484
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2469 retired or non-retired nodes, depending on the flag. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2470 """ |
|
1707
3d627e34f18e
sqlite backend now passes all tests under 2.3.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1599
diff
changeset
|
2471 # flip the sense of the 'retired' flag if we don't want all of them |
|
1484
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2472 if retired is not None: |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2473 args = (0, ) |
|
1719
eeb167fb8faf
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
1707
diff
changeset
|
2474 if retired: |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2475 compare = '>' |
|
1719
eeb167fb8faf
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
1707
diff
changeset
|
2476 else: |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2477 compare = '=' |
| 6965 | 2478 sql = 'select id from _%s where __retired__%s%s' % ( |
| 2479 self.classname, compare, self.db.arg) | |
|
1539
800b5896e14a
fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents:
1528
diff
changeset
|
2480 else: |
|
800b5896e14a
fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents:
1528
diff
changeset
|
2481 args = () |
| 6965 | 2482 sql = 'select id from _%s' % self.classname |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
2483 self.db.sql(sql, args) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2484 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2485 ids = [str(x[0]) for x in self.db.cursor.fetchall()] |
|
1539
800b5896e14a
fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents:
1528
diff
changeset
|
2486 return ids |
| 1165 | 2487 |
|
6399
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2488 def _subselect(self, proptree, parentname=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2489 """Create a subselect. This is factored out because some |
|
3633
a292054b4393
The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3612
diff
changeset
|
2490 databases (hmm only one, so far) doesn't support subselects |
|
a292054b4393
The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3612
diff
changeset
|
2491 look for "I can't believe it's not a toy RDBMS" in the mysql |
|
a292054b4393
The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3612
diff
changeset
|
2492 backend. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2493 """ |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2494 multilink_table = proptree.propclass.table_name |
| 6965 | 2495 nodeid_name = proptree.propclass.nodeid_name |
| 2496 linkid_name = proptree.propclass.linkid_name | |
|
6399
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2497 if parentname is None: |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2498 parentname = '_' + proptree.parent.classname |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2499 |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2500 w = '' |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2501 if proptree.need_retired: |
| 6965 | 2502 w = ' where %s.__retired__=0' % (multilink_table) |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2503 if proptree.need_child_retired: |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2504 tn1 = multilink_table |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2505 tn2 = '_' + proptree.classname |
| 6965 | 2506 w = ', %s where %s.%s=%s.id and %s.__retired__=0' % ( |
| 2507 tn2, tn1, linkid_name, tn2, tn2) | |
| 2508 return '%s.id not in (select %s from %s%s)' % ( | |
| 2509 parentname, nodeid_name, multilink_table, w) | |
|
3633
a292054b4393
The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3612
diff
changeset
|
2510 |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2511 def _filter_multilink_expression_fallback(self, proptree, expr): |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2512 '''This is a fallback for database that do not support |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2513 subselects.''' |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2514 classname = proptree.parent.uniqname |
|
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2515 multilink_table = proptree.propclass.table_name |
|
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2516 nid = proptree.propclass.nodeid_name |
|
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2517 lid = proptree.propclass.linkid_name |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2518 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2519 is_valid = expr.evaluate |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2520 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2521 last_id, kws = None, [] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2522 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2523 ids = IdListOptimizer() |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2524 append = ids.append |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2525 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2526 # This join and the evaluation in program space |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2527 # can be expensive for larger databases! |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2528 # TODO: Find a faster way to collect the data needed |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2529 # to evalute the expression. |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2530 # Moving the expression evaluation into the database |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2531 # would be nice but this tricky: Think about the cases |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2532 # where the multilink table does not have join values |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2533 # needed in evaluation. |
|
6402
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2534 w = j = '' |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2535 s = 'm.%s' % lid |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2536 if proptree.need_retired: |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2537 w = ' and m.__retired__=0' |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2538 elif proptree.need_child_retired: |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2539 tn2 = '_' + proptree.classname |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2540 j = ' LEFT OUTER JOIN %s ON %s.id = m.%s' % (tn2, tn2, lid) |
| 6965 | 2541 w = ' and %s.__retired__=0' % (tn2) |
|
6402
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2542 s = '%s.id' % tn2 |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2543 |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2544 stmnt = "SELECT c.id, %s FROM _%s as c " \ |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2545 "LEFT OUTER JOIN %s as m " \ |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2546 "ON c.id = m.%s%s%s ORDER BY c.id" % ( |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2547 s, classname, multilink_table, nid, j, w) |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2548 self.db.sql(stmnt) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2549 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2550 # collect all multilink items for a class item |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2551 for nodeid, kw in self.db.sql_fetchiter(): |
|
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2552 if nodeid != last_id: |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2553 if last_id is None: |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2554 last_id = nodeid |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2555 else: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2556 # we have all multilink items -> evaluate! |
| 6965 | 2557 if is_valid(kws): append(last_id) # noqa: E701 |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2558 last_id, kws = nodeid, [] |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2559 if kw is not None: |
|
6395
8baf81d1cfc1
Make multilink expressions work with mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6392
diff
changeset
|
2560 kws.append(int(kw)) |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2561 |
| 6965 | 2562 if last_id is not None and is_valid(kws): |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2563 append(last_id) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2564 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2565 # we have ids of the classname table |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2566 return ids.where("_%s.id" % classname, self.db.arg) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2567 |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2568 def _filter_link_expression(self, proptree, v): |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2569 """ Filter elements in the table that match the given expression |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2570 """ |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2571 pln = proptree.parent.uniqname |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2572 prp = proptree.name |
|
7936
a9b136565838
feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents:
7860
diff
changeset
|
2573 |
|
a9b136565838
feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents:
7860
diff
changeset
|
2574 if proptree.classname == 'user' and '@current_user' in v: |
|
a9b136565838
feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents:
7860
diff
changeset
|
2575 cu = self.db.getuid() |
|
a9b136565838
feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents:
7860
diff
changeset
|
2576 v = [x if x != "@current_user" else cu for x in v] |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2577 try: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2578 opcodes = [int(x) for x in v] |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2579 if min(opcodes) >= -1: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2580 raise ValueError() |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2581 expr = compile_expression(opcodes) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2582 # NULL doesn't compare to NULL in SQL |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2583 # So not (x = '1') will *not* include NULL values for x |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2584 # That's why we need that and clause: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2585 atom = "_%s._%s = %s and _%s._%s is not NULL" % ( |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2586 pln, prp, self.db.arg, pln, prp) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2587 atom_nil = "_%s._%s is NULL" % (pln, prp) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2588 lambda_atom = lambda n: atom if n.x >= 0 else atom_nil |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2589 values = [] |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2590 w = expr.generate(lambda_atom) |
| 6965 | 2591 |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2592 def collect_values(n): |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2593 if n.x >= 0: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2594 values.append(n.x) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2595 expr.visit(collect_values) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2596 return w, values |
|
8241
741ea8a86012
fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents:
7936
diff
changeset
|
2597 except ExpressionError as e: |
|
741ea8a86012
fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents:
7936
diff
changeset
|
2598 e.context['class'] = pln |
|
741ea8a86012
fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents:
7936
diff
changeset
|
2599 e.context['attr'] = prp |
|
741ea8a86012
fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents:
7936
diff
changeset
|
2600 raise |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2601 except: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2602 pass |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2603 # Fallback to original code |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2604 args = [] |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2605 where = None |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2606 d = {} |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2607 for entry in v: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2608 if entry == '-1': |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2609 entry = None |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2610 d[entry] = entry |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2611 l = [] |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2612 if None in d or not d: |
| 6965 | 2613 if None in d: del d[None] # noqa: E701 |
| 2614 l.append('_%s._%s is NULL' % (pln, prp)) | |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2615 if d: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2616 v = list(d) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2617 s = ','.join([self.db.arg for x in v]) |
| 6965 | 2618 l.append('(_%s._%s in (%s))' % (pln, prp, s)) |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2619 args = v |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2620 if l: |
| 6965 | 2621 where = '(' + ' or '.join(l) + ')' |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2622 return where, args |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2623 |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2624 def _filter_multilink_expression(self, proptree, v): |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2625 """ Filters out elements of the classname table that do not |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2626 match the given expression. |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2627 Returns tuple of 'WHERE' introns for the overall filter. |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2628 """ |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2629 classname = proptree.parent.uniqname |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2630 multilink_table = proptree.propclass.table_name |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2631 nid = proptree.propclass.nodeid_name |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2632 lid = proptree.propclass.linkid_name |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2633 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2634 try: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2635 opcodes = [int(x) for x in v] |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2636 if min(opcodes) >= -1: |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2637 raise ValueError() |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2638 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2639 expr = compile_expression(opcodes) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2640 |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2641 if not self.supports_subselects: |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2642 # We heavily rely on subselects. If there is |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2643 # no decent support fall back to slower variant. |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2644 return self._filter_multilink_expression_fallback( |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2645 proptree, expr) |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2646 |
|
6402
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2647 w = j = '' |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2648 if proptree.need_retired: |
| 6965 | 2649 w = ' and %s.__retired__=0' % (multilink_table) |
|
6402
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2650 elif proptree.need_child_retired: |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2651 tn1 = multilink_table |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2652 tn2 = '_' + proptree.classname |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2653 j = ', %s' % tn2 |
| 6965 | 2654 w = ' and %s.%s=%s.id and %s.__retired__=0' % ( |
| 2655 tn1, lid, tn2, tn2) | |
|
6402
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2656 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2657 atom = \ |
|
6402
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2658 "%s IN(SELECT %s FROM %s%s WHERE %s=a.id%s)" % ( |
| 6965 | 2659 self.db.arg, lid, multilink_table, j, nid, w) |
|
6399
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2660 atom_nil = self._subselect(proptree, 'a') |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2661 |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2662 lambda_atom = lambda n: atom if n.x >= 0 else atom_nil |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2663 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2664 intron = \ |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2665 "_%(classname)s.id in (SELECT id " \ |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2666 "FROM _%(classname)s AS a WHERE %(condition)s) " % { |
| 6965 | 2667 'classname': classname, |
| 2668 'condition': expr.generate(lambda_atom)} | |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2669 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2670 values = [] |
| 6965 | 2671 |
|
6399
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2672 def collect_values(n): |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2673 if n.x >= 0: |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2674 values.append(n.x) |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2675 expr.visit(collect_values) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2676 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2677 return intron, values |
|
8241
741ea8a86012
fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents:
7936
diff
changeset
|
2678 except ExpressionError as e: |
|
741ea8a86012
fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents:
7936
diff
changeset
|
2679 e.context['class'] = classname |
|
741ea8a86012
fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents:
7936
diff
changeset
|
2680 e.context['attr'] = proptree.name |
|
741ea8a86012
fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents:
7936
diff
changeset
|
2681 raise |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2682 except: |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2683 # fallback behavior when expression parsing above fails |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2684 orclause = '' |
| 6965 | 2685 if '-1' in v: |
| 2686 v = [x for x in v if int(x) > 0] | |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2687 orclause = self._subselect(proptree) |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2688 where = [] |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2689 where.append("%s.%s in (%s)" % (multilink_table, lid, |
| 6965 | 2690 ','.join([self.db.arg] * len(v)))) |
| 2691 where.append('_%s.id=%s.%s' % (classname, multilink_table, nid)) | |
| 2692 where = ' and '.join(where) | |
| 2693 if orclause: | |
| 2694 where = '((' + ' or '.join((where + ')', orclause)) + ')' | |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2695 |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2696 return where, v |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2697 |
| 6965 | 2698 def _filter_sql(self, search_matches, filterspec, srt=[], grp=[], retr=0, |
| 2699 retired=False, exact_match_spec={}, limit=None, | |
| 2700 offset=None): | |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2701 """ Compute the proptree and the SQL/ARGS for a filter. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2702 For argument description see filter below. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2703 We return a 3-tuple, the proptree, the sql and the sql-args |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2704 or None if no SQL is necessary. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2705 The flag retr serves to retrieve *all* non-Multilink properties |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2706 (for filling the cache during a filter_iter) |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2707 """ |
|
2578
7f25486ff85e
fixed RDBMS filter() for no matches from full-text search [SF#990778]
Richard Jones <richard@users.sourceforge.net>
parents:
2546
diff
changeset
|
2708 # we can't match anything if search_matches is empty |
| 4044 | 2709 if not search_matches and search_matches is not None: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2710 return None |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
2711 |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2712 icn = self.classname |
| 1165 | 2713 |
|
2379
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2714 # vars to hold the components of the SQL statement |
|
2586
3c67f4bfa225
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2578
diff
changeset
|
2715 frum = [] # FROM clauses |
|
2379
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2716 loj = [] # LEFT OUTER JOIN clauses |
|
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2717 where = [] # WHERE clauses |
|
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2718 args = [] # *any* positional arguments |
|
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2719 a = self.db.arg |
|
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2720 |
| 1165 | 2721 # figure the WHERE clause from the filterspec |
|
6413
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2722 use_distinct = False # Do we need a distinct clause? |
| 6965 | 2723 sortattr = self._sortattr(group=grp, sort=srt) |
|
5871
acc4a128ab9b
Backwards-compatible signature of _proptree method
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5869
diff
changeset
|
2724 proptree = self._proptree(filterspec, exact_match_spec, sortattr, retr) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2725 mlseen = 0 |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2726 for pt in reversed(proptree.sortattr): |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2727 p = pt |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2728 while p.parent: |
| 6965 | 2729 if isinstance(p.propclass, Multilink): |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2730 mlseen = True |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2731 if mlseen: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2732 p.sort_ids_needed = True |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2733 p.tree_sort_done = False |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2734 p = p.parent |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2735 if not mlseen: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2736 pt.attr_sort_done = pt.tree_sort_done = True |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2737 proptree.compute_sort_done() |
|
3693
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2738 |
| 6965 | 2739 cols = ['_%s.id' % icn] |
|
3635
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
2740 for p in proptree: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2741 rc = ac = oc = None |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2742 cn = p.classname |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2743 ln = p.uniqname |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2744 pln = p.parent.uniqname |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2745 pcn = p.parent.classname |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2746 k = p.name |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2747 v = p.val |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2748 propclass = p.propclass |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2749 if p.parent == proptree and p.name == 'id' \ |
| 6965 | 2750 and 'retrieve' in p.need_for: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2751 p.sql_idx = 0 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2752 if 'sort' in p.need_for or 'retrieve' in p.need_for: |
| 6965 | 2753 rc = oc = ac = '_%s._%s' % (pln, k) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2754 if isinstance(propclass, Multilink): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2755 if 'search' in p.need_for: |
|
6413
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2756 # if we joining with Multilink tables we need distinct |
|
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2757 use_distinct = True |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
2758 tn = propclass.table_name |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
2759 nid = propclass.nodeid_name |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
2760 lid = propclass.linkid_name |
|
6403
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2761 frum.append(tn) |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2762 if p.children or p.need_child_retired: |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2763 frum.append('_%s as _%s' % (cn, ln)) |
| 6965 | 2764 where.append('%s.%s=_%s.id' % (tn, lid, ln)) |
|
6403
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2765 if p.need_child_retired: |
| 6965 | 2766 where.append('_%s.__retired__=0' % (ln)) |
|
6403
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2767 # Note: need the where-clause if p has |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2768 # children that compute additional restrictions |
| 6965 | 2769 if (not p.has_values or |
| 2770 (not isinstance(v, type([])) and v != '-1') or | |
| 2771 p.children): | |
| 2772 where.append('_%s.id=%s.%s' % (pln, tn, nid)) | |
|
4061
52431f2ad76b
Correct handling of a Multilink filterspec with an empty list of linked-to items.
Stefan Seefeld <stefan@seefeld.name>
parents:
4060
diff
changeset
|
2773 if v in ('-1', ['-1'], []): |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2774 # only match rows that have count(linkid)=0 in the |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2775 # corresponding multilink table) |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2776 where.append(self._subselect(p)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2777 else: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2778 if p.has_values: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2779 if isinstance(v, type([])): |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2780 # The where-clause above is conditionally |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2781 # created in _filter_multilink_expression |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2782 w, arg = self._filter_multilink_expression(p, v) |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2783 where.append(w) |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2784 args += arg |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2785 else: |
| 6965 | 2786 where.append('%s.%s=%s' % (tn, lid, a)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2787 args.append(v) |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2788 # Don't match retired nodes if rev_multilink |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2789 if p.need_retired: |
| 6965 | 2790 where.append('%s.__retired__=0' % (tn)) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2791 if 'sort' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2792 assert not p.attr_sort_done and not p.sort_ids_needed |
|
1365
4884fb0860f9
fixed rdbms searching by ID [SF#666615]
Richard Jones <richard@users.sourceforge.net>
parents:
1351
diff
changeset
|
2793 elif k == 'id': |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2794 if 'search' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2795 if isinstance(v, type([])): |
|
4462
0f4a8fcb9a1d
fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents:
4448
diff
changeset
|
2796 # If there are no permitted values, then the |
|
0f4a8fcb9a1d
fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents:
4448
diff
changeset
|
2797 # where clause will always be false, and we |
|
0f4a8fcb9a1d
fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents:
4448
diff
changeset
|
2798 # can optimize the query away. |
|
0f4a8fcb9a1d
fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents:
4448
diff
changeset
|
2799 if not v: |
|
4803
5e679e364f9a
Bug-fix: need to return None for empty query
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4787
diff
changeset
|
2800 return None |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2801 s = ','.join([a for x in v]) |
| 6965 | 2802 where.append('_%s.%s in (%s)' % (pln, k, s)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2803 args = args + v |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2804 else: |
| 6965 | 2805 where.append('_%s.%s=%s' % (pln, k, a)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2806 args.append(v) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2807 if 'sort' in p.need_for or 'retrieve' in p.need_for: |
| 6965 | 2808 rc = oc = ac = '_%s.id' % pln |
|
3683
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2809 elif isinstance(propclass, String): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2810 if 'search' in p.need_for: |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2811 exact = [] |
|
3683
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2812 if not isinstance(v, type([])): |
|
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2813 v = [v] |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2814 new_v = [] |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2815 for x in v: |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2816 if isinstance(x, hyperdb.Exact_Match): |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2817 exact.append(True) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2818 new_v.append(x.value) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2819 else: |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2820 exact.append(False) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2821 # Quote special search characters '%' and '_' for |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2822 # correct matching with LIKE/ILIKE |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2823 # Note that we now pass the elements of v as query |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2824 # arguments and don't interpolate the quoted string |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2825 # into the sql statement. Should be safer. |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2826 new_v.append(self.db.search_stringquote(x)) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2827 v = new_v |
|
3683
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2828 |
|
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2829 # now add to the where clause |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2830 w = [] |
|
7860
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
2831 if not self.case_sensitive_equal: |
|
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
2832 self.case_sensitive_equal = \ |
|
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
2833 self.get_case_sensitive_equal() |
|
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
2834 cse = self.case_sensitive_equal |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2835 for vv, ex in zip(v, exact): |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2836 if ex: |
| 6965 | 2837 w.append("_%s._%s %s %s" % ( |
|
7860
8b31893f5930
issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents:
7750
diff
changeset
|
2838 pln, k, cse, a)) |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2839 args.append(vv) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2840 else: |
| 6965 | 2841 w.append("_%s._%s %s %s ESCAPE %s" % ( |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2842 pln, k, self.case_insensitive_like, a, a)) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2843 args.extend((vv, '\\')) |
| 6965 | 2844 where.append('(' + ' and '.join(w) + ')') |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2845 if 'sort' in p.need_for: |
| 6965 | 2846 oc = ac = 'lower(_%s._%s)' % (pln, k) |
|
1170
af104fa52746
Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1168
diff
changeset
|
2847 elif isinstance(propclass, Link): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2848 if 'search' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2849 if p.children: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2850 if 'sort' not in p.need_for: |
|
3694
595041d78104
Fix a peculiarity in sql generation.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3693
diff
changeset
|
2851 frum.append('_%s as _%s' % (cn, ln)) |
|
6413
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2852 c = [x for x in p.children if 'search' in x.need_for] |
|
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2853 if c: |
| 6965 | 2854 where.append('_%s._%s=_%s.id' % (pln, k, ln)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2855 if p.has_values: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2856 if isinstance(v, type([])): |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2857 w, arg = self._filter_link_expression(p, v) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2858 if w: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2859 where.append(w) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2860 args += arg |
|
3636
fa7becc62534
Of course for Links there is the same bug as I fixed before for multilinks.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3635
diff
changeset
|
2861 else: |
|
7936
a9b136565838
feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents:
7860
diff
changeset
|
2862 if v == '@current_user' and \ |
|
a9b136565838
feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents:
7860
diff
changeset
|
2863 propclass.classname == 'user': |
|
a9b136565838
feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents:
7860
diff
changeset
|
2864 v = self.db.getuid() |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2865 if v in ('-1', None): |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2866 v = None |
| 6965 | 2867 where.append('_%s._%s is NULL' % (pln, k)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2868 else: |
| 6965 | 2869 where.append('_%s._%s=%s' % (pln, k, a)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2870 args.append(v) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2871 if 'sort' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2872 lp = p.cls.labelprop() |
| 6965 | 2873 oc = ac = '_%s._%s' % (pln, k) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2874 if lp != 'id': |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2875 if p.tree_sort_done: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2876 loj.append( |
| 6965 | 2877 'LEFT OUTER JOIN _%s as _%s on _%s._%s=_%s.id' % ( |
| 2878 cn, ln, pln, k, ln)) | |
| 2879 oc = '_%s._%s' % (ln, lp) | |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2880 if 'retrieve' in p.need_for: |
| 6965 | 2881 rc = '_%s._%s' % (pln, k) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2882 elif isinstance(propclass, Date) and 'search' in p.need_for: |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
2883 dc = self.db.to_sql_value(hyperdb.Date) |
|
1351
d1bfb479e527
fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents:
1345
diff
changeset
|
2884 if isinstance(v, type([])): |
|
d1bfb479e527
fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents:
1345
diff
changeset
|
2885 s = ','.join([a for x in v]) |
| 6965 | 2886 where.append('_%s._%s in (%s)' % (pln, k, s)) |
|
3633
a292054b4393
The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3612
diff
changeset
|
2887 args = args + [dc(date.Date(x)) for x in v] |
|
1351
d1bfb479e527
fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents:
1345
diff
changeset
|
2888 else: |
|
1499
8ee69708da0c
added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1496
diff
changeset
|
2889 try: |
| 6118 | 2890 wh = [] |
| 2891 ar = [] | |
| 2892 for d in v.split(','): | |
| 2893 w1 = [] | |
| 2894 if d == '-': | |
| 6965 | 2895 wh.append('_%s._%s is NULL' % (pln, k)) |
| 6118 | 2896 continue |
| 2897 # Try to filter on range of dates | |
| 2898 date_rng = propclass.range_from_raw(d, self.db) | |
| 2899 if date_rng.from_value: | |
| 6965 | 2900 w1.append('_%s._%s >= %s' % (pln, k, a)) |
| 6118 | 2901 ar.append(dc(date_rng.from_value)) |
| 2902 if date_rng.to_value: | |
| 6965 | 2903 w1.append('_%s._%s <= %s' % (pln, k, a)) |
| 6118 | 2904 ar.append(dc(date_rng.to_value)) |
| 6965 | 2905 wh.append(' and '.join(w1)) |
| 2906 where.append('(' + ' or '.join(wh) + ')') | |
| 2907 args.extend(ar) | |
|
1499
8ee69708da0c
added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1496
diff
changeset
|
2908 except ValueError: |
|
8ee69708da0c
added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1496
diff
changeset
|
2909 # If range creation fails - ignore that search parameter |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2910 pass |
|
1351
d1bfb479e527
fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents:
1345
diff
changeset
|
2911 elif isinstance(propclass, Interval): |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2912 # filter/sort using the __<prop>_int__ column |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2913 if 'search' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2914 if isinstance(v, type([])): |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2915 s = ','.join([a for x in v]) |
| 6965 | 2916 where.append('_%s.__%s_int__ in (%s)' % (pln, k, s)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2917 args = args + [date.Interval(x).as_seconds() for x in v] |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2918 else: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2919 try: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2920 # Try to filter on range of intervals |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2921 date_rng = Range(v, date.Interval) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2922 if date_rng.from_value: |
| 6965 | 2923 where.append('_%s.__%s_int__ >= %s' % ( |
| 2924 pln, k, a)) | |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2925 args.append(date_rng.from_value.as_seconds()) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2926 if date_rng.to_value: |
| 6965 | 2927 where.append('_%s.__%s_int__ <= %s' % ( |
| 2928 pln, k, a)) | |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2929 args.append(date_rng.to_value.as_seconds()) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2930 except ValueError: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2931 # If range creation fails - ignore search parameter |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2932 pass |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2933 if 'sort' in p.need_for: |
| 6965 | 2934 oc = ac = '_%s.__%s_int__' % (pln, k) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2935 if 'retrieve' in p.need_for: |
| 6965 | 2936 rc = '_%s._%s' % (pln, k) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2937 elif isinstance(propclass, Boolean) and 'search' in p.need_for: |
| 6965 | 2938 if isinstance(v, str): |
|
4365
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2939 v = v.split(',') |
| 6965 | 2940 if not isinstance(v, list): |
|
4365
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2941 v = [v] |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2942 bv = [] |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2943 for val in v: |
| 6965 | 2944 if isinstance(val, str): |
| 2945 bv.append(propclass.from_raw(val)) | |
|
4365
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2946 else: |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2947 bv.append(bool(val)) |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2948 if len(bv) == 1: |
| 6965 | 2949 where.append('_%s._%s=%s' % (pln, k, a)) |
|
4365
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2950 args = args + bv |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2951 else: |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2952 s = ','.join([a for x in v]) |
| 6965 | 2953 where.append('_%s._%s in (%s)' % (pln, k, s)) |
|
4365
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2954 args = args + bv |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2955 elif 'search' in p.need_for: |
| 1165 | 2956 if isinstance(v, type([])): |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
2957 s = ','.join([a for x in v]) |
| 6965 | 2958 where.append('_%s._%s in (%s)' % (pln, k, s)) |
| 1165 | 2959 args = args + v |
| 2960 else: | |
| 6965 | 2961 where.append('_%s._%s=%s' % (pln, k, a)) |
| 1165 | 2962 args.append(v) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2963 if oc: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2964 if p.sort_ids_needed: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2965 if rc == ac: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2966 p.sql_idx = len(cols) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2967 p.auxcol = len(cols) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2968 cols.append(ac) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2969 if p.tree_sort_done and p.sort_direction: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2970 # Don't select top-level id or multilink twice |
| 6965 | 2971 if (not p.sort_ids_needed or ac != oc) and ( |
| 2972 p.name != 'id' or p.parent != proptree): | |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2973 if rc == oc: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2974 p.sql_idx = len(cols) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2975 cols.append(oc) |
|
3685
4d9adb8bc3b1
Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3683
diff
changeset
|
2976 desc = ['', ' desc'][p.sort_direction == '-'] |
|
4d9adb8bc3b1
Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3683
diff
changeset
|
2977 # Some SQL dbs sort NULL values last -- we want them first. |
|
4d9adb8bc3b1
Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3683
diff
changeset
|
2978 if (self.order_by_null_values and p.name != 'id'): |
|
4d9adb8bc3b1
Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3683
diff
changeset
|
2979 nv = self.order_by_null_values % oc |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2980 cols.append(nv) |
|
3693
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2981 p.orderby.append(nv + desc) |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2982 p.orderby.append(oc + desc) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2983 if 'retrieve' in p.need_for and p.sql_idx is None: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2984 assert(rc) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2985 p.sql_idx = len(cols) |
| 6965 | 2986 cols.append(rc) |
| 1165 | 2987 |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2988 props = self.getprops() |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2989 |
|
1740
5ca448ff8052
don't have RDBMS backends list retired nodes [SF#767319]
Richard Jones <richard@users.sourceforge.net>
parents:
1719
diff
changeset
|
2990 # don't match retired nodes |
|
5318
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2991 if retired is not None: |
|
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2992 op = '=' |
|
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2993 if retired: |
|
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2994 op = '!=' |
| 6965 | 2995 where.append('_%s.__retired__%s0' % (icn, op)) |
|
1740
5ca448ff8052
don't have RDBMS backends list retired nodes [SF#767319]
Richard Jones <richard@users.sourceforge.net>
parents:
1719
diff
changeset
|
2996 |
| 1165 | 2997 # add results of full text search |
| 2998 if search_matches is not None: | |
| 4044 | 2999 s = ','.join([a for x in search_matches]) |
| 6965 | 3000 where.append('_%s.id in (%s)' % (icn, s)) |
| 4113 | 3001 args = args + [x for x in search_matches] |
| 1165 | 3002 |
| 3003 # construct the SQL | |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
3004 frum.append('_'+icn) |
| 1165 | 3005 frum = ','.join(frum) |
|
1170
af104fa52746
Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1168
diff
changeset
|
3006 if where: |
|
af104fa52746
Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1168
diff
changeset
|
3007 where = ' where ' + (' and '.join(where)) |
|
af104fa52746
Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1168
diff
changeset
|
3008 else: |
|
af104fa52746
Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1168
diff
changeset
|
3009 where = '' |
|
6413
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
3010 if use_distinct: |
|
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
3011 # Avoid dupes |
| 6965 | 3012 cols[0] = 'distinct(_%s.id)' % icn |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3013 |
|
3693
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
3014 order = [] |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
3015 # keep correct sequence of order attributes. |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
3016 for sa in proptree.sortattr: |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
3017 if not sa.attr_sort_done: |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
3018 continue |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
3019 order.extend(sa.orderby) |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
3020 if order: |
| 6965 | 3021 order = ' order by %s' % (','.join(order)) |
| 1165 | 3022 else: |
| 3023 order = '' | |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
3024 |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3025 if limit is not None: |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3026 limit = ' LIMIT %s' % limit |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3027 else: |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3028 limit = '' |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3029 if offset is not None: |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3030 offset = ' OFFSET %s' % offset |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3031 else: |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3032 offset = '' |
| 1165 | 3033 cols = ','.join(cols) |
|
2379
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
3034 loj = ' '.join(loj) |
| 6965 | 3035 sql = 'select %s from %s %s %s%s%s%s' % ( |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3036 cols, frum, loj, where, order, limit, offset) |
| 1165 | 3037 args = tuple(args) |
|
2319
7cf7e3bd1b31
more column uniqueness fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2318
diff
changeset
|
3038 __traceback_info__ = (sql, args) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3039 return proptree, sql, args |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3040 |
|
5318
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
3041 def filter(self, search_matches, filterspec, sort=[], group=[], |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3042 retired=False, exact_match_spec={}, limit=None, offset=None): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3043 """Return a list of the ids of the active nodes in this class that |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3044 match the 'filter' spec, sorted by the group spec and then the |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3045 sort spec |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3046 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3047 "filterspec" is {propname: value(s)} |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3048 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3049 "sort" and "group" are [(dir, prop), ...] where dir is '+', '-' |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3050 or None and prop is a prop name or None. Note that for |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3051 backward-compatibility reasons a single (dir, prop) tuple is |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3052 also allowed. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3053 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3054 "search_matches" is a container type or None |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3055 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3056 The filter must match all properties specificed. If the property |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3057 value to match is a list: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3058 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3059 1. String properties must match all elements in the list, and |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3060 2. Other properties must match any of the elements in the list. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3061 """ |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3062 if __debug__: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3063 start_t = time.time() |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3064 |
| 6965 | 3065 sq = self._filter_sql(search_matches, filterspec, sort, group, |
| 3066 retired=retired, | |
| 3067 exact_match_spec=exact_match_spec, | |
| 3068 limit=limit, offset=offset) | |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3069 # nothing to match? |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3070 if sq is None: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3071 return [] |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3072 proptree, sql, args = sq |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3073 |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3074 cursor = self.db.sql_new_cursor(name='filter') |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3075 self.db.sql(sql, args, cursor) |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3076 # Reduce this to only the first row (the ID), this can save a |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3077 # lot of space for large query results (not using fetchall) |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3078 # We cannot do this if sorting by multilink |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3079 if proptree.tree_sort_done: |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3080 l = [str(row[0]) for row in cursor] |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3081 else: |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3082 l = cursor.fetchall() |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3083 cursor.close() |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3084 |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3085 # Multilink sorting |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
3086 # Compute values needed for sorting in proptree.sort |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3087 if not proptree.tree_sort_done: |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3088 for p in proptree: |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3089 if hasattr(p, 'auxcol'): |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3090 p.sort_ids = [row[p.auxcol] for row in l] |
| 6965 | 3091 p.sort_result = p._sort_repr( |
| 3092 p.propclass.sort_repr, p.sort_ids) | |
| 3093 l = proptree.sort([str(row[0]) for row in l]) | |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
3094 |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
3095 if __debug__: |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
3096 self.db.stats['filtering'] += (time.time() - start_t) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
3097 return l |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
3098 |
|
5318
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
3099 def filter_iter(self, search_matches, filterspec, sort=[], group=[], |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3100 retired=False, exact_match_spec={}, limit=None, |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3101 offset=None): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3102 """Iterator similar to filter above with same args. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3103 Limitation: We don't sort on multilinks. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3104 This uses an optimisation: We put all nodes that are in the |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3105 current row into the node cache. Then we return the node id. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3106 That way a fetch of a node won't create another sql-fetch (with |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3107 a join) from the database because the nodes are already in the |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3108 cache. We're using our own temporary cursor. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3109 """ |
|
5318
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
3110 sq = self._filter_sql(search_matches, filterspec, sort, group, retr=1, |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
3111 retired=retired, |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3112 exact_match_spec=exact_match_spec, |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3113 limit=limit, offset=offset) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3114 # nothing to match? |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3115 if sq is None: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3116 return |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3117 proptree, sql, args = sq |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3118 cursor = self.db.sql_new_cursor(name='filter_iter') |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3119 self.db.sql(sql, args, cursor) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3120 classes = {} |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3121 for p in proptree: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3122 if 'retrieve' in p.need_for: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3123 cn = p.parent.classname |
| 6965 | 3124 ptid = p.parent.id # not the nodeid! |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3125 key = (cn, ptid) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3126 if key not in classes: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3127 classes[key] = {} |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3128 name = p.name |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3129 assert (name) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3130 classes[key][name] = p |
|
4473
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
3131 p.to_hyperdb = self.db.to_hyperdb_value(p.propclass.__class__) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3132 while True: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3133 row = cursor.fetchone() |
| 6965 | 3134 if not row: break # noqa: E701 |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3135 # populate cache with current items |
| 6965 | 3136 for (classname, _ptid), pt in classes.items(): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3137 nodeid = str(row[pt['id'].sql_idx]) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3138 key = (classname, nodeid) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3139 if key in self.db.cache: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3140 self.db._cache_refresh(key) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3141 continue |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3142 node = {} |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
3143 for propname, p in pt.items(): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3144 value = row[p.sql_idx] |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3145 if value is not None: |
|
4473
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
3146 value = p.to_hyperdb(value) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3147 node[propname] = value |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3148 self.db._cache_save(key, node) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3149 yield str(row[0]) |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3150 cursor.close() |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3151 |
|
3688
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3152 def filter_sql(self, sql): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3153 """Return a list of the ids of the items in this class that match |
|
3688
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3154 the SQL provided. The SQL is a complete "select" statement. |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3155 |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3156 The SQL select must include the item id as the first column. |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3157 |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3158 This function DOES NOT filter out retired items, add on a where |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
3159 clause "__retired__=0" if you don't want retired nodes. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3160 """ |
|
3688
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3161 if __debug__: |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3162 start_t = time.time() |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3163 |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3164 self.db.sql(sql) |
| 6965 | 3165 results = self.db.sql_fetchall() |
|
3688
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3166 |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3167 if __debug__: |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3168 self.db.stats['filtering'] += (time.time() - start_t) |
| 6965 | 3169 return results |
|
3688
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3170 |
| 1165 | 3171 def count(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3172 """Get the number of nodes in this class. |
| 1165 | 3173 |
| 3174 If the returned integer is 'numnodes', the ids of all the nodes | |
| 3175 in this class run from 1 to numnodes, and numnodes+1 will be the | |
| 3176 id of the next node to be created in this class. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3177 """ |
| 1165 | 3178 return self.db.countnodes(self.classname) |
| 3179 | |
| 3180 # Manipulating properties: | |
| 3181 def getprops(self, protected=1): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3182 """Return a dictionary mapping property names to property objects. |
| 1165 | 3183 If the "protected" flag is true, we include protected properties - |
| 3184 those which may not be modified. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3185 """ |
| 1165 | 3186 d = self.properties.copy() |
| 3187 if protected: | |
| 3188 d['id'] = String() | |
| 3189 d['creation'] = hyperdb.Date() | |
| 3190 d['activity'] = hyperdb.Date() | |
|
1176
bd3b57859c37
On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents:
1175
diff
changeset
|
3191 d['creator'] = hyperdb.Link('user') |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
3192 d['actor'] = hyperdb.Link('user') |
| 1165 | 3193 return d |
| 3194 | |
| 3195 def addprop(self, **properties): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3196 """Add properties to this class. |
| 1165 | 3197 |
| 3198 The keyword arguments in 'properties' must map names to property | |
| 3199 objects, or a TypeError is raised. None of the keys in 'properties' | |
| 3200 may collide with the names of existing properties, or a ValueError | |
| 3201 is raised before any properties have been added. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3202 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3203 for key in properties: |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3204 if key in self.properties: |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3205 raise ValueError(key) |
| 1165 | 3206 self.properties.update(properties) |
| 3207 | |
| 3208 def index(self, nodeid): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3209 """Add (or refresh) the node to search indexes |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3210 """ |
| 1165 | 3211 # find all the String properties that have indexme |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
3212 for prop, propclass in self.getprops().items(): |
| 1165 | 3213 if isinstance(propclass, String) and propclass.indexme: |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3214 self.db.indexer.add_text((self.classname, nodeid, prop), |
| 6965 | 3215 str(self.get(nodeid, prop))) |
| 1165 | 3216 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3217 # |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3218 # import / export support |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3219 # |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3220 def export_list(self, propnames, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3221 """ Export a node - generate a list of CSV-able data in the order |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3222 specified by propnames for the given node. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3223 """ |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3224 properties = self.getprops() |
| 6965 | 3225 return_list = [] |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3226 for prop in propnames: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3227 proptype = properties[prop] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3228 value = self.get(nodeid, prop) |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3229 # "marshal" data where needed |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3230 if value is None: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3231 pass |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3232 elif isinstance(proptype, hyperdb.Date): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3233 value = value.get_tuple() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3234 elif isinstance(proptype, hyperdb.Interval): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3235 value = value.get_tuple() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3236 elif isinstance(proptype, hyperdb.Password): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3237 value = str(value) |
| 6965 | 3238 return_list.append(repr_export(value)) |
| 3239 return_list.append(repr_export(self.is_retired(nodeid))) | |
| 3240 return return_list | |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3241 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3242 def import_list(self, propnames, proplist): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3243 """ Import a node - all information including "id" is present and |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3244 should not be sanity checked. Triggers are not triggered. The |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3245 journal should be initialised using the "creator" and "created" |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3246 information. |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3247 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3248 Return the nodeid of the node imported. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3249 """ |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3250 |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3251 logger = logging.getLogger('roundup.hyperdb.backend') |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3252 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3253 if self.db.journaltag is None: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3254 raise DatabaseError(_('Database open read-only')) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3255 properties = self.getprops() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3256 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3257 # make the new node's property map |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3258 d = {} |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3259 retire = 0 |
| 6965 | 3260 if "id" not in propnames: |
|
2505
bdd112cf61ba
rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
3261 newid = self.db.newid(self.classname) |
|
bdd112cf61ba
rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
3262 else: |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3263 newid = eval_import(proplist[propnames.index("id")]) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3264 for i in range(len(propnames)): |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3265 # Use eval_import to reverse the repr_export() used to |
|
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3266 # output the CSV |
|
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3267 value = eval_import(proplist[i]) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3268 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3269 # Figure the property for this column |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3270 propname = propnames[i] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3271 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3272 # "unmarshal" where necessary |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3273 if propname == 'id': |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3274 continue |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3275 elif propname == 'is retired': |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3276 # is the item retired? |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3277 if int(value): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3278 retire = 1 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3279 continue |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3280 elif value is None: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3281 d[propname] = None |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3282 continue |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3283 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3284 prop = properties[propname] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3285 if value is None: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3286 # don't set Nones |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3287 continue |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3288 elif isinstance(prop, hyperdb.Date): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3289 value = date.Date(value) |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3290 elif isinstance(prop, hyperdb.Interval): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3291 value = date.Interval(value) |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3292 elif isinstance(prop, hyperdb.Password): |
|
7211
506c86823abb
Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents:
7038
diff
changeset
|
3293 value = password.Password(encrypted=value, |
|
506c86823abb
Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents:
7038
diff
changeset
|
3294 config=self.db.config) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
3295 elif isinstance(prop, String): |
|
5416
56c9bcdea47f
Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5395
diff
changeset
|
3296 value = us2s(value) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
3297 if not isinstance(value, str): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3298 raise TypeError('new property "%(propname)s" not a ' |
| 6965 | 3299 'string: %(value)r' % locals()) |
|
2892
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
3300 if prop.indexme: |
|
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
3301 self.db.indexer.add_text((self.classname, newid, propname), |
| 6965 | 3302 value) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
3303 d[propname] = value |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3304 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3305 # get a new id if necessary |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
3306 if newid is None: |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3307 newid = self.db.newid(self.classname) |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
3308 |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3309 activeid = None |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3310 has_node = False |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3311 |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3312 # use the arg for __retired__ to cope with any odd database type |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3313 # conversion (hello, sqlite) |
| 6965 | 3314 retired_sql = 'update _%s set __retired__=%s where id=%s' % ( |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3315 self.classname, self.db.arg, self.db.arg) |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3316 |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
3317 # insert new node or update existing? |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3318 # if integrity error raised try to recover |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3319 try: |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3320 has_node = self.hasnode(newid) |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3321 if not has_node: |
| 6965 | 3322 self.db.addnode(self.classname, newid, d) # insert |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3323 else: |
| 6965 | 3324 self.db.setnode(self.classname, newid, d) # update |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
3325 self.db.checkpoint_data() |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3326 # Blech, different db's return different exceptions |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3327 # so I can't list them here as some might not be defined |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3328 # on a given system. So capture all exceptions from the |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3329 # code above and try to correct it. If it's correctable its |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3330 # some form of Uniqueness Failure/Integrity Error otherwise |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3331 # undo the fixup and pass on the error. |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
3332 except Exception as e: # nosec |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3333 logger.info('Attempting to handle import exception ' |
| 6965 | 3334 'for id %s: %s' % (newid, e)) |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3335 |
|
7665
6f841a55eabf
fix: database error handling during import of a non-user item
John Rouillard <rouilj@ieee.org>
parents:
7211
diff
changeset
|
3336 classdb = self.db.getclass(self.classname) |
|
6f841a55eabf
fix: database error handling during import of a non-user item
John Rouillard <rouilj@ieee.org>
parents:
7211
diff
changeset
|
3337 keyname = classdb.getkey() |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3338 if has_node or not keyname: # Not an integrity error |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3339 raise |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
3340 self.db.restore_connection_on_error() |
|
7665
6f841a55eabf
fix: database error handling during import of a non-user item
John Rouillard <rouilj@ieee.org>
parents:
7211
diff
changeset
|
3341 activeid = classdb.lookup(d[keyname]) |
| 6965 | 3342 self.db.sql(retired_sql, (-1, activeid)) # clear the active node |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3343 # this can only happen on an addnode, so retry |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3344 try: |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3345 # if this raises an error, let it propagate upward |
| 6965 | 3346 self.db.addnode(self.classname, newid, d) # insert |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3347 except Exception: |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3348 # undo the database change |
| 6965 | 3349 self.db.sql(retired_sql, (0, activeid)) # clear active node |
| 3350 raise # propagate | |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3351 logger.info('Successfully handled import exception ' |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3352 'for id %s which conflicted with %s' % ( |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3353 newid, activeid)) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3354 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3355 # retire? |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3356 if retire: |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3357 self.db.sql(retired_sql, (newid, newid)) |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3358 |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3359 if activeid: |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3360 # unretire the active node |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3361 self.db.sql(retired_sql, ('0', activeid)) |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3362 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3363 return newid |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3364 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3365 def export_journals(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3366 """Export a class's journal - generate a list of lists of |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3367 CSV-able data: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3368 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3369 nodeid, date, user, action, params |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3370 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3371 No heading here - the columns are fixed. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3372 """ |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3373 properties = self.getprops() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3374 r = [] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3375 for nodeid in self.getnodeids(): |
| 6965 | 3376 for nodeid, date_, user, action, params in self.history( |
| 3377 nodeid, enforceperm=False, skipquiet=False): | |
| 3378 date_ = date_.get_tuple() | |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3379 if action == 'set': |
|
3365
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3380 export_data = {} |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
3381 for propname, value in params.items(): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3382 if propname not in properties: |
|
3365
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3383 # property no longer in the schema |
|
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3384 continue |
|
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3385 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3386 prop = properties[propname] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3387 # make sure the params are eval()'able |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3388 if value is None: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3389 pass |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3390 elif isinstance(prop, Date): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3391 value = value.get_tuple() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3392 elif isinstance(prop, Interval): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3393 value = value.get_tuple() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3394 elif isinstance(prop, Password): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3395 value = str(value) |
|
3365
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3396 export_data[propname] = value |
|
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3397 params = export_data |
|
3933
9095a4da67f9
Handle export and import of old trackers that have data attached to...
Richard Jones <richard@users.sourceforge.net>
parents:
3925
diff
changeset
|
3398 elif action == 'create' and params: |
|
9095a4da67f9
Handle export and import of old trackers that have data attached to...
Richard Jones <richard@users.sourceforge.net>
parents:
3925
diff
changeset
|
3399 # old tracker with data stored in the create! |
|
9095a4da67f9
Handle export and import of old trackers that have data attached to...
Richard Jones <richard@users.sourceforge.net>
parents:
3925
diff
changeset
|
3400 params = {} |
| 6965 | 3401 l = [nodeid, date_, user, action, params] |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3402 r.append(list(map(repr_export, l))) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3403 return r |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3404 |
| 6965 | 3405 |
|
2597
c86b2179085b
fix journal export of files to remove content from CSV files
Richard Jones <richard@users.sourceforge.net>
parents:
2586
diff
changeset
|
3406 class FileClass(hyperdb.FileClass, Class): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3407 """This class defines a large chunk of data. To support this, it has a |
| 1165 | 3408 mandatory String property "content" which is typically saved off |
| 3409 externally to the hyperdb. | |
| 3410 | |
| 3411 The default MIME type of this data is defined by the | |
| 3412 "default_mime_type" class attribute, which may be overridden by each | |
| 3413 node if the class defines a "type" String property. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3414 """ |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3415 def __init__(self, db, classname, **properties): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3416 """The newly-created class automatically includes the "content" |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3417 and "type" properties. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3418 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3419 if 'content' not in properties: |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3420 properties['content'] = hyperdb.String(indexme='yes') |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3421 if 'type' not in properties: |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3422 properties['type'] = hyperdb.String() |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3423 Class.__init__(self, db, classname, **properties) |
| 1165 | 3424 |
| 3425 def create(self, **propvalues): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3426 """ snaffle the file propvalue and store in a file |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3427 """ |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3428 # we need to fire the auditors now, or the content property won't |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3429 # be in propvalues for the auditors to play with |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3430 self.fireAuditors('create', None, propvalues) |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3431 |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3432 # now remove the content property so it's not stored in the db |
| 1165 | 3433 content = propvalues['content'] |
| 3434 del propvalues['content'] | |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3435 |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3436 # do the database create |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3437 newid = self.create_inner(**propvalues) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3438 |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3439 # figure the mime type |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3440 mime_type = propvalues.get('type', self.default_mime_type) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3441 |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3442 # and index! |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3443 if self.properties['content'].indexme: |
|
5544
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3444 index_content = content |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3445 if bytes != str and isinstance(content, bytes): |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3446 index_content = content.decode('utf-8', errors='ignore') |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3447 self.db.indexer.add_text((self.classname, newid, 'content'), |
| 6965 | 3448 index_content, mime_type) |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3449 |
|
3979
954971d612ee
Fire reactors after file storage is all done (patch [SF#2001243])
Richard Jones <richard@users.sourceforge.net>
parents:
3977
diff
changeset
|
3450 # store off the content as a file |
|
5492
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3451 self.db.storefile(self.classname, newid, None, bs2b(content)) |
|
3979
954971d612ee
Fire reactors after file storage is all done (patch [SF#2001243])
Richard Jones <richard@users.sourceforge.net>
parents:
3977
diff
changeset
|
3452 |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3453 # fire reactors |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3454 self.fireReactors('create', newid, None) |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3455 |
| 1165 | 3456 return newid |
| 3457 | |
| 3458 def get(self, nodeid, propname, default=_marker, cache=1): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3459 """ Trap the content propname and get it from the file |
|
1780
d2801a2b0a77
Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents:
1751
diff
changeset
|
3460 |
|
d2801a2b0a77
Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents:
1751
diff
changeset
|
3461 'cache' exists for backwards compatibility, and is not used. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3462 """ |
| 1165 | 3463 poss_msg = 'Possibly a access right configuration problem.' |
| 3464 if propname == 'content': | |
| 3465 try: | |
|
5492
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3466 return b2s(self.db.getfile(self.classname, nodeid, None)) |
|
5248
198b6e810c67
Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents:
5236
diff
changeset
|
3467 except IOError as strerror: |
| 1165 | 3468 # BUG: by catching this we donot see an error in the log. |
| 6965 | 3469 return 'ERROR reading file: %s%s\n%s\n%s' % ( |
| 1165 | 3470 self.classname, nodeid, poss_msg, strerror) |
| 6965 | 3471 except UnicodeDecodeError: |
|
5725
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3472 # if content is not text (e.g. jpeg file) we get |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3473 # unicode error trying to convert to string in python 3. |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3474 # trap it and supply an error message. Include md5sum |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3475 # of content as this string is included in the etag |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3476 # calculation of the object. |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3477 return ('%s%s is not text, retrieve using ' |
| 6965 | 3478 'binary_content property. mdsum: %s') % ( |
| 3479 self.classname, nodeid, | |
| 3480 md5(self.db.getfile( | |
| 3481 self.classname, | |
| 3482 nodeid, | |
| 3483 None)).hexdigest()) # nosec - bandit md5 use ok | |
|
5492
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3484 elif propname == 'binary_content': |
|
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3485 return self.db.getfile(self.classname, nodeid, None) |
|
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3486 |
|
2692
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
3487 if default is not _marker: |
|
1780
d2801a2b0a77
Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents:
1751
diff
changeset
|
3488 return Class.get(self, nodeid, propname, default) |
| 1165 | 3489 else: |
|
1780
d2801a2b0a77
Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents:
1751
diff
changeset
|
3490 return Class.get(self, nodeid, propname) |
| 1165 | 3491 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3492 def set(self, itemid, **propvalues): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3493 """ Snarf the "content" propvalue and update it in a file |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3494 """ |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3495 self.fireAuditors('set', itemid, propvalues) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3496 oldvalues = copy.deepcopy(self.db.getnode(self.classname, itemid)) |
| 1165 | 3497 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3498 # now remove the content property so it's not stored in the db |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3499 content = None |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3500 if 'content' in propvalues: |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3501 content = propvalues['content'] |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3502 del propvalues['content'] |
| 1165 | 3503 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3504 # do the database create |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3505 propvalues = self.set_inner(itemid, **propvalues) |
| 1165 | 3506 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3507 # do content? |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3508 if content: |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3509 # store and possibly index |
|
5492
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3510 self.db.storefile(self.classname, itemid, None, bs2b(content)) |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3511 if self.properties['content'].indexme: |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3512 mime_type = self.get(itemid, 'type', self.default_mime_type) |
|
5544
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3513 index_content = content |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3514 if bytes != str and isinstance(content, bytes): |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3515 index_content = content.decode('utf-8', errors='ignore') |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3516 self.db.indexer.add_text((self.classname, itemid, 'content'), |
| 6965 | 3517 index_content, mime_type) |
|
2640
b01eca163779
The "type" parameter is supposed to be optional
Richard Jones <richard@users.sourceforge.net>
parents:
2634
diff
changeset
|
3518 propvalues['content'] = content |
|
b01eca163779
The "type" parameter is supposed to be optional
Richard Jones <richard@users.sourceforge.net>
parents:
2634
diff
changeset
|
3519 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3520 # fire reactors |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3521 self.fireReactors('set', itemid, oldvalues) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3522 return propvalues |
| 1165 | 3523 |
|
2505
bdd112cf61ba
rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
3524 def index(self, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3525 """ Add (or refresh) the node to search indexes. |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3526 |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3527 Use the content-type property for the content property. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3528 """ |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3529 # find all the String properties that have indexme |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
3530 for prop, propclass in self.getprops().items(): |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3531 if prop == 'content' and propclass.indexme: |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3532 mime_type = self.get(nodeid, 'type', self.default_mime_type) |
|
5544
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3533 index_content = self.get(nodeid, 'binary_content') |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3534 if bytes != str and isinstance(index_content, bytes): |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3535 index_content = index_content.decode('utf-8', |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3536 errors='ignore') |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3537 self.db.indexer.add_text((self.classname, nodeid, 'content'), |
| 6965 | 3538 index_content, mime_type) |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3539 elif isinstance(propclass, hyperdb.String) and propclass.indexme: |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3540 # index them under (classname, nodeid, property) |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3541 try: |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3542 value = str(self.get(nodeid, prop)) |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3543 except IndexError: |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3544 # node has been destroyed |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3545 continue |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3546 self.db.indexer.add_text((self.classname, nodeid, prop), value) |
|
2505
bdd112cf61ba
rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
3547 |
| 6965 | 3548 |
| 1165 | 3549 # XXX deviation from spec - was called ItemClass |
| 3550 class IssueClass(Class, roundupdb.IssueClass): | |
| 3551 # Overridden methods: | |
| 3552 def __init__(self, db, classname, **properties): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3553 """The newly-created class automatically includes the "messages", |
| 1165 | 3554 "files", "nosy", and "superseder" properties. If the 'properties' |
| 3555 dictionary attempts to specify any of these properties or a | |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
3556 "creation", "creator", "activity" or "actor" property, a ValueError |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
3557 is raised. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3558 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3559 if 'title' not in properties: |
| 1165 | 3560 properties['title'] = hyperdb.String(indexme='yes') |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3561 if 'messages' not in properties: |
| 1165 | 3562 properties['messages'] = hyperdb.Multilink("msg") |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3563 if 'files' not in properties: |
| 1165 | 3564 properties['files'] = hyperdb.Multilink("file") |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3565 if 'nosy' not in properties: |
| 1165 | 3566 # note: journalling is turned off as it really just wastes |
| 3567 # space. this behaviour may be overridden in an instance | |
| 3568 properties['nosy'] = hyperdb.Multilink("user", do_journal="no") | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3569 if 'superseder' not in properties: |
| 1165 | 3570 properties['superseder'] = hyperdb.Multilink(classname) |
| 3571 Class.__init__(self, db, classname, **properties) | |
| 3572 | |
|
2692
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
3573 # vim: set et sts=4 sw=4 : |
