Mercurial > p > roundup > code
annotate roundup/backends/rdbms_common.py @ 6588:91ab3e0ffcd0
Summary: Add test cases for sqlite fts
Add support for using the FTS5 full text query engine for sqlite.
Also stubbed out some sections for adding postgresql FTS support as
well.
Added nee indexer type native-fts. It is not selected by default. The
indexer=native is used if no indexer is set. This prevents an upgrade
from seeming to wipe out the native index if upgraded and
indexer=native is not explicitly set.
Docs updated. Also changed section headers to sentence case for the
current release notes.
Indexing backend can control if the full text search phrase is broken
into a list of words or passed intact. For backends with query
languages (sqlite and can be enabled for whoosh and xapian) we do not
want the phrase "tokenized" on whitespace.
This also updates the rdbms database version to version 7 to add FTS
table. I will be using the same version when I add postgresql. If
somebody runs this version on postgresql, they will have to manually
add the fts tables for postgresql if they want to use it.
Added a new renderError method to client. This allows errors to be
reported still using page.html rather than raw html. It also supports
templates for any error code. If no template for the error code
(e.g. 400) is found, the error in raw html with no page frame is
shown.
New IndexerQueryError exception to pass back message about query syntax
errors.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Sun, 23 Jan 2022 18:57:45 -0500 |
| parents | c1d3fbcdbfbd |
| children | 39189dd94f2c |
| rev | line source |
|---|---|
|
3554
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
1 # |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/) |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
3 # This module is free software, and you may redistribute it and/or modify |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
4 # under the same terms as Python, so long as this copyright message and |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
5 # disclaimer are retained in their original form. |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
6 # |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
10 # POSSIBILITY OF SUCH DAMAGE. |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
11 # |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS" |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
17 # |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
18 """ Relational database (SQL) backend common code. |
|
1244
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
19 |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
20 Basics: |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
21 |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
22 - map roundup classes to relational tables |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
23 - automatically detect schema changes and modify the table schemas |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
24 appropriately (we store the "database version" of the schema in the |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
25 database itself as the only row of the "schema" table) |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
26 - multilinks (which represent a many-to-many relationship) are handled through |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
27 intermediate tables |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
28 - journals are stored adjunct to the per-class tables |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
29 - table names and columns have "_" prepended so the names can't clash with |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
30 restricted names (like "order") |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
31 - retirement is determined by the __retired__ column being > 0 |
|
1244
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
32 |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
33 Database-specific changes may generally be pushed out to the overridable |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
34 sql_* methods, since everything else should be fairly generic. There's |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
35 probably a bit of work to be done if a database is used that actually |
|
8dd4f736370b
merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents:
1222
diff
changeset
|
36 honors column typing, since the initial databases don't (sqlite stores |
|
1528
96cd422532ef
bye bye gadfly - you served your purpose well [SF#701127]
Richard Jones <richard@users.sourceforge.net>
parents:
1523
diff
changeset
|
37 everything as a string.) |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
38 |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
39 The schema of the hyperdb being mapped to the database is stored in the |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
40 database itself as a repr()'ed dictionary of information about each Class |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
41 that maps to a table. If that information differs from the hyperdb schema, |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
42 then we update it. We also store in the schema dict a version which |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
43 allows us to upgrade the database schema when necessary. See upgrade_db(). |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
44 |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
45 To force a unqiueness constraint on the key properties we put the item |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
46 id into the __retired__ column duing retirement (so it's 0 for "active" |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
47 items) and place a unqiueness constraint on key + __retired__. This is |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
48 particularly important for the users class where multiple users may |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
49 try to have the same username, with potentially many retired users with |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
50 the same name. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
51 """ |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
52 __docformat__ = 'restructuredtext' |
| 1165 | 53 |
| 54 # standard python modules | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
55 import os, time, re, weakref, copy, logging, datetime |
| 1165 | 56 |
| 57 # roundup modules | |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
58 from roundup import hyperdb, date, password, roundupdb, security, support |
| 1165 | 59 from roundup.hyperdb import String, Password, Date, Interval, Link, \ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
60 Multilink, DatabaseError, Boolean, Number, Integer |
|
3925
603ec9630b08
i18n for hyperdb and backend errors
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3924
diff
changeset
|
61 from roundup.i18n import _ |
| 1165 | 62 |
| 63 # support | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
64 from roundup.backends.blobfiles import FileStorage |
|
5096
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
5086
diff
changeset
|
65 from roundup.backends.indexer_common import get_indexer |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
66 from roundup.backends.sessions_rdbms import Sessions, OneTimeKeys |
|
1499
8ee69708da0c
added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1496
diff
changeset
|
67 from roundup.date import Range |
| 1165 | 68 |
|
6401
8bc5faeb7677
Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6400
diff
changeset
|
69 from roundup.mlink_expr import compile_expression |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
70 from roundup.anypy.strings import b2s, bs2b, us2s, repr_export, eval_import |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
71 |
|
5725
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
72 from hashlib import md5 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
73 |
|
2692
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
74 # dummy value meaning "argument not passed" |
|
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
75 _marker = [] |
|
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
76 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
77 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
78 def _num_cvt(num): |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
79 num = str(num) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
80 try: |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
81 return int(num) |
| 6002 | 82 except ValueError: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
83 return float(num) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
84 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
85 |
|
2472
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
86 def _bool_cvt(value): |
|
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
87 if value in ('TRUE', 'FALSE'): |
|
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
88 return {'TRUE': 1, 'FALSE': 0}[value] |
|
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
89 # assume it's a number returned from the db API |
|
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
90 return int(value) |
|
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
91 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
92 |
|
4473
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
93 def date_to_hyperdb_value(d): |
|
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
94 """ convert date d to a roundup date """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
95 if isinstance(d, datetime.datetime): |
|
4473
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
96 return date.Date(d) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
97 return date.Date(str(d).replace(' ', '.')) |
|
4473
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
98 |
|
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
99 |
|
2634
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
100 def connection_dict(config, dbnamestr=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
101 """ Used by Postgresql and MySQL to detemine the keyword args for |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
102 opening the database connection.""" |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
103 d = {} |
|
2634
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
104 if dbnamestr: |
|
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
105 d[dbnamestr] = config.RDBMS_NAME |
|
3099
519b92df37dc
handle ~/.my.cnf files for MySQL defaults [SF#1096031]
Richard Jones <richard@users.sourceforge.net>
parents:
3092
diff
changeset
|
106 for name in ('host', 'port', 'password', 'user', 'read_default_group', |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
107 'read_default_file'): |
|
2634
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
108 cvar = 'RDBMS_'+name.upper() |
|
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
109 if config[cvar] is not None: |
|
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
110 d[name] = config[cvar] |
|
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
111 return d |
|
f47ca4541770
Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents:
2597
diff
changeset
|
112 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
113 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
114 class IdListOptimizer: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
115 """ To prevent flooding the SQL parser of the underlaying |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
116 db engine with "x IN (1, 2, 3, ..., <large number>)" collapses |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
117 these cases to "x BETWEEN 1 AND <large number>". |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
118 """ |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
119 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
120 def __init__(self): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
121 self.ranges = [] |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
122 self.singles = [] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
123 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
124 def append(self, nid): |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
125 """ Invariant: nid are ordered ascending """ |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
126 if self.ranges: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
127 last = self.ranges[-1] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
128 if last[1] == nid-1: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
129 last[1] = nid |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
130 return |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
131 if self.singles: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
132 last = self.singles[-1] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
133 if last == nid-1: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
134 self.singles.pop() |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
135 self.ranges.append([last, nid]) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
136 return |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
137 self.singles.append(nid) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
138 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
139 def where(self, field, placeholder): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
140 ranges = self.ranges |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
141 singles = self.singles |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
142 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
143 if not singles and not ranges: return "(1=0)", [] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
144 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
145 if ranges: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
146 between = '%s BETWEEN %s AND %s' % ( |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
147 field, placeholder, placeholder) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
148 stmnt = [between] * len(ranges) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
149 else: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
150 stmnt = [] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
151 if singles: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
152 stmnt.append('%s in (%s)' % ( |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
153 field, ','.join([placeholder]*len(singles)))) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
154 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
155 return '(%s)' % ' OR '.join(stmnt), sum(ranges, []) + singles |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
156 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
157 def __str__(self): |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
158 return "ranges: %r / singles: %r" % (self.ranges, self.singles) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
159 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
160 |
| 1165 | 161 class Database(FileStorage, hyperdb.Database, roundupdb.Database): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
162 """ Wrapper around an SQL database that presents a hyperdb interface. |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
163 |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
164 - some functionality is specific to the actual SQL database, hence |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
165 the sql_* methods that are NotImplemented |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
166 - we keep a cache of the latest N row fetches (where N is |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
167 configurable). |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
168 """ |
| 1165 | 169 def __init__(self, config, journaltag=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
170 """ Open the database and load the schema from it. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
171 """ |
|
3897
e2c2d91932ad
respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3872
diff
changeset
|
172 FileStorage.__init__(self, config.UMASK) |
| 1165 | 173 self.config, self.journaltag = config, journaltag |
| 174 self.dir = config.DATABASE | |
| 175 self.classes = {} | |
|
5096
e74c3611b138
- issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents:
5086
diff
changeset
|
176 self.indexer = get_indexer(config, self) |
| 1165 | 177 self.security = security.Security(self) |
| 178 | |
| 179 # additional transaction support for external files and the like | |
| 180 self.transactions = [] | |
| 181 | |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
182 # keep a cache of the N most recently retrieved rows of any kind |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
183 # (classname, nodeid) = row |
|
4074
e039f3cbbb96
Make RDBMS cache-size configurable.
Stefan Seefeld <stefan@seefeld.name>
parents:
4062
diff
changeset
|
184 self.cache_size = config.RDBMS_CACHE_SIZE |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
185 self.clearCache() |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
186 self.stats = {'cache_hits': 0, 'cache_misses': 0, 'get_items': 0, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
187 'filtering': 0} |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
188 |
|
5041
5251e97b1de0
Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents:
4995
diff
changeset
|
189 # make sure the database directory exists |
|
5251e97b1de0
Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents:
4995
diff
changeset
|
190 if not os.path.isdir(self.config.DATABASE): |
|
5251e97b1de0
Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents:
4995
diff
changeset
|
191 os.makedirs(self.config.DATABASE) |
|
5251e97b1de0
Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents:
4995
diff
changeset
|
192 |
|
1333
80d27b7d6db5
implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents:
1304
diff
changeset
|
193 # database lock |
|
80d27b7d6db5
implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents:
1304
diff
changeset
|
194 self.lockfile = None |
|
80d27b7d6db5
implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents:
1304
diff
changeset
|
195 |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
196 # Uppercase to not collide with Class names |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
197 self.Session = None |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
198 self.Otk = None |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
199 |
| 1165 | 200 # open a connection to the database, creating the "conn" attribute |
|
2082
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
201 self.open_connection() |
| 1165 | 202 |
|
1181
49aebf5a8691
some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents:
1176
diff
changeset
|
203 def clearCache(self): |
|
49aebf5a8691
some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents:
1176
diff
changeset
|
204 self.cache = {} |
|
49aebf5a8691
some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents:
1176
diff
changeset
|
205 self.cache_lru = [] |
|
4652
dfbc0cfa9811
Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4517
diff
changeset
|
206 # upcall is necessary! |
|
dfbc0cfa9811
Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4517
diff
changeset
|
207 roundupdb.Database.clearCache(self) |
|
1181
49aebf5a8691
some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents:
1176
diff
changeset
|
208 |
|
2082
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
209 def getSessionManager(self): |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
210 if not self.Session: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
211 self.Session = Sessions(self) |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
212 return self.Session |
|
2082
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
213 |
|
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
214 def getOTKManager(self): |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
215 if not self.Otk: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
216 self.Otk = OneTimeKeys(self) |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
217 return self.Otk |
|
2082
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
218 |
|
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
2081
diff
changeset
|
219 def open_connection(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
220 """ Open a connection to the database, creating it if necessary. |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
221 |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
222 Must call self.load_dbschema() |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
223 """ |
|
5236
0a20e1965f4b
Pylint checks bark about:
John Rouillard <rouilj@ieee.org>
parents:
5232
diff
changeset
|
224 raise NotImplementedError |
| 1165 | 225 |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
226 def sql(self, sql, args=None, cursor=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
227 """ Execute the sql with the optional args. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
228 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
229 self.log_debug('SQL %r %r' % (sql, args)) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
230 if not cursor: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
231 cursor = self.cursor |
| 1165 | 232 if args: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
233 cursor.execute(sql, args) |
| 1165 | 234 else: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
235 cursor.execute(sql) |
| 1165 | 236 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
237 def sql_fetchone(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
238 """ Fetch a single row. If there's nothing to fetch, return None. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
239 """ |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
240 return self.cursor.fetchone() |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
241 |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
242 def sql_fetchall(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
243 """ Fetch all rows. If there's nothing to fetch, return []. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
244 """ |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
245 return self.cursor.fetchall() |
| 1165 | 246 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
247 def sql_fetchiter(self): |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
248 """ Fetch all row as a generator |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
249 """ |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
250 while True: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
251 row = self.cursor.fetchone() |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
252 if not row: break |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
253 yield row |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
254 |
|
4849
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
255 def search_stringquote(self, value): |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
256 """ Quote a search string to escape magic search characters |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
257 '%' and '_', also need to quote '\' (first) |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
258 Then put '%' around resulting string for LIKE (or ILIKE) operator |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
259 """ |
|
4849
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
260 v = value.replace('\\', '\\\\') |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
261 v = v.replace('%', '\\%') |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
262 v = v.replace('_', '\\_') |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
263 return '%' + v + '%' |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
264 |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
265 def init_dbschema(self): |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
266 self.database_schema = { |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
267 'version': self.current_db_version, |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
268 'tables': {} |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
269 } |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
270 |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
271 def load_dbschema(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
272 """ Load the schema definition that the database currently implements |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
273 """ |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
274 self.cursor.execute('select schema from schema') |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
275 schema = self.cursor.fetchone() |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
276 if schema: |
| 6002 | 277 # bandit - schema is trusted |
| 278 self.database_schema = eval(schema[0]) # nosec | |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
279 else: |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
280 self.database_schema = {} |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
281 |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
282 def save_dbschema(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
283 """ Save the schema definition that the database currently implements |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
284 """ |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
285 s = repr(self.database_schema) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
286 self.sql('delete from schema') |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
287 self.sql('insert into schema values (%s)' % self.arg, (s,)) |
| 1165 | 288 |
| 289 def post_init(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
290 """ Called once the schema initialisation has finished. |
| 1165 | 291 |
| 292 We should now confirm that the schema defined by our "classes" | |
| 293 attribute actually matches the schema in the database. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
294 """ |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
295 super(Database, self).post_init() |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
296 |
|
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
297 # upgrade the database for column type changes, new internal |
|
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
298 # tables, etc. |
|
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
299 save = self.upgrade_db() |
| 3553 | 300 |
| 301 # handle changes in the schema | |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
302 tables = self.database_schema['tables'] |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
303 for classname, spec in self.classes.items(): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
304 if classname in tables: |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
305 dbspec = tables[classname] |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
306 if self.update_class(spec, dbspec): |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
307 tables[classname] = spec.schema() |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
308 save = 1 |
| 1165 | 309 else: |
| 310 self.create_class(spec) | |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
311 tables[classname] = spec.schema() |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
312 save = 1 |
| 1165 | 313 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
314 for classname, _spec in list(tables.items()): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
315 if classname not in self.classes: |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
316 self.drop_class(classname, tables[classname]) |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
317 del tables[classname] |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
318 save = 1 |
| 1165 | 319 |
| 320 # update the database version of the schema | |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
321 if save: |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
322 self.save_dbschema() |
| 1165 | 323 |
| 324 # reindex the db if necessary | |
| 325 if self.indexer.should_reindex(): | |
| 326 self.reindex() | |
| 327 | |
| 328 # commit | |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
2089
diff
changeset
|
329 self.sql_commit() |
| 1165 | 330 |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
331 # update this number when we need to make changes to the SQL structure |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
332 # of the backend database |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
333 current_db_version = 7 |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
334 db_version_updated = False |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
335 |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
336 def upgrade_db(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
337 """ Update the SQL database to reflect changes in the backend code. |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
338 |
|
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
339 Return boolean whether we need to save the schema. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
340 """ |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
341 version = self.database_schema.get('version', 1) |
|
3993
1eba65a4b0aa
Don't run old code on newer database
Richard Jones <richard@users.sourceforge.net>
parents:
3979
diff
changeset
|
342 if version > self.current_db_version: |
|
1eba65a4b0aa
Don't run old code on newer database
Richard Jones <richard@users.sourceforge.net>
parents:
3979
diff
changeset
|
343 raise DatabaseError('attempting to run rev %d DATABASE with rev ' |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
344 '%d CODE!' % (version, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
345 self.current_db_version)) |
|
2081
fb4bf55b94d7
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2077
diff
changeset
|
346 if version == self.current_db_version: |
|
fb4bf55b94d7
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2077
diff
changeset
|
347 # nothing to do |
|
fb4bf55b94d7
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2077
diff
changeset
|
348 return 0 |
|
fb4bf55b94d7
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2077
diff
changeset
|
349 |
|
2429
e28a5d5d95a7
better upgrading
Richard Jones <richard@users.sourceforge.net>
parents:
2424
diff
changeset
|
350 if version < 2: |
| 4062 | 351 self.log_info('upgrade to version 2') |
|
2100
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
352 # change the schema structure |
|
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
353 self.database_schema = {'tables': self.database_schema} |
|
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
354 |
|
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
355 # version 1 didn't have the actor column (note that in |
|
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
356 # MySQL this will also transition the tables to typed columns) |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
357 self.add_new_columns_v2() |
|
2100
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
358 |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
359 # version 1 doesn't have the OTK, session and indexing in the |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
360 # database |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
361 self.create_version_2_tables() |
|
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
362 |
|
2429
e28a5d5d95a7
better upgrading
Richard Jones <richard@users.sourceforge.net>
parents:
2424
diff
changeset
|
363 if version < 3: |
| 4062 | 364 self.log_info('upgrade to version 3') |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
365 self.fix_version_2_tables() |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
366 |
|
2721
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
367 if version < 4: |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
368 self.log_info('upgrade to version 4') |
|
2721
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
369 self.fix_version_3_tables() |
|
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
370 |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
371 if version < 5: |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
372 self.log_info('upgrade to version 5') |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
373 self.fix_version_4_tables() |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
374 |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
375 if version < 6: |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
376 self.log_info('upgrade to version 6') |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
377 self.fix_version_5_tables() |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
378 |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
379 if version < 7: |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
380 self.log_info('upgrade to version 7') |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
381 self.fix_version_6_tables() |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
382 |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
383 self.database_schema['version'] = self.current_db_version |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
384 self.db_version_updated = True |
|
2075
b1704ba7be41
make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents:
2073
diff
changeset
|
385 return 1 |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
386 |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
387 def fix_version_2_tables(self): |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
388 # Default (used by sqlite): NOOP |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
389 pass |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
390 |
|
2721
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
391 def fix_version_3_tables(self): |
|
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
392 # drop the shorter VARCHAR OTK column and add a new TEXT one |
|
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
393 for name in ('otk', 'session'): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
394 self.sql('DELETE FROM %ss' % name) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
395 self.sql('ALTER TABLE %ss DROP %s_value' % (name, name)) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
396 self.sql('ALTER TABLE %ss ADD %s_value TEXT' % (name, name)) |
|
2721
1cd01cf106e1
extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents:
2720
diff
changeset
|
397 |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
398 def fix_version_4_tables(self): |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
399 # note this is an explicit call now |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
400 c = self.cursor |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
401 for cn, klass in self.classes.items(): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
402 c.execute('select id from _%s where __retired__<>0' % (cn,)) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
403 for (id,) in c.fetchall(): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
404 c.execute('update _%s set __retired__=%s where id=%s' % (cn, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
405 self.arg, self.arg), (id, id)) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
406 |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
407 if klass.key: |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
408 self.add_class_key_required_unique_constraint(cn, klass.key) |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
409 |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
410 def fix_version_5_tables(self): |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
411 # Default (used by sqlite, postgres): NOOP |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
412 # mysql overrides this because it is missing |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
413 # _<class>_key_retired_idx index used to make |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
414 # sure that the key is unique if it was created |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
415 # as version 5. |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
416 pass |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
417 |
|
6588
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
418 def fix_version_6_tables(self): |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
419 # Default (used by mysql): NOOP |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
420 # sqlite/postgres override this to add fts |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
421 # full text search tables. |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
422 pass |
|
91ab3e0ffcd0
Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents:
6433
diff
changeset
|
423 |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
424 def _convert_journal_tables(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
425 """Get current journal table contents, drop the table and re-create""" |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
426 c = self.cursor |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
427 cols = ','.join('nodeid date tag action params'.split()) |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
428 for klass in self.classes.values(): |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
429 # slurp and drop |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
430 sql = 'select %s from %s__journal order by date' % \ |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
431 (cols, klass.classname) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
432 c.execute(sql) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
433 contents = c.fetchall() |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
434 self.drop_journal_table_indexes(klass.classname) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
435 c.execute('drop table %s__journal' % klass.classname) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
436 |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
437 # re-create and re-populate |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
438 self.create_journal_table(klass) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
439 a = self.arg |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
440 sql = 'insert into %s__journal (%s) values (%s,%s,%s,%s,%s)' % ( |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
441 klass.classname, cols, a, a, a, a, a) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
442 for row in contents: |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
443 # no data conversion needed |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
444 self.cursor.execute(sql, row) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
445 |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
446 def _convert_string_properties(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
447 """Get current Class tables that contain String properties, and |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
448 convert the VARCHAR columns to TEXT""" |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
449 c = self.cursor |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
450 for klass in self.classes.values(): |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
451 # slurp and drop |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
452 cols, mls = self.determine_columns(list(klass.properties.items())) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
453 scols = ','.join([i[0] for i in cols]) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
454 sql = 'select id,%s from _%s' % (scols, klass.classname) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
455 c.execute(sql) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
456 contents = c.fetchall() |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
457 self.drop_class_table_indexes(klass.classname, klass.getkey()) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
458 c.execute('drop table _%s' % klass.classname) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
459 |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
460 # re-create and re-populate |
|
2424
74474ec41050
argh! backwards compat
Richard Jones <richard@users.sourceforge.net>
parents:
2416
diff
changeset
|
461 self.create_class_table(klass, create_sequence=0) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
462 a = ','.join([self.arg for i in range(len(cols)+1)]) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
463 sql = 'insert into _%s (id,%s) values (%s)' % (klass.classname, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
464 scols, a) |
|
2413
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
465 for row in contents: |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
466 l = [] |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
467 for entry in row: |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
468 # mysql will already be a string - psql needs "help" |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
469 if entry is not None and not isinstance(entry, type('')): |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
470 entry = str(entry) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
471 l.append(entry) |
|
7d0bb6601809
fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents:
2379
diff
changeset
|
472 self.cursor.execute(sql, l) |
|
2073
261c2e6ceb1e
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
473 |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
474 def refresh_database(self): |
|
1951
767ff2a03eee
more unit tests to improve coverage (up from 85% to 88% for anydbm! :)
Richard Jones <richard@users.sourceforge.net>
parents:
1926
diff
changeset
|
475 self.post_init() |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
476 |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
477 def reindex(self, classname=None, show_progress=False): |
|
2650
d68a444fcce3
roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents:
2649
diff
changeset
|
478 if classname: |
|
d68a444fcce3
roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents:
2649
diff
changeset
|
479 classes = [self.getclass(classname)] |
|
d68a444fcce3
roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents:
2649
diff
changeset
|
480 else: |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
481 classes = list(self.classes.values()) |
|
2650
d68a444fcce3
roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents:
2649
diff
changeset
|
482 for klass in classes: |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
483 if show_progress: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
484 for nodeid in support.Progress('Reindex %s' % klass.classname, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
485 klass.list()): |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
486 klass.index(nodeid) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
487 else: |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
488 for nodeid in klass.list(): |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
489 klass.index(nodeid) |
|
3295
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3239
diff
changeset
|
490 self.indexer.save_index() |
| 1165 | 491 |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
492 def checkpoint_data(self): |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
493 """Call if you need to commit the state of the database |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
494 so you can try to fix the error rather than rolling back |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
495 |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
496 Needed for postgres when importing data. |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
497 """ |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
498 pass |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
499 |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
500 def restore_connection_on_error(self): |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
501 """on a database error/exception recover the db connection |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
502 if left in an unusable state (e.g. postgres requires |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
503 a rollback). |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
504 """ |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
505 pass |
|
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
506 |
|
5175
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
507 # Used here in the generic backend to determine if the database |
|
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
508 # supports 'DOUBLE PRECISION' for floating point numbers. |
|
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
509 implements_double_precision = True |
|
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
510 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
511 hyperdb_to_sql_datatypes = { |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
512 hyperdb.String : 'TEXT', |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
513 hyperdb.Date : 'TIMESTAMP', |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
514 hyperdb.Link : 'INTEGER', |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
515 hyperdb.Interval : 'VARCHAR(255)', |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
516 hyperdb.Password : 'VARCHAR(255)', |
|
2166
cd42c3c7173a
MySQL and Postgresql use BOOL/BOOLEAN for Boolean types
Richard Jones <richard@users.sourceforge.net>
parents:
2102
diff
changeset
|
517 hyperdb.Boolean : 'BOOLEAN', |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
518 hyperdb.Number : 'REAL', |
|
5067
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
519 hyperdb.Integer : 'INTEGER', |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
520 } |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
521 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
522 def hyperdb_to_sql_datatype(self, propclass, prop=None): |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
523 |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
524 datatype = self.hyperdb_to_sql_datatypes.get(propclass) |
|
5175
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
525 if self.implements_double_precision and prop and \ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
526 isinstance(prop, Number) and prop.use_double: |
|
5175
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
527 datatype = 'DOUBLE PRECISION' |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
528 if datatype: |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
529 return datatype |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
530 |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
531 for k, v in self.hyperdb_to_sql_datatypes.items(): |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
532 if issubclass(propclass, k): |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
533 return v |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
534 |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
535 raise ValueError('%r is not a hyperdb property class' % propclass) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
536 |
| 1165 | 537 def determine_columns(self, properties): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
538 """ Figure the column names and multilink properties from the spec |
| 1165 | 539 |
| 540 "properties" is a list of (name, prop) where prop may be an | |
| 541 instance of a hyperdb "type" _or_ a string repr of that type. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
542 """ |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
543 cols = [ |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
544 ('_actor', self.hyperdb_to_sql_datatype(hyperdb.Link)), |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
545 ('_activity', self.hyperdb_to_sql_datatype(hyperdb.Date)), |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
546 ('_creator', self.hyperdb_to_sql_datatype(hyperdb.Link)), |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
547 ('_creation', self.hyperdb_to_sql_datatype(hyperdb.Date)), |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
548 ] |
| 1165 | 549 mls = [] |
| 550 # add the multilinks separately | |
| 551 for col, prop in properties: | |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
552 # Computed props are not in the db |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
553 if prop.computed: |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
554 continue |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
555 |
| 1165 | 556 if isinstance(prop, Multilink): |
| 557 mls.append(col) | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
558 continue |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
559 |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
560 if isinstance(prop, type('')): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
561 raise ValueError("string property spec!") |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
562 # and prop.find('Multilink') != -1: |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
563 # mls.append(col) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
564 |
|
5175
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
565 datatype = self.hyperdb_to_sql_datatype(prop.__class__, prop) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
566 cols.append(('_'+col, datatype)) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
567 |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
568 # Intervals stored as two columns |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
569 if isinstance(prop, Interval): |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
570 cols.append(('__'+col+'_int__', 'BIGINT')) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
571 |
| 1165 | 572 cols.sort() |
| 573 return cols, mls | |
| 574 | |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
575 def update_class(self, spec, old_spec, force=0): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
576 """ Determine the differences between the current spec and the |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
577 database version of the spec, and update where necessary. |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
578 |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
579 If 'force' is true, update the database anyway. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
580 """ |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
581 new_spec = spec.schema() |
|
1515
a516bbb9896b
fixed rdbms table update detection logic [SF#703297]
Richard Jones <richard@users.sourceforge.net>
parents:
1500
diff
changeset
|
582 new_spec[1].sort() |
|
a516bbb9896b
fixed rdbms table update detection logic [SF#703297]
Richard Jones <richard@users.sourceforge.net>
parents:
1500
diff
changeset
|
583 old_spec[1].sort() |
|
1840
91a4619b1a14
hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1838
diff
changeset
|
584 if not force and new_spec == old_spec: |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
585 # no changes |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
586 return 0 |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
587 |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
588 if not self.config.RDBMS_ALLOW_ALTER: |
|
6133
e7cb0147e6fe
Make format strings use named placeholders rather than %s/%r
John Rouillard <rouilj@ieee.org>
parents:
6118
diff
changeset
|
589 raise DatabaseError(_( |
|
e7cb0147e6fe
Make format strings use named placeholders rather than %s/%r
John Rouillard <rouilj@ieee.org>
parents:
6118
diff
changeset
|
590 'ALTER operation disallowed: %(old)r -> %(new)r.'% { |
|
e7cb0147e6fe
Make format strings use named placeholders rather than %s/%r
John Rouillard <rouilj@ieee.org>
parents:
6118
diff
changeset
|
591 'old': old_spec, 'new': new_spec})) |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
592 |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
593 logger = logging.getLogger('roundup.hyperdb.backend') |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
594 logger.info('update_class %s' % spec.classname) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
595 |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
596 logger.debug('old_spec %r' % (old_spec,)) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
597 logger.debug('new_spec %r' % (new_spec,)) |
| 1165 | 598 |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
599 # detect key prop change for potential index change |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
600 keyprop_changes = {} |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
601 if new_spec[0] != old_spec[0]: |
|
3554
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
602 if old_spec[0]: |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
603 keyprop_changes['remove'] = old_spec[0] |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
604 if new_spec[0]: |
|
5e70726a86dd
fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents:
3553
diff
changeset
|
605 keyprop_changes['add'] = new_spec[0] |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
606 |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
607 # detect multilinks that have been removed, and drop their table |
|
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
608 old_has = {} |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
609 for name, prop in old_spec[1]: |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
610 old_has[name] = 1 |
|
6392
99455aeec1ae
Fix dropping of multilinks on schema change
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6332
diff
changeset
|
611 if name in spec.properties and not spec.properties[name].computed: |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
612 continue |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
613 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
614 if prop.find('Multilink to') != -1: |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
615 # first drop indexes. |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
616 self.drop_multilink_table_indexes(spec.classname, name) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
617 |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
618 # now the multilink table itself |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
619 sql = 'drop table %s_%s' % (spec.classname, name) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
620 else: |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
621 # if this is the key prop, drop the index first |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
622 if old_spec[0] == prop: |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
623 self.drop_class_table_key_index(spec.classname, name) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
624 del keyprop_changes['remove'] |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
625 |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
626 # drop the column |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
627 sql = 'alter table _%s drop column _%s' % (spec.classname, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
628 name) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
629 |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
630 self.sql(sql) |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
631 |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
632 # if we didn't remove the key prop just then, but the key prop has |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
633 # changed, we still need to remove the old index |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
634 if 'remove' in keyprop_changes: |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
635 self.drop_class_table_key_index(spec.classname, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
636 keyprop_changes['remove']) |
| 1165 | 637 |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
638 # add new columns |
|
2196
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
639 for propname, prop in new_spec[1]: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
640 if propname in old_has: |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
641 continue |
|
2196
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
642 prop = spec.properties[propname] |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
643 if isinstance(prop, Multilink): |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
644 self.create_multilink_table(spec, propname) |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
645 else: |
|
2727
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
646 # add the column |
|
5175
e1e40674a0bc
Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5112
diff
changeset
|
647 coltype = self.hyperdb_to_sql_datatype(prop.__class__, prop) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
648 sql = 'alter table _%s add column _%s %s' % ( |
|
2727
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
649 spec.classname, propname, coltype) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
650 self.sql(sql) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
651 |
|
2727
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
652 # extra Interval column |
|
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
653 if isinstance(prop, Interval): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
654 sql = 'alter table _%s add column __%s_int__ BIGINT' % ( |
|
2727
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
655 spec.classname, propname) |
|
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
656 self.sql(sql) |
|
93e2e5b55a3c
new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents:
2721
diff
changeset
|
657 |
|
2196
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
658 # if the new column is a key prop, we need an index! |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
659 if new_spec[0] == propname: |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
660 self.create_class_table_key_index(spec.classname, propname) |
|
85954067e496
mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents:
2185
diff
changeset
|
661 del keyprop_changes['add'] |
| 1165 | 662 |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
663 # if we didn't add the key prop just then, but the key prop has |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
664 # changed, we still need to add the new index |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
665 if 'add' in keyprop_changes: |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
666 self.create_class_table_key_index(spec.classname, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
667 keyprop_changes['add']) |
|
1479
405e91b5be46
fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents:
1476
diff
changeset
|
668 |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
669 return 1 |
| 1165 | 670 |
|
3016
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
671 def determine_all_columns(self, spec): |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
672 """Figure out the columns from the spec and also add internal columns |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
673 |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
674 """ |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
675 cols, mls = self.determine_columns(list(spec.properties.items())) |
|
3082
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
676 |
|
3016
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
677 # add on our special columns |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
678 cols.append(('id', 'INTEGER PRIMARY KEY')) |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
679 cols.append(('__retired__', 'INTEGER DEFAULT 0')) |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
680 return cols, mls |
|
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
681 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
682 def create_class_table(self, spec): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
683 """Create the class table for the given Class "spec". Creates the |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
684 indexes too.""" |
|
3016
224c7c0b9708
First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2985
diff
changeset
|
685 cols, mls = self.determine_all_columns(spec) |
| 1165 | 686 |
| 687 # create the base table | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
688 scols = ','.join(['%s %s' % x for x in cols]) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
689 sql = 'create table _%s (%s)' % (spec.classname, scols) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
690 self.sql(sql) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
691 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
692 self.create_class_table_indexes(spec) |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
693 |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
694 return cols, mls |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
695 |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
696 def create_class_table_indexes(self, spec): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
697 """ create the class table for the given spec |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
698 """ |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
699 # create __retired__ index |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
700 index_sql2 = 'create index _%s_retired_idx on _%s(__retired__)' % ( |
|
1836
94e430ad4fdb
make the RDBMS common backend and the SQLite and MYsql backend create...
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1800
diff
changeset
|
701 spec.classname, spec.classname) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
702 self.sql(index_sql2) |
| 1165 | 703 |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
704 # create index for key property |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
705 if spec.key: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
706 index_sql3 = 'create index _%s_%s_idx on _%s(_%s)' % ( |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
707 spec.classname, spec.key, |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
708 spec.classname, spec.key) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
709 self.sql(index_sql3) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
710 |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
711 # and the unique index for key / retired(id) |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
712 self.add_class_key_required_unique_constraint(spec.classname, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
713 spec.key) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
714 |
|
2228
1d1362c54c94
Some doc / comment fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
2217
diff
changeset
|
715 # TODO: create indexes on (selected?) Link property columns, as |
|
1d1362c54c94
Some doc / comment fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
2217
diff
changeset
|
716 # they're more likely to be used for lookup |
|
1d1362c54c94
Some doc / comment fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
2217
diff
changeset
|
717 |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
718 def add_class_key_required_unique_constraint(self, cn, key): |
|
3969
905faf52a51f
fix mysql breakage in 1.4.2
Richard Jones <richard@users.sourceforge.net>
parents:
3967
diff
changeset
|
719 sql = '''create unique index _%s_key_retired_idx |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
720 on _%s(__retired__, _%s)''' % (cn, cn, key) |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
721 try: |
|
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
722 self.sql(sql) |
| 6002 | 723 except Exception: # nosec |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
724 # XXX catch e.g.: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
725 # _sqlite.DatabaseError: index _status_key_retired_idx |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
726 # already exists |
|
4774
3adff0fb0207
Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents:
4652
diff
changeset
|
727 pass |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
728 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
729 def drop_class_table_indexes(self, cn, key): |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
730 # drop the old table indexes first |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
731 l = ['_%s_id_idx' % cn, '_%s_retired_idx' % cn] |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
732 if key: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
733 l.append('_%s_%s_idx' % (cn, key)) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
734 |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
735 table_name = '_%s' % cn |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
736 for index_name in l: |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
737 if not self.sql_index_exists(table_name, index_name): |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
738 continue |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
739 index_sql = 'drop index '+index_name |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
740 self.sql(index_sql) |
| 1165 | 741 |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
742 def create_class_table_key_index(self, cn, key): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
743 """ create the class table for the given spec |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
744 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
745 sql = 'create index _%s_%s_idx on _%s(_%s)' % (cn, key, cn, key) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
746 self.sql(sql) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
747 |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
748 def drop_class_table_key_index(self, cn, key): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
749 table_name = '_%s' % cn |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
750 index_name = '_%s_%s_idx' % (cn, key) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
751 if self.sql_index_exists(table_name, index_name): |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
752 sql = 'drop index '+index_name |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
753 self.sql(sql) |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
754 |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
755 # and now the retired unique index too |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
756 index_name = '_%s_key_retired_idx' % cn |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
757 if self.sql_index_exists(table_name, index_name): |
|
3967
b1e81ad3fa6a
don't recompute the index name
Richard Jones <richard@users.sourceforge.net>
parents:
3963
diff
changeset
|
758 sql = 'drop index '+index_name |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
759 self.sql(sql) |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
760 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
761 def create_journal_table(self, spec): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
762 """ create the journal table for a class given the spec and |
| 1165 | 763 already-determined cols |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
764 """ |
| 1165 | 765 # journal table |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
766 cols = ','.join(['%s varchar' % x |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
767 for x in 'nodeid date tag action params'.split()]) |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
768 sql = """create table %s__journal ( |
|
3224
1fd11a9803bb
use backend datatype for journal timestamps.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3155
diff
changeset
|
769 nodeid integer, date %s, tag varchar(255), |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
770 action varchar(255), params text)""" % (spec.classname, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
771 self.hyperdb_to_sql_datatype(hyperdb.Date)) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
772 self.sql(sql) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
773 self.create_journal_table_indexes(spec) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
774 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
775 def create_journal_table_indexes(self, spec): |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
776 # index on nodeid |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
777 sql = 'create index %s_journ_idx on %s__journal(nodeid)' % ( |
|
1836
94e430ad4fdb
make the RDBMS common backend and the SQLite and MYsql backend create...
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1800
diff
changeset
|
778 spec.classname, spec.classname) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
779 self.sql(sql) |
| 1165 | 780 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
781 def drop_journal_table_indexes(self, classname): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
782 index_name = '%s_journ_idx' % classname |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
783 if not self.sql_index_exists('%s__journal' % classname, index_name): |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
784 return |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
785 index_sql = 'drop index '+index_name |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
786 self.sql(index_sql) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
787 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
788 def create_multilink_table(self, spec, ml): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
789 """ Create a multilink table for the "ml" property of the class |
| 1165 | 790 given by the spec |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
791 """ |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
792 # create the table |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
793 sql = 'create table %s_%s (linkid INTEGER, nodeid INTEGER)' % ( |
| 1165 | 794 spec.classname, ml) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
795 self.sql(sql) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
796 self.create_multilink_table_indexes(spec, ml) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
797 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
798 def create_multilink_table_indexes(self, spec, ml): |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
799 # create index on linkid |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
800 index_sql = 'create index %s_%s_l_idx on %s_%s(linkid)' % ( |
|
2100
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
801 spec.classname, ml, spec.classname, ml) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
802 self.sql(index_sql) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
803 |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
804 # create index on nodeid |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
805 index_sql = 'create index %s_%s_n_idx on %s_%s(nodeid)' % ( |
|
2100
62ed6505cbec
MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents:
2098
diff
changeset
|
806 spec.classname, ml, spec.classname, ml) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
807 self.sql(index_sql) |
| 1165 | 808 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
809 def drop_multilink_table_indexes(self, classname, ml): |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
810 l = [ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
811 '%s_%s_l_idx' % (classname, ml), |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
812 '%s_%s_n_idx' % (classname, ml) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
813 ] |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
814 table_name = '%s_%s' % (classname, ml) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
815 for index_name in l: |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
816 if not self.sql_index_exists(table_name, index_name): |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
817 continue |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
818 index_sql = 'drop index %s' % index_name |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
819 self.sql(index_sql) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
820 |
| 1165 | 821 def create_class(self, spec): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
822 """ Create a database table according to the given spec. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
823 """ |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
824 |
|
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
825 if not self.config.RDBMS_ALLOW_CREATE: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
826 raise DatabaseError(_('CREATE operation disallowed: "%s".' % |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
827 spec.classname)) |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
828 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
829 cols, mls = self.create_class_table(spec) |
|
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
830 self.create_journal_table(spec) |
| 1165 | 831 |
| 832 # now create the multilink tables | |
| 833 for ml in mls: | |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
834 self.create_multilink_table(spec, ml) |
| 1165 | 835 |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
836 def drop_class(self, cn, spec): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
837 """ Drop the given table from the database. |
| 1165 | 838 |
| 839 Drop the journal and multilink tables too. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
840 """ |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
841 |
|
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
842 if not self.config.RDBMS_ALLOW_DROP: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
843 raise DatabaseError(_('DROP operation disallowed: "%s".' % cn)) |
|
4474
9b4cf6c96ee2
Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents:
4473
diff
changeset
|
844 |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
845 properties = spec[1] |
| 1165 | 846 # figure the multilinks |
| 847 mls = [] | |
|
3924
21d3d7eeea8c
assorted pyflakes fixes
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3906
diff
changeset
|
848 for propname, prop in properties: |
| 1165 | 849 if isinstance(prop, Multilink): |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
850 mls.append(propname) |
| 1165 | 851 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
852 # drop class table and indexes |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
853 self.drop_class_table_indexes(cn, spec[0]) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
854 |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
855 self.drop_class_table(cn) |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
1840
diff
changeset
|
856 |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
857 # drop journal table and indexes |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
858 self.drop_journal_table_indexes(cn) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
859 sql = 'drop table %s__journal' % cn |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
860 self.sql(sql) |
| 1165 | 861 |
| 862 for ml in mls: | |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
863 # drop multilink table and indexes |
|
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
864 self.drop_multilink_table_indexes(cn, ml) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
865 sql = 'drop table %s_%s' % (spec.classname, ml) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
866 self.sql(sql) |
| 1165 | 867 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
868 def drop_class_table(self, cn): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
869 sql = 'drop table _%s' % cn |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
870 self.sql(sql) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
871 |
| 1165 | 872 # |
| 873 # Classes | |
| 874 # | |
| 875 def __getattr__(self, classname): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
876 """ A convenient way of calling self.getclass(classname). |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
877 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
878 if classname in self.classes: |
| 1165 | 879 return self.classes[classname] |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
880 raise AttributeError(classname) |
| 1165 | 881 |
| 882 def addclass(self, cl): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
883 """ Add a Class to the hyperdatabase. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
884 """ |
| 1165 | 885 cn = cl.classname |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
886 if cn in self.classes: |
|
6238
6834bb5473da
Summary: Constrain format of classname and document
John Rouillard <rouilj@ieee.org>
parents:
6197
diff
changeset
|
887 raise ValueError(_('Class "%s" already defined.'%cn)) |
| 1165 | 888 self.classes[cn] = cl |
| 889 | |
|
2076
2a4309450202
security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents:
2075
diff
changeset
|
890 # add default Edit and View permissions |
|
2649
1df7d4a41da4
Buncha stuff (sorry about the large checkin):
Richard Jones <richard@users.sourceforge.net>
parents:
2640
diff
changeset
|
891 self.security.addPermission(name="Create", klass=cn, |
|
1df7d4a41da4
Buncha stuff (sorry about the large checkin):
Richard Jones <richard@users.sourceforge.net>
parents:
2640
diff
changeset
|
892 description="User is allowed to create "+cn) |
|
2076
2a4309450202
security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents:
2075
diff
changeset
|
893 self.security.addPermission(name="Edit", klass=cn, |
|
2a4309450202
security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents:
2075
diff
changeset
|
894 description="User is allowed to edit "+cn) |
|
2a4309450202
security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents:
2075
diff
changeset
|
895 self.security.addPermission(name="View", klass=cn, |
|
2a4309450202
security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents:
2075
diff
changeset
|
896 description="User is allowed to access "+cn) |
|
4517
f8e85cf5f0fe
how odd, the Retire permission wasn't being registered;
Richard Jones <richard@users.sourceforge.net>
parents:
4490
diff
changeset
|
897 self.security.addPermission(name="Retire", klass=cn, |
|
f8e85cf5f0fe
how odd, the Retire permission wasn't being registered;
Richard Jones <richard@users.sourceforge.net>
parents:
4490
diff
changeset
|
898 description="User is allowed to retire "+cn) |
|
2076
2a4309450202
security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents:
2075
diff
changeset
|
899 |
| 1165 | 900 def getclasses(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
901 """ Return a list of the names of all existing classes. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
902 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
903 return sorted(self.classes) |
| 1165 | 904 |
| 905 def getclass(self, classname): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
906 """Get the Class object representing a particular class. |
| 1165 | 907 |
| 908 If 'classname' is not a valid class name, a KeyError is raised. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
909 """ |
| 1165 | 910 try: |
| 911 return self.classes[classname] | |
| 912 except KeyError: | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
913 raise KeyError('There is no class called "%s"' % classname) |
| 1165 | 914 |
| 915 def clear(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
916 """Delete all database contents. |
| 1165 | 917 |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
918 Note: I don't commit here, which is different behaviour to the |
|
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
919 "nuke from orbit" behaviour in the dbs. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
920 """ |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
921 logging.getLogger('roundup.hyperdb.backend').info('clear') |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
922 for cn in self.classes: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
923 sql = 'delete from _%s' % cn |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
924 self.sql(sql) |
| 1165 | 925 |
| 926 # | |
| 927 # Nodes | |
| 928 # | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
929 |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
930 hyperdb_to_sql_value = { |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
931 hyperdb.String : str, |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
932 # fractional seconds by default |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
933 hyperdb.Date : lambda x: x.formal(sep=' ', sec='%06.3f'), |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
934 hyperdb.Link : int, |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
935 hyperdb.Interval : str, |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
936 hyperdb.Password : str, |
|
2166
cd42c3c7173a
MySQL and Postgresql use BOOL/BOOLEAN for Boolean types
Richard Jones <richard@users.sourceforge.net>
parents:
2102
diff
changeset
|
937 hyperdb.Boolean : lambda x: x and 'TRUE' or 'FALSE', |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
938 hyperdb.Number : lambda x: x, |
|
5067
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
939 hyperdb.Integer : lambda x: x, |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
940 hyperdb.Multilink : lambda x: x, # used in journal marshalling |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
941 } |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
942 |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
943 def to_sql_value(self, propklass): |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
944 |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
945 fn = self.hyperdb_to_sql_value.get(propklass) |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
946 if fn: |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
947 return fn |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
948 |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
949 for k, v in self.hyperdb_to_sql_value.items(): |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
950 if issubclass(propklass, k): |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
951 return v |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
952 |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
953 raise ValueError('%r is not a hyperdb property class' % propklass) |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
954 |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
955 def _cache_del(self, key): |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
956 del self.cache[key] |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
957 self.cache_lru.remove(key) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
958 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
959 def _cache_refresh(self, key): |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
960 self.cache_lru.remove(key) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
961 self.cache_lru.insert(0, key) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
962 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
963 def _cache_save(self, key, node): |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
964 self.cache[key] = node |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
965 # update the LRU |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
966 self.cache_lru.insert(0, key) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
967 if len(self.cache_lru) > self.cache_size: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
968 del self.cache[self.cache_lru.pop()] |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
969 |
| 1165 | 970 def addnode(self, classname, nodeid, node): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
971 """ Add the specified node to its class's db. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
972 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
973 self.log_debug('addnode %s%s %r' % (classname, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
974 nodeid, node)) |
|
1528
96cd422532ef
bye bye gadfly - you served your purpose well [SF#701127]
Richard Jones <richard@users.sourceforge.net>
parents:
1523
diff
changeset
|
975 |
|
96cd422532ef
bye bye gadfly - you served your purpose well [SF#701127]
Richard Jones <richard@users.sourceforge.net>
parents:
1523
diff
changeset
|
976 # determine the column definitions and multilink tables |
| 1165 | 977 cl = self.classes[classname] |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
978 cols, mls = self.determine_columns(list(cl.properties.items())) |
| 1165 | 979 |
|
1189
23b8d1e87fe3
import fixes
Richard Jones <richard@users.sourceforge.net>
parents:
1187
diff
changeset
|
980 # we'll be supplied these props if we're doing an import |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
981 values = node.copy() |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
982 if 'creator' not in values: |
|
1189
23b8d1e87fe3
import fixes
Richard Jones <richard@users.sourceforge.net>
parents:
1187
diff
changeset
|
983 # add in the "calculated" properties (dupe so we don't affect |
|
23b8d1e87fe3
import fixes
Richard Jones <richard@users.sourceforge.net>
parents:
1187
diff
changeset
|
984 # calling code's node assumptions) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
985 values['creation'] = values['activity'] = date.Date() |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
986 values['actor'] = values['creator'] = self.getuid() |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
987 |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
988 cl = self.classes[classname] |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
989 props = cl.getprops(protected=1) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
990 del props['id'] |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
991 |
| 1165 | 992 # default the non-multilink columns |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
993 for col, prop in props.items(): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
994 if col not in values: |
|
1496
e6ac4e074acb
relaxed CVS importing (feature [SF#693277])
Richard Jones <richard@users.sourceforge.net>
parents:
1492
diff
changeset
|
995 if isinstance(prop, Multilink): |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
996 values[col] = [] |
|
1496
e6ac4e074acb
relaxed CVS importing (feature [SF#693277])
Richard Jones <richard@users.sourceforge.net>
parents:
1492
diff
changeset
|
997 else: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
998 values[col] = None |
| 1165 | 999 |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1000 # clear this node out of the cache if it's in there |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1001 key = (classname, nodeid) |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1002 if key in self.cache: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1003 self._cache_del(key) |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1004 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1005 # figure the values to insert |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1006 vals = [] |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1007 for col, _dt in cols: |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1008 # this is somewhat dodgy.... |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1009 if col.endswith('_int__'): |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1010 # XXX eugh, this test suxxors |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1011 value = values[col[2:-6]] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1012 # this is an Interval special "int" column |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1013 if value is not None: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1014 vals.append(value.as_seconds()) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1015 else: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1016 vals.append(value) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1017 continue |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1018 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1019 prop = props[col[1:]] |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1020 value = values[col[1:]] |
|
2472
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
1021 if value is not None: |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1022 value = self.to_sql_value(prop.__class__)(value) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1023 vals.append(value) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1024 vals.append(nodeid) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1025 vals = tuple(vals) |
| 1165 | 1026 |
| 1027 # make sure the ordering is correct for column name -> column value | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1028 s = ','.join([self.arg for x in cols]) + ',%s' % self.arg |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1029 cols = ','.join([col for col, dt in cols]) + ',id' |
| 1165 | 1030 |
| 1031 # perform the inserts | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1032 sql = 'insert into _%s (%s) values (%s)' % (classname, cols, s) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1033 self.sql(sql, vals) |
| 1165 | 1034 |
| 1035 # insert the multilink rows | |
| 1036 for col in mls: | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1037 t = '%s_%s' % (classname, col) |
| 1165 | 1038 for entry in node[col]: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1039 sql = 'insert into %s (linkid, nodeid) values (%s,%s)' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1040 t, self.arg, self.arg) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1041 self.sql(sql, (entry, nodeid)) |
| 1165 | 1042 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1043 def setnode(self, classname, nodeid, values, multilink_changes={}): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1044 """ Change the specified node. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1045 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1046 self.log_debug('setnode %s%s %r' % (classname, nodeid, values)) |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1047 |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1048 # clear this node out of the cache if it's in there |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1049 key = (classname, nodeid) |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1050 if key in self.cache: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1051 self._cache_del(key) |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1052 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1053 cl = self.classes[classname] |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1054 props = cl.getprops() |
| 1165 | 1055 |
| 1056 cols = [] | |
| 1057 mls = [] | |
| 1058 # add the multilinks separately | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1059 for col in values: |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1060 prop = props[col] |
| 1165 | 1061 if isinstance(prop, Multilink): |
| 1062 mls.append(col) | |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1063 elif isinstance(prop, Interval): |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1064 # Intervals store the seconds value too |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1065 cols.append(col) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1066 # extra leading '_' added by code below |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1067 cols.append('_' + col + '_int__') |
| 1165 | 1068 else: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1069 cols.append(col) |
| 1165 | 1070 cols.sort() |
| 1071 | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1072 # figure the values to insert |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1073 vals = [] |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1074 for col in cols: |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1075 if col.endswith('_int__'): |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1076 # XXX eugh, this test suxxors |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1077 # Intervals store the seconds value too |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1078 col = col[1:-6] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1079 prop = props[col] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1080 value = values[col] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1081 if value is None: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1082 vals.append(None) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1083 else: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1084 vals.append(value.as_seconds()) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1085 else: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1086 prop = props[col] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1087 value = values[col] |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1088 if value is None: |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1089 e = None |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1090 else: |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1091 e = self.to_sql_value(prop.__class__)(value) |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1092 vals.append(e) |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1093 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1094 vals.append(int(nodeid)) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1095 vals = tuple(vals) |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1096 |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1097 # if there's any updates to regular columns, do them |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1098 if cols: |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1099 # make sure the ordering is correct for column name -> column value |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1100 s = ','.join(['_%s=%s' % (x, self.arg) for x in cols]) |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1101 cols = ','.join(cols) |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1102 |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1103 # perform the update |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1104 sql = 'update _%s set %s where id=%s' % (classname, s, self.arg) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1105 self.sql(sql, vals) |
| 1165 | 1106 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1107 # we're probably coming from an import, not a change |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1108 if not multilink_changes: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1109 for name in mls: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1110 prop = props[name] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1111 value = values[name] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1112 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1113 t = '%s_%s' % (classname, name) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1114 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1115 # clear out previous values for this node |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1116 # XXX numeric ids |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1117 self.sql('delete from %s where nodeid=%s' % (t, self.arg), |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1118 (nodeid,)) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1119 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1120 # insert the values for this node |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1121 for entry in values[name]: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1122 sql = 'insert into %s (linkid, nodeid) values (%s,%s)' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1123 t, self.arg, self.arg) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1124 # XXX numeric ids |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1125 self.sql(sql, (entry, nodeid)) |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1126 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1127 # we have multilink changes to apply |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1128 for col, (add, remove) in multilink_changes.items(): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1129 tn = '%s_%s' % (classname, col) |
| 1165 | 1130 if add: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1131 sql = 'insert into %s (nodeid, linkid) values (%s,%s)' % (tn, |
| 1165 | 1132 self.arg, self.arg) |
| 1133 for addid in add: | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1134 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1135 self.sql(sql, (int(nodeid), int(addid))) |
| 1165 | 1136 if remove: |
|
4039
f44ec45cedb4
Fuse multiple DELETE calls into one for multilinks.
Stefan Seefeld <stefan@seefeld.name>
parents:
4038
diff
changeset
|
1137 s = ','.join([self.arg]*len(remove)) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1138 sql = 'delete from %s where nodeid=%s and linkid in (%s)' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1139 tn, self.arg, s) |
|
4039
f44ec45cedb4
Fuse multiple DELETE calls into one for multilinks.
Stefan Seefeld <stefan@seefeld.name>
parents:
4038
diff
changeset
|
1140 # XXX numeric ids |
|
f44ec45cedb4
Fuse multiple DELETE calls into one for multilinks.
Stefan Seefeld <stefan@seefeld.name>
parents:
4038
diff
changeset
|
1141 self.sql(sql, [int(nodeid)] + remove) |
| 1165 | 1142 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1143 sql_to_hyperdb_value = { |
|
6157
3c52093d7490
Fix string conversion from database
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6155
diff
changeset
|
1144 hyperdb.String : us2s, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1145 hyperdb.Date : date_to_hyperdb_value, |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1146 # hyperdb.Link : int, # XXX numeric ids |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1147 hyperdb.Link : str, |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1148 hyperdb.Interval : date.Interval, |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1149 hyperdb.Password : lambda x: password.Password(encrypted=x), |
|
2472
f41539b3c486
fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents:
2456
diff
changeset
|
1150 hyperdb.Boolean : _bool_cvt, |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1151 hyperdb.Number : _num_cvt, |
|
5067
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1152 hyperdb.Integer : int, |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1153 hyperdb.Multilink : lambda x: x, # used in journal marshalling |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1154 } |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1155 |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1156 def to_hyperdb_value(self, propklass): |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1157 |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1158 fn = self.sql_to_hyperdb_value.get(propklass) |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1159 if fn: |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1160 return fn |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1161 |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1162 for k, v in self.sql_to_hyperdb_value.items(): |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1163 if issubclass(propklass, k): |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1164 return v |
|
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1165 |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1166 raise ValueError('%r is not a hyperdb property class' % propklass) |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1167 |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1168 def _materialize_multilink(self, classname, nodeid, node, propname): |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1169 """ evaluation of single Multilink (lazy eval may have skipped this) |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1170 """ |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1171 if propname not in node: |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1172 prop = self.getclass(classname).properties[propname] |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1173 tn = prop.table_name |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1174 lid = prop.linkid_name |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1175 nid = prop.nodeid_name |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1176 w = '' |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1177 joi = '' |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1178 if prop.computed: |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1179 if isinstance(prop.rev_property, Link): |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1180 w = ' and %s.__retired__=0'%tn |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1181 else: |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1182 tn2 = '_' + prop.classname |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1183 joi = ', %s' % tn2 |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1184 w = ' and %s.%s=%s.id and %s.__retired__=0'%(tn, lid, |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1185 tn2, tn2) |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1186 cursor = self.sql_new_cursor(name='_materialize_multilink') |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1187 sql = 'select %s from %s%s where %s=%s%s' %(lid, tn, joi, nid, |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
1188 self.arg, w) |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1189 self.sql(sql, (nodeid,), cursor) |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1190 # Reduce this to only the first row (the ID), this can save a |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1191 # lot of space for large query results (not using fetchall) |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1192 node[propname] = [str(x) for x in sorted(int(r[0]) for r in cursor)] |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1193 cursor.close() |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1194 |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1195 def _materialize_multilinks(self, classname, nodeid, node, props=None): |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1196 """ get all Multilinks of a node (lazy eval may have skipped this) |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1197 """ |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1198 cl = self.classes[classname] |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1199 props = props or [pn for (pn, p) in cl.properties.items() |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1200 if isinstance(p, Multilink)] |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1201 for propname in props: |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1202 if propname not in node: |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1203 self._materialize_multilink(classname, nodeid, node, propname) |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1204 |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1205 def getnode(self, classname, nodeid, fetch_multilinks=True): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1206 """ Get a node from the database. |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1207 For optimisation optionally we don't fetch multilinks |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1208 (lazy Multilinks). |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1209 But for internal database operations we need them. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1210 """ |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1211 # see if we have this node cached |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1212 key = (classname, nodeid) |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1213 if key in self.cache: |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1214 # push us back to the top of the LRU |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1215 self._cache_refresh(key) |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1216 if __debug__: |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1217 self.stats['cache_hits'] += 1 |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1218 # return the cached information |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1219 if fetch_multilinks: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1220 self._materialize_multilinks(classname, nodeid, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1221 self.cache[key]) |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1222 return self.cache[key] |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1223 |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1224 if __debug__: |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1225 self.stats['cache_misses'] += 1 |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1226 start_t = time.time() |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1227 |
| 1165 | 1228 # figure the columns we're fetching |
| 1229 cl = self.classes[classname] | |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1230 cols, mls = self.determine_columns(list(cl.properties.items())) |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1231 scols = ','.join([col for col, dt in cols]) |
| 1165 | 1232 |
| 1233 # perform the basic property fetch | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1234 sql = 'select %s from _%s where id=%s' % (scols, classname, self.arg) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1235 self.sql(sql, (nodeid,)) |
| 1165 | 1236 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1237 values = self.sql_fetchone() |
| 1165 | 1238 if values is None: |
|
6197
2492e2e17371
Change text from no such <class> node <node id> to drop "node"
John Rouillard <rouilj@ieee.org>
parents:
6179
diff
changeset
|
1239 raise IndexError('no such %s %s' % (classname, nodeid)) |
| 1165 | 1240 |
| 1241 # make up the node | |
| 1242 node = {} | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1243 props = cl.getprops(protected=1) |
| 1165 | 1244 for col in range(len(cols)): |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1245 name = cols[col][0][1:] |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1246 if name.endswith('_int__'): |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1247 # XXX eugh, this test suxxors |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1248 # ignore the special Interval-as-seconds column |
|
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
1249 continue |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1250 value = values[col] |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1251 if value is not None: |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1252 value = self.to_hyperdb_value(props[name].__class__)(value) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1253 node[name] = value |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1254 |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1255 if fetch_multilinks and mls: |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1256 self._materialize_multilinks(classname, nodeid, node, mls) |
|
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1257 |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1258 # save off in the cache |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1259 key = (classname, nodeid) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1260 self._cache_save(key, node) |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1261 |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1262 if __debug__: |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1263 self.stats['get_items'] += (time.time() - start_t) |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
1264 |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1265 return node |
| 1165 | 1266 |
| 1267 def destroynode(self, classname, nodeid): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1268 """Remove a node from the database. Called exclusively by the |
| 1165 | 1269 destroy() method on Class. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1270 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1271 message = 'destroynode %s%s' % (classname, nodeid) |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1272 logging.getLogger('roundup.hyperdb.backend').info(message) |
| 1165 | 1273 |
| 1274 # make sure the node exists | |
| 1275 if not self.hasnode(classname, nodeid): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1276 raise IndexError('%s has no node %s' % (classname, nodeid)) |
| 1165 | 1277 |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1278 # see if we have this node cached |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1279 if (classname, nodeid) in self.cache: |
|
1173
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1280 del self.cache[(classname, nodeid)] |
|
58f1a2c174ed
simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents:
1172
diff
changeset
|
1281 |
| 1165 | 1282 # see if there's any obvious commit actions that we should get rid of |
| 1283 for entry in self.transactions[:]: | |
| 1284 if entry[1][:2] == (classname, nodeid): | |
| 1285 self.transactions.remove(entry) | |
| 1286 | |
| 1287 # now do the SQL | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1288 sql = 'delete from _%s where id=%s' % (classname, self.arg) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1289 self.sql(sql, (nodeid,)) |
| 1165 | 1290 |
| 1291 # remove from multilnks | |
| 1292 cl = self.getclass(classname) | |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1293 x, mls = self.determine_columns(list(cl.properties.items())) |
| 1165 | 1294 for col in mls: |
| 1295 # get the link ids | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1296 sql = 'delete from %s_%s where nodeid=%s' % (classname, col, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1297 self.arg) |
|
1906
f255363e6d97
PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
1298 self.sql(sql, (nodeid,)) |
| 1165 | 1299 |
| 1300 # remove journal entries | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1301 sql = 'delete from %s__journal where nodeid=%s' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1302 classname, self.arg) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1303 self.sql(sql, (nodeid,)) |
| 1165 | 1304 |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
1305 # cleanup any blob filestorage when we commit |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1306 self.transactions.append((FileStorage.destroy, (self, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1307 classname, nodeid))) |
|
3906
666b70676ec6
destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3897
diff
changeset
|
1308 |
| 1165 | 1309 def hasnode(self, classname, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1310 """ Determine if the database has a given node. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1311 """ |
|
5086
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1312 # nodeid (aka link) is sql type INTEGER. max positive value |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1313 # for INTEGER is 2^31-1 for Postgres and MySQL. The max |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1314 # positive value for SqLite is 2^63 -1, so arguably this check |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1315 # needs to adapt for the type of the RDBMS. For right now, |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1316 # choose lowest common denominator. |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1317 if int(nodeid) >= 2**31: |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1318 # value out of range return false |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1319 return 0 |
|
2b551b6b0830
issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents:
5067
diff
changeset
|
1320 |
| 4031 | 1321 # If this node is in the cache, then we do not need to go to |
| 1322 # the database. (We don't consider this an LRU hit, though.) | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1323 if (classname, nodeid) in self.cache: |
| 4031 | 1324 # Return 1, not True, to match the type of the result of |
| 1325 # the SQL operation below. | |
| 1326 return 1 | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1327 sql = 'select count(*) from _%s where id=%s' % (classname, self.arg) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1328 self.sql(sql, (nodeid,)) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1329 return int(self.cursor.fetchone()[0]) |
| 1165 | 1330 |
| 1331 def countnodes(self, classname): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1332 """ Count the number of nodes that exist for a particular Class. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1333 """ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1334 sql = 'select count(*) from _%s' % classname |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1335 self.sql(sql) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1336 return self.cursor.fetchone()[0] |
| 1165 | 1337 |
| 1338 def addjournal(self, classname, nodeid, action, params, creator=None, | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1339 creation=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1340 """ Journal the Action |
| 1165 | 1341 'action' may be: |
| 1342 | |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1343 'set' -- 'params' is a dictionary of property values |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1344 'create' -- 'params' is an empty dictionary as of |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1345 Wed Nov 06 11:38:43 2002 +0000 |
| 1165 | 1346 'link' or 'unlink' -- 'params' is (classname, nodeid, propname) |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1347 'retired' or 'restored' -- 'params' is None |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1348 |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1349 'creator' -- the user performing the action, which defaults to |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1350 the current user. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1351 """ |
| 1165 | 1352 # handle supply of the special journalling parameters (usually |
| 1353 # supplied on importing an existing database) | |
| 1354 if creator: | |
| 1355 journaltag = creator | |
| 1356 else: | |
|
1800
a3b1b1dcf639
Use getuid(), not figure_curuserid()
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
1794
diff
changeset
|
1357 journaltag = self.getuid() |
| 1165 | 1358 if creation: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1359 journaldate = creation |
| 1165 | 1360 else: |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1361 journaldate = date.Date() |
| 1165 | 1362 |
| 1363 # create the journal entry | |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1364 cols = 'nodeid,date,tag,action,params' |
| 1165 | 1365 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1366 self.log_debug('addjournal %s%s %r %s %s %r' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1367 classname, nodeid, journaldate, journaltag, action, params)) |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1368 |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1369 # make the journalled data marshallable |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1370 if isinstance(params, type({})): |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1371 self._journal_marshal(params, classname) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1372 |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
1373 params = repr_export(params) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1374 |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1375 dc = self.to_sql_value(hyperdb.Date) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1376 journaldate = dc(journaldate) |
| 1165 | 1377 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1378 self.save_journal(classname, cols, nodeid, journaldate, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1379 journaltag, action, params) |
| 1165 | 1380 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1381 def setjournal(self, classname, nodeid, journal): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1382 """Set the journal to the "journal" list.""" |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1383 # clear out any existing entries |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1384 self.sql('delete from %s__journal where nodeid=%s' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1385 classname, self.arg), (nodeid,)) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1386 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1387 # create the journal entry |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1388 cols = 'nodeid,date,tag,action,params' |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1389 |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1390 dc = self.to_sql_value(hyperdb.Date) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1391 for nodeid, journaldate, journaltag, action, params in journal: |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1392 self.log_debug('addjournal %s%s %r %s %s %r' % ( |
|
4059
ef0b4396888a
Enhance and simplify logging.
Stefan Seefeld <stefan@seefeld.name>
parents:
4044
diff
changeset
|
1393 classname, nodeid, journaldate, journaltag, action, |
|
ef0b4396888a
Enhance and simplify logging.
Stefan Seefeld <stefan@seefeld.name>
parents:
4044
diff
changeset
|
1394 params)) |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1395 |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1396 # make the journalled data marshallable |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1397 if isinstance(params, type({})): |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1398 self._journal_marshal(params, classname) |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
1399 params = repr_export(params) |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1400 |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1401 self.save_journal(classname, cols, nodeid, dc(journaldate), |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1402 journaltag, action, params) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
1403 |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1404 def _journal_marshal(self, params, classname): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1405 """Convert the journal params values into safely repr'able and |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1406 eval'able values.""" |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1407 properties = self.getclass(classname).getprops() |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1408 for param, value in params.items(): |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1409 if not value: |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1410 continue |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1411 property = properties[param] |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1412 cvt = self.to_sql_value(property.__class__) |
|
2275
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1413 if isinstance(property, Password): |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1414 params[param] = cvt(value) |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1415 elif isinstance(property, Date): |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1416 params[param] = cvt(value) |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1417 elif isinstance(property, Interval): |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1418 params[param] = cvt(value) |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1419 elif isinstance(property, Boolean): |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1420 params[param] = cvt(value) |
|
3197e37346de
merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents:
2256
diff
changeset
|
1421 |
| 1165 | 1422 def getjournal(self, classname, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1423 """ get the journal for id |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1424 """ |
| 1165 | 1425 # make sure the node exists |
| 1426 if not self.hasnode(classname, nodeid): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1427 raise IndexError('%s has no node %s' % (classname, nodeid)) |
| 1165 | 1428 |
| 1429 cols = ','.join('nodeid date tag action params'.split()) | |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1430 journal = self.load_journal(classname, cols, nodeid) |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1431 |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1432 # now unmarshal the data |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1433 dc = self.to_hyperdb_value(hyperdb.Date) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1434 res = [] |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1435 properties = self.getclass(classname).getprops() |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1436 for nodeid, date_stamp, user, action, params in journal: |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
1437 params = eval_import(params) |
|
2248
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1438 if isinstance(params, type({})): |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1439 for param, value in params.items(): |
|
2248
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1440 if not value: |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1441 continue |
|
2731
7c472ed6babf
handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents:
2727
diff
changeset
|
1442 property = properties.get(param, None) |
|
7c472ed6babf
handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents:
2727
diff
changeset
|
1443 if property is None: |
|
7c472ed6babf
handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents:
2727
diff
changeset
|
1444 # deleted property |
|
7c472ed6babf
handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents:
2727
diff
changeset
|
1445 continue |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1446 cvt = self.to_hyperdb_value(property.__class__) |
|
2248
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1447 if isinstance(property, Password): |
|
4483
22bc0426e348
Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4480
diff
changeset
|
1448 params[param] = password.JournalPassword(value) |
|
2248
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1449 elif isinstance(property, Date): |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1450 params[param] = cvt(value) |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1451 elif isinstance(property, Interval): |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1452 params[param] = cvt(value) |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1453 elif isinstance(property, Boolean): |
|
cd7e6d6288c6
fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents:
2244
diff
changeset
|
1454 params[param] = cvt(value) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1455 # XXX numeric ids |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1456 res.append((str(nodeid), dc(date_stamp), user, action, params)) |
|
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1457 return res |
| 1165 | 1458 |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1459 def save_journal(self, classname, cols, nodeid, journaldate, |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1460 journaltag, action, params): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1461 """ Save the journal entry to the database |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1462 """ |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1463 entry = (nodeid, journaldate, journaltag, action, params) |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1464 |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1465 # do the insert |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1466 a = self.arg |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1467 sql = 'insert into %s__journal (%s) values (%s,%s,%s,%s,%s)' % ( |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1468 classname, cols, a, a, a, a, a) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1469 self.sql(sql, entry) |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1470 |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
1471 def load_journal(self, classname, cols, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1472 """ Load the journal from the database |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1473 """ |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1474 # now get the journal entries |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1475 sql = 'select %s from %s__journal where nodeid=%s order by date' % ( |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1476 cols, classname, self.arg) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1477 self.sql(sql, (nodeid,)) |
|
2244
ac4f295499a4
fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents:
2240
diff
changeset
|
1478 return self.cursor.fetchall() |
| 1165 | 1479 |
| 1480 def pack(self, pack_before): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1481 """ Delete all journal entries except "create" before 'pack_before'. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1482 """ |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
1483 date_stamp = self.to_sql_value(Date)(pack_before) |
| 1165 | 1484 |
| 1485 # do the delete | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1486 for classname in self.classes: |
| 1165 | 1487 sql = "delete from %s__journal where date<%s and "\ |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1488 "action<>'create'" % (classname, self.arg) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
1489 self.sql(sql, (date_stamp,)) |
| 1165 | 1490 |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1491 def sql_commit(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1492 """ Actually commit to the database. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1493 """ |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1494 logging.getLogger('roundup.hyperdb.backend').info('commit') |
|
3687
ff9f4ca42454
Postgres backend allows transaction collisions to be ignored when...
Richard Jones <richard@users.sourceforge.net>
parents:
3685
diff
changeset
|
1495 |
| 1165 | 1496 self.conn.commit() |
| 1497 | |
|
2374
31cb1014300c
Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents:
2362
diff
changeset
|
1498 # open a new cursor for subsequent work |
|
31cb1014300c
Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents:
2362
diff
changeset
|
1499 self.cursor = self.conn.cursor() |
|
31cb1014300c
Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents:
2362
diff
changeset
|
1500 |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1501 def sql_new_cursor(self, conn=None, *args, **kw): |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1502 """ Create new cursor, this may need additional parameters for |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1503 performance optimization for different backends. |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1504 """ |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1505 if conn is None: |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1506 conn = self.conn |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1507 return conn.cursor() |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
1508 |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1509 def commit(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1510 """ Commit the current transactions. |
| 1165 | 1511 |
| 1512 Save all data changed since the database was opened or since the | |
| 1513 last commit() or rollback(). | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1514 """ |
| 1165 | 1515 # commit the database |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1516 self.sql_commit() |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1517 |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1518 # session and otk are committed with the db but not the other |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1519 # way round |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1520 if self.Session: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1521 self.Session.commit() |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1522 if self.Otk: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1523 self.Otk.commit() |
| 1165 | 1524 |
| 1525 # now, do all the other transaction stuff | |
| 1526 for method, args in self.transactions: | |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1527 method(*args) |
| 1165 | 1528 |
|
3295
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3239
diff
changeset
|
1529 # save the indexer |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3239
diff
changeset
|
1530 self.indexer.save_index() |
|
a615cc230160
added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents:
3239
diff
changeset
|
1531 |
| 1165 | 1532 # clear out the transactions |
| 1533 self.transactions = [] | |
| 1534 | |
|
4448
2784c239e6c8
clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4430
diff
changeset
|
1535 # clear the cache: Don't carry over cached values from one |
|
2784c239e6c8
clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4430
diff
changeset
|
1536 # transaction to the next (there may be other changes from other |
|
2784c239e6c8
clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4430
diff
changeset
|
1537 # transactions) |
|
2784c239e6c8
clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4430
diff
changeset
|
1538 self.clearCache() |
|
2784c239e6c8
clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4430
diff
changeset
|
1539 |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1540 def sql_rollback(self): |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1541 self.conn.rollback() |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1542 |
| 1165 | 1543 def rollback(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1544 """ Reverse all actions from the current transaction. |
| 1165 | 1545 |
| 1546 Undo all the changes made since the database was opened or the last | |
| 1547 commit() or rollback() was performed. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1548 """ |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1549 logging.getLogger('roundup.hyperdb.backend').info('rollback') |
| 1165 | 1550 |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1551 self.sql_rollback() |
| 1165 | 1552 |
| 1553 # roll back "other" transaction stuff | |
| 1554 for method, args in self.transactions: | |
| 1555 # delete temporary files | |
| 1556 if method == self.doStoreFile: | |
| 1557 self.rollbackStoreFile(*args) | |
| 1558 self.transactions = [] | |
| 1559 | |
|
1492
2fc7d4a8c9e7
fixed sqlite rollback/caching bug [SF#689383]
Richard Jones <richard@users.sourceforge.net>
parents:
1484
diff
changeset
|
1560 # clear the cache |
|
2fc7d4a8c9e7
fixed sqlite rollback/caching bug [SF#689383]
Richard Jones <richard@users.sourceforge.net>
parents:
1484
diff
changeset
|
1561 self.clearCache() |
|
2fc7d4a8c9e7
fixed sqlite rollback/caching bug [SF#689383]
Richard Jones <richard@users.sourceforge.net>
parents:
1484
diff
changeset
|
1562 |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1563 def sql_close(self): |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
1564 logging.getLogger('roundup.hyperdb.backend').info('close') |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1565 self.conn.close() |
|
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1566 |
| 1165 | 1567 def close(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1568 """ Close off the connection. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1569 """ |
|
2093
3f6024ab2c7a
That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents:
2089
diff
changeset
|
1570 self.indexer.close() |
|
1911
f5c804379c85
fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents:
1906
diff
changeset
|
1571 self.sql_close() |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1572 if self.Session: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1573 self.Session.close() |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1574 self.Session = None |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1575 if self.Otk: |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1576 self.Otk.close() |
|
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5318
diff
changeset
|
1577 self.Otk = None |
| 1165 | 1578 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1579 |
| 1165 | 1580 # |
| 1581 # The base Class class | |
| 1582 # | |
| 1583 class Class(hyperdb.Class): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1584 """ The handle to a particular class of nodes in a hyperdatabase. |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1585 |
| 1165 | 1586 All methods except __repr__ and getnode must be implemented by a |
| 1587 concrete backend Class. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1588 """ |
|
4849
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
1589 # For many databases the LIKE operator ignores case. |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
1590 # Postgres and Oracle have an ILIKE operator to support this. |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
1591 # We define the default here, can be changed in derivative class |
|
e68920390aad
Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4803
diff
changeset
|
1592 case_insensitive_like = 'LIKE' |
| 1165 | 1593 |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
1594 # For some databases (mysql) the = operator for strings ignores case. |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
1595 # We define the default here, can be changed in derivative class |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
1596 case_sensitive_equal = '=' |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
1597 |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1598 # Some DBs order NULL values last. Set this variable in the backend |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1599 # for prepending an order by clause for each attribute that causes |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1600 # correct sort order for NULLs. Examples: |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1601 # order_by_null_values = '(%s is not NULL)' |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1602 # order_by_null_values = 'notnull(%s)' |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1603 # The format parameter is replaced with the attribute. |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1604 order_by_null_values = None |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1605 |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1606 # Assuming DBs can do subselects, overwrite if they cannot. |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1607 supports_subselects = True |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
1608 |
| 1165 | 1609 def schema(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1610 """ A dumpable version of the schema that we can store in the |
| 1165 | 1611 database |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1612 """ |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1613 return (self.key, |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1614 [(x, repr(y)) for x, y in self.properties.items() if not y.computed]) |
| 1165 | 1615 |
| 1616 def enableJournalling(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1617 """Turn journalling on for this class |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1618 """ |
| 1165 | 1619 self.do_journal = 1 |
| 1620 | |
| 1621 def disableJournalling(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1622 """Turn journalling off for this class |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1623 """ |
| 1165 | 1624 self.do_journal = 0 |
| 1625 | |
| 1626 # Editing nodes: | |
| 1627 def create(self, **propvalues): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1628 """ Create a new node of this class and return its id. |
| 1165 | 1629 |
| 1630 The keyword arguments in 'propvalues' map property names to values. | |
| 1631 | |
| 1632 The values of arguments must be acceptable for the types of their | |
| 1633 corresponding properties or a TypeError is raised. | |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1634 |
| 1165 | 1635 If this class has a key property, it must be present and its value |
| 1636 must not collide with other key strings or a ValueError is raised. | |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1637 |
| 1165 | 1638 Any other properties on this class that are missing from the |
| 1639 'propvalues' dictionary are set to None. | |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1640 |
| 1165 | 1641 If an id in a link or multilink property does not refer to a valid |
| 1642 node, an IndexError is raised. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1643 """ |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
1644 self.fireAuditors('create', None, propvalues) |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
1645 newid = self.create_inner(**propvalues) |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
1646 self.fireReactors('create', newid, None) |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
1647 return newid |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1648 |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
1649 def create_inner(self, **propvalues): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1650 """ Called by create, in-between the audit and react calls. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1651 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1652 if 'id' in propvalues: |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1653 raise KeyError('"id" is reserved') |
| 1165 | 1654 |
| 1655 if self.db.journaltag is None: | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1656 raise DatabaseError(_('Database open read-only')) |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1657 |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1658 if ('creator' in propvalues or 'actor' in propvalues or |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1659 'creation' in propvalues or 'activity' in propvalues): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1660 raise KeyError('"creator", "actor", "creation" and ' |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1661 '"activity" are reserved') |
| 1165 | 1662 |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1663 for p in propvalues: |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1664 prop = self.properties[p] |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1665 if prop.computed: |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1666 raise KeyError('"%s" is a computed property'%p) |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1667 |
| 1165 | 1668 # new node's id |
| 1669 newid = self.db.newid(self.classname) | |
| 1670 | |
| 1671 # validate propvalues | |
| 5809 | 1672 num_re = re.compile(r'^\d+$') |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1673 for key, value in propvalues.items(): |
| 1165 | 1674 if key == self.key: |
| 1675 try: | |
| 1676 self.lookup(value) | |
| 1677 except KeyError: | |
| 1678 pass | |
| 1679 else: | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1680 raise ValueError('node with key "%s" exists' % value) |
| 1165 | 1681 |
| 1682 # try to handle this property | |
| 1683 try: | |
| 1684 prop = self.properties[key] | |
| 1685 except KeyError: | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1686 raise KeyError('"%s" has no property "%s"' % (self.classname, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1687 key)) |
| 1165 | 1688 |
| 1689 if value is not None and isinstance(prop, Link): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1690 if not isinstance(value, type('')): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1691 raise ValueError('link value must be String') |
| 1165 | 1692 link_class = self.properties[key].classname |
| 1693 # if it isn't a number, it's a key | |
| 1694 if not num_re.match(value): | |
| 1695 try: | |
| 1696 value = self.db.classes[link_class].lookup(value) | |
| 1697 except (TypeError, KeyError): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1698 raise IndexError('new property "%s": %s not a %s' % ( |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1699 key, value, link_class)) |
| 1165 | 1700 elif not self.db.getclass(link_class).hasnode(value): |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1701 raise IndexError('%s has no node %s' % (link_class, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1702 value)) |
| 1165 | 1703 |
| 1704 # save off the value | |
| 1705 propvalues[key] = value | |
| 1706 | |
| 1707 # register the link with the newly linked node | |
| 1708 if self.do_journal and self.properties[key].do_journal: | |
| 1709 self.db.addjournal(link_class, value, 'link', | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1710 (self.classname, newid, key)) |
| 1165 | 1711 |
| 1712 elif isinstance(prop, Multilink): | |
|
3872
34128a809e22
Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3869
diff
changeset
|
1713 if value is None: |
|
34128a809e22
Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3869
diff
changeset
|
1714 value = [] |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1715 if not hasattr(value, '__iter__') or \ |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1716 isinstance(value, type('')): |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1717 raise TypeError('new property "%s" not an iterable of ids' |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1718 % key) |
| 1165 | 1719 # clean up and validate the list of links |
| 1720 link_class = self.properties[key].classname | |
| 1721 l = [] | |
| 1722 for entry in value: | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1723 if not isinstance(entry, type('')): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1724 raise ValueError('"%s" multilink value (%r) ' |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1725 'must contain Strings' % ( |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1726 key, value)) |
| 1165 | 1727 # if it isn't a number, it's a key |
| 1728 if not num_re.match(entry): | |
| 1729 try: | |
| 1730 entry = self.db.classes[link_class].lookup(entry) | |
| 1731 except (TypeError, KeyError): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1732 raise IndexError('new property "%s": %s not a %s' % ( |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1733 key, entry, self.properties[key].classname)) |
| 1165 | 1734 l.append(entry) |
| 1735 value = l | |
| 1736 propvalues[key] = value | |
| 1737 | |
| 1738 # handle additions | |
| 1739 for nodeid in value: | |
| 1740 if not self.db.getclass(link_class).hasnode(nodeid): | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1741 raise IndexError('%s has no node %s' % (link_class, |
|
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1742 nodeid)) |
| 1165 | 1743 # register the link with the newly linked node |
| 1744 if self.do_journal and self.properties[key].do_journal: | |
| 1745 self.db.addjournal(link_class, nodeid, 'link', | |
|
6065
1ec4aa670b0c
Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents:
6002
diff
changeset
|
1746 (self.classname, newid, key)) |
| 1165 | 1747 |
| 1748 elif isinstance(prop, String): | |
|
1383
f19dde90e473
applied unicode patch
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1365
diff
changeset
|
1749 if type(value) != type('') and type(value) != type(u''): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1750 raise TypeError('new property "%s" not a string'%key) |
|
2892
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
1751 if prop.indexme: |
|
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
1752 self.db.indexer.add_text((self.classname, newid, key), |
|
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
1753 value) |
| 1165 | 1754 |
| 1755 elif isinstance(prop, Password): | |
| 1756 if not isinstance(value, password.Password): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1757 raise TypeError('new property "%s" not a Password'%key) |
| 1165 | 1758 |
| 1759 elif isinstance(prop, Date): | |
| 1760 if value is not None and not isinstance(value, date.Date): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1761 raise TypeError('new property "%s" not a Date'%key) |
| 1165 | 1762 |
| 1763 elif isinstance(prop, Interval): | |
| 1764 if value is not None and not isinstance(value, date.Interval): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1765 raise TypeError('new property "%s" not an Interval'%key) |
| 1165 | 1766 |
| 1767 elif value is not None and isinstance(prop, Number): | |
| 1768 try: | |
| 1769 float(value) | |
| 1770 except ValueError: | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1771 raise TypeError('new property "%s" not numeric'%key) |
| 1165 | 1772 |
|
5067
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1773 elif value is not None and isinstance(prop, Integer): |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1774 try: |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1775 int(value) |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1776 except ValueError: |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1777 raise TypeError('new property "%s" not integer'%key) |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
1778 |
| 1165 | 1779 elif value is not None and isinstance(prop, Boolean): |
| 1780 try: | |
| 1781 int(value) | |
| 1782 except ValueError: | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1783 raise TypeError('new property "%s" not boolean'%key) |
| 1165 | 1784 |
| 1785 # make sure there's data where there needs to be | |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
1786 for key, prop in self.properties.items(): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1787 if key in propvalues: |
| 1165 | 1788 continue |
| 1789 if key == self.key: | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1790 raise ValueError('key property "%s" is required'%key) |
| 1165 | 1791 if isinstance(prop, Multilink): |
| 1792 propvalues[key] = [] | |
| 1793 else: | |
| 1794 propvalues[key] = None | |
| 1795 | |
| 1796 # done | |
| 1797 self.db.addnode(self.classname, newid, propvalues) | |
| 1798 if self.do_journal: | |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
1799 self.db.addjournal(self.classname, newid, ''"create", {}) |
| 1165 | 1800 |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1801 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
1802 return str(newid) |
| 1165 | 1803 |
| 1804 def get(self, nodeid, propname, default=_marker, cache=1): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1805 """Get the value of a property on an existing node of this class. |
| 1165 | 1806 |
| 1807 'nodeid' must be the id of an existing node of this class or an | |
| 1808 IndexError is raised. 'propname' must be the name of a property | |
| 1809 of this class or a KeyError is raised. | |
| 1810 | |
|
1780
d2801a2b0a77
Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents:
1751
diff
changeset
|
1811 'cache' exists for backwards compatibility, and is not used. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1812 """ |
| 1165 | 1813 if propname == 'id': |
| 1814 return nodeid | |
| 1815 | |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1816 # get the node's dict |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1817 d = self.db.getnode(self.classname, nodeid, fetch_multilinks=False) |
|
4476
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1818 # handle common case -- that property is in dict -- first |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1819 # if None and one of creator/creation actor/activity return None |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1820 if propname in d: |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1821 r = d [propname] |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1822 # return copy of our list |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1823 if isinstance (r, list): |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1824 return r[:] |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1825 if r is not None: |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1826 return r |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1827 elif propname in ('creation', 'activity', 'creator', 'actor'): |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1828 return r |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1829 |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1830 # propname not in d: |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1831 if propname == 'creation' or propname == 'activity': |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1832 return date.Date() |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1833 if propname == 'creator' or propname == 'actor': |
|
143f52d48e60
- small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4474
diff
changeset
|
1834 return self.db.getuid() |
| 1165 | 1835 |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1836 # get the property (raises KeyError if invalid) |
| 1165 | 1837 prop = self.properties[propname] |
| 1838 | |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1839 # lazy evaluation of Multilink |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1840 if propname not in d and isinstance(prop, Multilink): |
|
4490
559d9a2a0191
Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4483
diff
changeset
|
1841 self.db._materialize_multilink(self.classname, nodeid, d, propname) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
1842 |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1843 # handle there being no value in the table for the property |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1844 if propname not in d or d[propname] is None: |
|
2692
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
1845 if default is _marker: |
| 1165 | 1846 if isinstance(prop, Multilink): |
| 1847 return [] | |
| 1848 else: | |
| 1849 return None | |
| 1850 else: | |
| 1851 return default | |
| 1852 | |
| 1853 # don't pass our list to other code | |
| 1854 if isinstance(prop, Multilink): | |
| 1855 return d[propname][:] | |
| 1856 | |
| 1857 return d[propname] | |
| 1858 | |
| 1859 def set(self, nodeid, **propvalues): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1860 """Modify a property on an existing node of this class. |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1861 |
| 1165 | 1862 'nodeid' must be the id of an existing node of this class or an |
| 1863 IndexError is raised. | |
| 1864 | |
| 1865 Each key in 'propvalues' must be the name of a property of this | |
| 1866 class or a KeyError is raised. | |
| 1867 | |
| 1868 All values in 'propvalues' must be acceptable types for their | |
| 1869 corresponding properties or a TypeError is raised. | |
| 1870 | |
| 1871 If the value of the key property is set, it must not collide with | |
| 1872 other key strings or a ValueError is raised. | |
| 1873 | |
| 1874 If the value of a Link or Multilink property contains an invalid | |
| 1875 node id, a ValueError is raised. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1876 """ |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1877 self.fireAuditors('set', nodeid, propvalues) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1878 oldvalues = copy.deepcopy(self.db.getnode(self.classname, nodeid)) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1879 propvalues = self.set_inner(nodeid, **propvalues) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1880 self.fireReactors('set', nodeid, oldvalues) |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
1881 return propvalues |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1882 |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
1883 def set_inner(self, nodeid, **propvalues): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1884 """ Called by set, in-between the audit and react calls. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
1885 """ |
| 1165 | 1886 if not propvalues: |
| 1887 return propvalues | |
| 1888 | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1889 if ('creator' in propvalues or 'actor' in propvalues or |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1890 'creation' in propvalues or 'activity' in propvalues): |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1891 raise KeyError('"creator", "actor", "creation" and ' |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1892 '"activity" are reserved') |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1893 |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1894 if 'id' in propvalues: |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1895 raise KeyError('"id" is reserved') |
| 1165 | 1896 |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1897 for p in propvalues: |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1898 prop = self.properties[p] |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1899 if prop.computed: |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1900 raise KeyError('"%s" is a computed property'%p) |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
1901 |
| 1165 | 1902 if self.db.journaltag is None: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1903 raise DatabaseError(_('Database open read-only')) |
| 1165 | 1904 |
| 1905 node = self.db.getnode(self.classname, nodeid) | |
| 1906 if self.is_retired(nodeid): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1907 raise IndexError('Requested item is retired') |
| 5809 | 1908 num_re = re.compile(r'^\d+$') |
| 1165 | 1909 |
|
3082
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
1910 # make a copy of the values dictionary - we'll modify the contents |
|
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
1911 propvalues = propvalues.copy() |
|
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
1912 |
| 1165 | 1913 # if the journal value is to be different, store it in here |
| 1914 journalvalues = {} | |
| 1915 | |
| 1916 # remember the add/remove stuff for multilinks, making it easier | |
| 1917 # for the Database layer to do its stuff | |
| 1918 multilink_changes = {} | |
| 1919 | |
|
5112
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
1920 # omit quiet properties from history/changelog |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
1921 quiet_props = [] |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
1922 |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1923 for propname, value in list(propvalues.items()): |
| 1165 | 1924 # check to make sure we're not duplicating an existing key |
| 1925 if propname == self.key and node[propname] != value: | |
| 1926 try: | |
| 1927 self.lookup(value) | |
| 1928 except KeyError: | |
| 1929 pass | |
| 1930 else: | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1931 raise ValueError('node with key "%s" exists'%value) |
| 1165 | 1932 |
| 1933 # this will raise the KeyError if the property isn't valid | |
| 1934 # ... we don't use getprops() here because we only care about | |
| 1935 # the writeable properties. | |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1936 try: |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1937 prop = self.properties[propname] |
|
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
1938 except KeyError: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1939 raise KeyError('"%s" has no property named "%s"'%( |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1940 self.classname, propname)) |
| 1165 | 1941 |
| 1942 # if the value's the same as the existing value, no sense in | |
| 1943 # doing anything | |
|
1304
61ad556cfc8d
working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents:
1295
diff
changeset
|
1944 current = node.get(propname, None) |
|
61ad556cfc8d
working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents:
1295
diff
changeset
|
1945 if value == current: |
| 1165 | 1946 del propvalues[propname] |
| 1947 continue | |
|
1304
61ad556cfc8d
working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents:
1295
diff
changeset
|
1948 journalvalues[propname] = current |
| 1165 | 1949 |
| 1950 # do stuff based on the prop type | |
| 1951 if isinstance(prop, Link): | |
| 1952 link_class = prop.classname | |
| 1953 # if it isn't a number, it's a key | |
| 1954 if value is not None and not isinstance(value, type('')): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1955 raise ValueError('property "%s" link value be a string'%( |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1956 propname)) |
| 1165 | 1957 if isinstance(value, type('')) and not num_re.match(value): |
| 1958 try: | |
| 1959 value = self.db.classes[link_class].lookup(value) | |
| 1960 except (TypeError, KeyError): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1961 raise IndexError('new property "%s": %s not a %s'%( |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1962 propname, value, prop.classname)) |
| 1165 | 1963 |
| 1964 if (value is not None and | |
| 1965 not self.db.getclass(link_class).hasnode(value)): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1966 raise IndexError('%s has no node %s'%(link_class, |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1967 value)) |
| 1165 | 1968 |
| 1969 if self.do_journal and prop.do_journal: | |
| 1970 # register the unlink with the old linked node | |
| 1971 if node[propname] is not None: | |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
1972 self.db.addjournal(link_class, node[propname], |
|
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
1973 ''"unlink", (self.classname, nodeid, propname)) |
| 1165 | 1974 |
| 1975 # register the link with the newly linked node | |
| 1976 if value is not None: | |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
1977 self.db.addjournal(link_class, value, ''"link", |
| 1165 | 1978 (self.classname, nodeid, propname)) |
| 1979 | |
| 1980 elif isinstance(prop, Multilink): | |
|
3872
34128a809e22
Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3869
diff
changeset
|
1981 if value is None: |
|
34128a809e22
Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents:
3869
diff
changeset
|
1982 value = [] |
|
5467
630a00b20394
strings have an __iter__ method on Python 3, but we don't want to consider them iterable
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5416
diff
changeset
|
1983 if not hasattr(value, '__iter__') or type(value) == type(''): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1984 raise TypeError('new property "%s" not an iterable of' |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1985 ' ids'%propname) |
| 1165 | 1986 link_class = self.properties[propname].classname |
| 1987 l = [] | |
| 1988 for entry in value: | |
| 1989 # if it isn't a number, it's a key | |
| 1990 if type(entry) != type(''): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1991 raise ValueError('new property "%s" link value ' |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1992 'must be a string'%propname) |
| 1165 | 1993 if not num_re.match(entry): |
| 1994 try: | |
| 1995 entry = self.db.classes[link_class].lookup(entry) | |
| 1996 except (TypeError, KeyError): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1997 raise IndexError('new property "%s": %s not a %s'%( |
| 1165 | 1998 propname, entry, |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
1999 self.properties[propname].classname)) |
| 1165 | 2000 l.append(entry) |
| 2001 value = l | |
| 2002 propvalues[propname] = value | |
| 2003 | |
| 2004 # figure the journal entry for this property | |
| 2005 add = [] | |
| 2006 remove = [] | |
| 2007 | |
| 2008 # handle removals | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2009 if propname in node: |
| 1165 | 2010 l = node[propname] |
| 2011 else: | |
| 2012 l = [] | |
| 2013 for id in l[:]: | |
| 2014 if id in value: | |
| 2015 continue | |
| 2016 # register the unlink with the old linked node | |
| 2017 if self.do_journal and self.properties[propname].do_journal: | |
| 2018 self.db.addjournal(link_class, id, 'unlink', | |
| 2019 (self.classname, nodeid, propname)) | |
| 2020 l.remove(id) | |
| 2021 remove.append(id) | |
| 2022 | |
| 2023 # handle additions | |
| 2024 for id in value: | |
|
4095
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2025 if id in l: |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2026 continue |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2027 # We can safely check this condition after |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2028 # checking that this is an addition to the |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2029 # multilink since the condition was checked for |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2030 # existing entries at the point they were added to |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2031 # the multilink. Since the hasnode call will |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2032 # result in a SQL query, it is more efficient to |
|
01eb89b07c13
Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents:
4085
diff
changeset
|
2033 # avoid the check if possible. |
| 1165 | 2034 if not self.db.getclass(link_class).hasnode(id): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2035 raise IndexError('%s has no node %s'%(link_class, |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2036 id)) |
| 1165 | 2037 # register the link with the newly linked node |
| 2038 if self.do_journal and self.properties[propname].do_journal: | |
| 2039 self.db.addjournal(link_class, id, 'link', | |
| 2040 (self.classname, nodeid, propname)) | |
| 2041 l.append(id) | |
| 2042 add.append(id) | |
| 2043 | |
| 2044 # figure the journal entry | |
| 2045 l = [] | |
| 2046 if add: | |
| 2047 l.append(('+', add)) | |
| 2048 if remove: | |
| 2049 l.append(('-', remove)) | |
| 2050 multilink_changes[propname] = (add, remove) | |
| 2051 if l: | |
| 2052 journalvalues[propname] = tuple(l) | |
| 2053 | |
| 2054 elif isinstance(prop, String): | |
|
1383
f19dde90e473
applied unicode patch
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1365
diff
changeset
|
2055 if value is not None and type(value) != type('') and type(value) != type(u''): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2056 raise TypeError('new property "%s" not a string'%propname) |
|
2892
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
2057 if prop.indexme: |
|
3399
3a0d4e4a0f34
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3383
diff
changeset
|
2058 if value is None: value = '' |
|
2892
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
2059 self.db.indexer.add_text((self.classname, nodeid, propname), |
|
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
2060 value) |
| 1165 | 2061 |
| 2062 elif isinstance(prop, Password): | |
| 2063 if not isinstance(value, password.Password): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2064 raise TypeError('new property "%s" not a Password'%propname) |
| 1165 | 2065 propvalues[propname] = value |
|
4483
22bc0426e348
Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4480
diff
changeset
|
2066 journalvalues[propname] = \ |
|
22bc0426e348
Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4480
diff
changeset
|
2067 current and password.JournalPassword(current) |
| 1165 | 2068 |
| 2069 elif value is not None and isinstance(prop, Date): | |
| 2070 if not isinstance(value, date.Date): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2071 raise TypeError('new property "%s" not a Date'% propname) |
| 1165 | 2072 propvalues[propname] = value |
| 2073 | |
| 2074 elif value is not None and isinstance(prop, Interval): | |
| 2075 if not isinstance(value, date.Interval): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2076 raise TypeError('new property "%s" not an ' |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2077 'Interval'%propname) |
| 1165 | 2078 propvalues[propname] = value |
| 2079 | |
| 2080 elif value is not None and isinstance(prop, Number): | |
| 2081 try: | |
| 2082 float(value) | |
| 2083 except ValueError: | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2084 raise TypeError('new property "%s" not numeric'%propname) |
| 1165 | 2085 |
|
5067
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2086 elif value is not None and isinstance(prop, Integer): |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2087 try: |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2088 int(value) |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2089 except ValueError: |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2090 raise TypeError('new property "%s" not integer'%propname) |
|
e424987d294a
Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents:
5041
diff
changeset
|
2091 |
| 1165 | 2092 elif value is not None and isinstance(prop, Boolean): |
| 2093 try: | |
| 2094 int(value) | |
| 2095 except ValueError: | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2096 raise TypeError('new property "%s" not boolean'%propname) |
| 1165 | 2097 |
|
5112
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2098 # record quiet properties to omit from history/changelog |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2099 if prop.quiet: |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2100 quiet_props.append(propname) |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2101 |
| 1165 | 2102 # nothing to do? |
| 2103 if not propvalues: | |
| 2104 return propvalues | |
| 2105 | |
|
3082
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
2106 # update the activity time |
|
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
2107 propvalues['activity'] = date.Date() |
|
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
2108 propvalues['actor'] = self.db.getuid() |
|
29bd9f06160e
actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3057
diff
changeset
|
2109 |
|
3328
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2110 # do the set |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
2111 self.db.setnode(self.classname, nodeid, propvalues, multilink_changes) |
| 1165 | 2112 |
|
3328
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2113 # remove the activity props now they're handled |
|
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2114 del propvalues['activity'] |
|
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2115 del propvalues['actor'] |
|
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2116 |
|
475c8560ef9b
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3310
diff
changeset
|
2117 # journal the set |
| 1165 | 2118 if self.do_journal: |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
2119 self.db.addjournal(self.classname, nodeid, ''"set", journalvalues) |
| 1165 | 2120 |
|
5112
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2121 # remove quiet properties from output |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2122 for propname in quiet_props: |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2123 if propname in propvalues: |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2124 del propvalues[propname] |
|
8901cc4ef0e0
- issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents:
5096
diff
changeset
|
2125 |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
2126 return propvalues |
| 1165 | 2127 |
| 2128 def retire(self, nodeid): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2129 """Retire a node. |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2130 |
| 1165 | 2131 The properties on the node remain available from the get() method, |
| 2132 and the node's id is never reused. | |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2133 |
| 1165 | 2134 Retired nodes are not returned by the find(), list(), or lookup() |
| 2135 methods, and other nodes may reuse the values of their key properties. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2136 """ |
| 1165 | 2137 if self.db.journaltag is None: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2138 raise DatabaseError(_('Database open read-only')) |
| 1165 | 2139 |
|
1345
618aa9c37d65
fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents:
1333
diff
changeset
|
2140 self.fireAuditors('retire', nodeid, None) |
|
618aa9c37d65
fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents:
1333
diff
changeset
|
2141 |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2142 # use the arg for __retired__ to cope with any odd database type |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2143 # conversion (hello, sqlite) |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2144 sql = 'update _%s set __retired__=%s where id=%s'%(self.classname, |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2145 self.db.arg, self.db.arg) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2146 self.db.sql(sql, (nodeid, nodeid)) |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2147 if self.do_journal: |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
2148 self.db.addjournal(self.classname, nodeid, ''"retired", None) |
| 1165 | 2149 |
|
1345
618aa9c37d65
fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents:
1333
diff
changeset
|
2150 self.fireReactors('retire', nodeid, None) |
|
618aa9c37d65
fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents:
1333
diff
changeset
|
2151 |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2152 def restore(self, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2153 """Restore a retired node. |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2154 |
|
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2155 Make node available for all operations like it was before retirement. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2156 """ |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2157 if self.db.journaltag is None: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2158 raise DatabaseError(_('Database open read-only')) |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2159 |
|
1523
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2160 node = self.db.getnode(self.classname, nodeid) |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2161 # check if key property was overrided |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2162 key = self.getkey() |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2163 try: |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2164 id = self.lookup(node[key]) |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2165 except KeyError: |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2166 pass |
|
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2167 else: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2168 raise KeyError("Key property (%s) of retired node clashes " |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2169 "with existing one (%s)" % (key, node[key])) |
|
1523
63aa7be52d2c
checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1519
diff
changeset
|
2170 |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2171 self.fireAuditors('restore', nodeid, None) |
|
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2172 # use the arg for __retired__ to cope with any odd database type |
|
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2173 # conversion (hello, sqlite) |
|
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2174 sql = 'update _%s set __retired__=%s where id=%s'%(self.classname, |
|
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2175 self.db.arg, self.db.arg) |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
2176 self.db.sql(sql, (0, nodeid)) |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2177 if self.do_journal: |
|
2546
1b07c9740bba
mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2539
diff
changeset
|
2178 self.db.addjournal(self.classname, nodeid, ''"restored", None) |
|
1519
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2179 |
|
6fede2aa6a12
added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1515
diff
changeset
|
2180 self.fireReactors('restore', nodeid, None) |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2181 |
| 1165 | 2182 def is_retired(self, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2183 """Return true if the node is rerired |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2184 """ |
| 1165 | 2185 sql = 'select __retired__ from _%s where id=%s'%(self.classname, |
| 2186 self.db.arg) | |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
2187 self.db.sql(sql, (nodeid,)) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2188 return int(self.db.sql_fetchone()[0]) > 0 |
| 1165 | 2189 |
| 2190 def destroy(self, nodeid): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2191 """Destroy a node. |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2192 |
| 1165 | 2193 WARNING: this method should never be used except in extremely rare |
| 2194 situations where there could never be links to the node being | |
| 2195 deleted | |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
2196 |
| 1165 | 2197 WARNING: use retire() instead |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
2198 |
| 1165 | 2199 WARNING: the properties of this node will not be available ever again |
|
2005
fc52d57c6c3e
documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents:
1988
diff
changeset
|
2200 |
| 1165 | 2201 WARNING: really, use retire() instead |
| 2202 | |
| 2203 Well, I think that's enough warnings. This method exists mostly to | |
| 2204 support the session storage of the cgi interface. | |
| 2205 | |
| 2206 The node is completely removed from the hyperdb, including all journal | |
| 2207 entries. It will no longer be available, and will generally break code | |
| 2208 if there are any references to the node. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2209 """ |
| 1165 | 2210 if self.db.journaltag is None: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2211 raise DatabaseError(_('Database open read-only')) |
| 1165 | 2212 self.db.destroynode(self.classname, nodeid) |
| 2213 | |
| 2214 # Locating nodes: | |
| 2215 def hasnode(self, nodeid): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2216 """Determine if the given nodeid actually exists |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2217 """ |
| 1165 | 2218 return self.db.hasnode(self.classname, nodeid) |
| 2219 | |
| 2220 def setkey(self, propname): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2221 """Select a String property of this class to be the key property. |
| 1165 | 2222 |
| 2223 'propname' must be the name of a String property of this class or | |
| 2224 None, or a TypeError is raised. The values of the key property on | |
| 2225 all existing nodes must be unique or a ValueError is raised. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2226 """ |
| 1165 | 2227 prop = self.getprops()[propname] |
| 2228 if not isinstance(prop, String): | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2229 raise TypeError('key properties must be String') |
| 1165 | 2230 self.key = propname |
| 2231 | |
| 2232 def getkey(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2233 """Return the name of the key property for this class or None.""" |
| 1165 | 2234 return self.key |
| 2235 | |
| 2236 def lookup(self, keyvalue): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2237 """Locate a particular node by its key property and return its id. |
| 1165 | 2238 |
| 2239 If this class has no key property, a TypeError is raised. If the | |
| 2240 'keyvalue' matches one of the values for the key property among | |
| 2241 the nodes in this class, the matching node's id is returned; | |
| 2242 otherwise a KeyError is raised. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2243 """ |
| 1165 | 2244 if not self.key: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2245 raise TypeError('No key property set for class %s'%self.classname) |
| 1165 | 2246 |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2247 # use the arg to handle any odd database type conversion (hello, |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2248 # sqlite) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2249 sql = "select id from _%s where _%s=%s and __retired__=%s"%( |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2250 self.classname, self.key, self.db.arg, self.db.arg) |
|
3977
732a37da3a10
Fix for postgres 8.3 compatibility (and bug) (patch [SF#2030479])
Richard Jones <richard@users.sourceforge.net>
parents:
3969
diff
changeset
|
2251 self.db.sql(sql, (str(keyvalue), 0)) |
| 1165 | 2252 |
|
1174
8e318dfaf479
Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents:
1173
diff
changeset
|
2253 # see if there was a result that's not retired |
|
1203
735adcbfc665
fix to SQL lookup() and retirement
Richard Jones <richard@users.sourceforge.net>
parents:
1195
diff
changeset
|
2254 row = self.db.sql_fetchone() |
|
735adcbfc665
fix to SQL lookup() and retirement
Richard Jones <richard@users.sourceforge.net>
parents:
1195
diff
changeset
|
2255 if not row: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2256 raise KeyError('No key (%s) value "%s" for "%s"'%(self.key, |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2257 keyvalue, self.classname)) |
| 1165 | 2258 |
| 2259 # return the id | |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2260 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2261 return str(row[0]) |
| 1165 | 2262 |
| 2263 def find(self, **propspec): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2264 """Get the ids of nodes in this class which link to the given nodes. |
| 1165 | 2265 |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2266 'propspec' consists of keyword args propname=nodeid or |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2267 propname={nodeid:1, } |
| 1165 | 2268 'propname' must be the name of a property in this class, or a |
|
1912
2b0ab61db194
fixes for [SF#818339]
Richard Jones <richard@users.sourceforge.net>
parents:
1911
diff
changeset
|
2269 KeyError is raised. That property must be a Link or |
|
2b0ab61db194
fixes for [SF#818339]
Richard Jones <richard@users.sourceforge.net>
parents:
1911
diff
changeset
|
2270 Multilink property, or a TypeError is raised. |
| 1165 | 2271 |
|
3239
440f0a6a2e3c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3224
diff
changeset
|
2272 Any node in this class whose 'propname' property links to any of |
|
440f0a6a2e3c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3224
diff
changeset
|
2273 the nodeids will be returned. Examples:: |
|
440f0a6a2e3c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3224
diff
changeset
|
2274 |
|
440f0a6a2e3c
merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents:
3224
diff
changeset
|
2275 db.issue.find(messages='1') |
| 1165 | 2276 db.issue.find(messages={'1':1,'3':1}, files={'7':1}) |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2277 """ |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2278 # shortcut |
| 1165 | 2279 if not propspec: |
| 2280 return [] | |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2281 |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2282 # validate the args |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2283 props = self.getprops() |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
2284 for propname, nodeids in propspec.items(): |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2285 # check the prop is OK |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2286 prop = props[propname] |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2287 if not isinstance(prop, Link) and not isinstance(prop, Multilink): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2288 raise TypeError("'%s' not a Link/Multilink property"%propname) |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2289 |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2290 # first, links |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2291 a = self.db.arg |
|
2733
ef396596a24e
more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents:
2731
diff
changeset
|
2292 allvalues = () |
|
ef396596a24e
more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents:
2731
diff
changeset
|
2293 sql = [] |
|
1988
5660b89f8903
more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents:
1986
diff
changeset
|
2294 where = [] |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
2295 for prop, values in propspec.items(): |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2296 if not isinstance(props[prop], hyperdb.Link): |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2297 continue |
|
1912
2b0ab61db194
fixes for [SF#818339]
Richard Jones <richard@users.sourceforge.net>
parents:
1911
diff
changeset
|
2298 if type(values) is type({}) and len(values) == 1: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2299 values = list(values)[0] |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2300 if type(values) is type(''): |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2301 allvalues += (values,) |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2302 where.append('_%s = %s'%(prop, a)) |
|
1563
e2a8ce4d2317
Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents:
1557
diff
changeset
|
2303 elif values is None: |
|
e2a8ce4d2317
Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents:
1557
diff
changeset
|
2304 where.append('_%s is NULL'%prop) |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2305 else: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2306 values = list(values) |
|
2494
ea7fb2f416db
fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents:
2472
diff
changeset
|
2307 s = '' |
|
ea7fb2f416db
fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents:
2472
diff
changeset
|
2308 if None in values: |
|
ea7fb2f416db
fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents:
2472
diff
changeset
|
2309 values.remove(None) |
|
ea7fb2f416db
fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents:
2472
diff
changeset
|
2310 s = '_%s is NULL or '%prop |
|
ea7fb2f416db
fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents:
2472
diff
changeset
|
2311 allvalues += tuple(values) |
|
ea7fb2f416db
fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents:
2472
diff
changeset
|
2312 s += '_%s in (%s)'%(prop, ','.join([a]*len(values))) |
|
2530
a182c536b72d
fix filtering by Link property:
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2516
diff
changeset
|
2313 where.append('(' + s +')') |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2314 if where: |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2315 allvalues = (0, ) + allvalues |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2316 sql.append("""select id from _%s where __retired__=%s |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2317 and %s"""%(self.classname, a, ' and '.join(where))) |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2318 |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2319 # now multilinks |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
2320 for prop, values in propspec.items(): |
|
6151
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2321 p = props[prop] |
|
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2322 if not isinstance(p, hyperdb.Multilink): |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2323 continue |
|
1988
5660b89f8903
more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents:
1986
diff
changeset
|
2324 if not values: |
|
5660b89f8903
more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents:
1986
diff
changeset
|
2325 continue |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2326 allvalues += (0, ) |
|
6151
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2327 tn = p.table_name |
|
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2328 ln = p.linkid_name |
|
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2329 nn = p.nodeid_name |
|
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2330 cn = '_' + self.classname |
|
ff059afae50a
Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6148
diff
changeset
|
2331 ret = '' |
|
6155
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2332 dis = '' |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2333 ord = '' |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2334 if p.rev_property: |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2335 if isinstance(p.rev_property, Link): |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2336 ret = 'and %s.__retired__=%s ' % (tn, a) |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2337 allvalues += (0, ) |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2338 dis = 'distinct ' |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2339 ord = ' order by %s.id' % cn |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2340 if type(values) is type(''): |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2341 allvalues += (values,) |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2342 s = a |
|
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2343 else: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2344 allvalues += tuple(values) |
|
1222
bc3bc3248dd1
added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents:
1212
diff
changeset
|
2345 s = ','.join([a]*len(values)) |
|
6155
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2346 sql.append("""select %s%s.id from %s, %s where %s.__retired__=%s |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2347 %sand %s.id = %s.%s and %s.%s in (%s)%s"""%(dis, cn, cn, |
|
ba0cfc1a87c9
Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6151
diff
changeset
|
2348 tn, cn, a, ret, cn, tn, nn, tn, ln, s, ord)) |
|
2733
ef396596a24e
more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents:
2731
diff
changeset
|
2349 |
|
ef396596a24e
more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents:
2731
diff
changeset
|
2350 if not sql: |
|
1988
5660b89f8903
more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents:
1986
diff
changeset
|
2351 return [] |
|
2733
ef396596a24e
more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents:
2731
diff
changeset
|
2352 sql = ' union '.join(sql) |
|
1183
08a13a84ed43
Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents:
1181
diff
changeset
|
2353 self.db.sql(sql, allvalues) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2354 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2355 l = [str(x[0]) for x in self.db.sql_fetchall()] |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2356 return l |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2357 |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2358 def stringFind(self, **requirements): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2359 """Locate a particular node by matching a set of its String |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2360 properties in a caseless search. |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2361 |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2362 If the property is not a String property, a TypeError is raised. |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2363 |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2364 The return is a list of the id of all nodes that match. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2365 """ |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2366 where = [] |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2367 args = [] |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2368 for propname in requirements: |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2369 prop = self.properties[propname] |
|
1951
767ff2a03eee
more unit tests to improve coverage (up from 85% to 88% for anydbm! :)
Richard Jones <richard@users.sourceforge.net>
parents:
1926
diff
changeset
|
2370 if not isinstance(prop, String): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
2371 raise TypeError("'%s' not a String property"%propname) |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2372 where.append(propname) |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2373 args.append(requirements[propname].lower()) |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2374 |
|
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2375 # generate the where clause |
|
1549
a53a7e197360
fixed rdbms email address lookup (case insensitivity)
Richard Jones <richard@users.sourceforge.net>
parents:
1539
diff
changeset
|
2376 s = ' and '.join(['lower(_%s)=%s'%(col, self.db.arg) for col in where]) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2377 sql = 'select id from _%s where %s and __retired__=%s'%( |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
2378 self.classname, s, self.db.arg) |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2379 args.append(0) |
|
1195
e0032f4ab334
added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents:
1189
diff
changeset
|
2380 self.db.sql(sql, tuple(args)) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2381 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2382 l = [str(x[0]) for x in self.db.sql_fetchall()] |
| 1165 | 2383 return l |
| 2384 | |
| 2385 def list(self): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2386 """ Return a list of the ids of the active nodes in this class. |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2387 """ |
|
1484
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2388 return self.getnodeids(retired=0) |
|
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2389 |
|
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2390 def getnodeids(self, retired=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2391 """ Retrieve all the ids of the nodes for a particular Class. |
|
1484
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2392 |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2393 Set retired=None to get all nodes. Otherwise it'll get all the |
|
1484
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2394 retired or non-retired nodes, depending on the flag. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2395 """ |
|
1707
3d627e34f18e
sqlite backend now passes all tests under 2.3.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents:
1599
diff
changeset
|
2396 # flip the sense of the 'retired' flag if we don't want all of them |
|
1484
b3f2484babce
fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents:
1479
diff
changeset
|
2397 if retired is not None: |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2398 args = (0, ) |
|
1719
eeb167fb8faf
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
1707
diff
changeset
|
2399 if retired: |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2400 compare = '>' |
|
1719
eeb167fb8faf
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
1707
diff
changeset
|
2401 else: |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2402 compare = '=' |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2403 sql = 'select id from _%s where __retired__%s%s'%(self.classname, |
|
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
2404 compare, self.db.arg) |
|
1539
800b5896e14a
fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents:
1528
diff
changeset
|
2405 else: |
|
800b5896e14a
fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents:
1528
diff
changeset
|
2406 args = () |
|
800b5896e14a
fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents:
1528
diff
changeset
|
2407 sql = 'select id from _%s'%self.classname |
|
2514
091711fb2f8c
Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents:
2512
diff
changeset
|
2408 self.db.sql(sql, args) |
|
2098
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2409 # XXX numeric ids |
|
18addf2a8596
Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents:
2093
diff
changeset
|
2410 ids = [str(x[0]) for x in self.db.cursor.fetchall()] |
|
1539
800b5896e14a
fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents:
1528
diff
changeset
|
2411 return ids |
| 1165 | 2412 |
|
6399
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2413 def _subselect(self, proptree, parentname=None): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2414 """Create a subselect. This is factored out because some |
|
3633
a292054b4393
The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3612
diff
changeset
|
2415 databases (hmm only one, so far) doesn't support subselects |
|
a292054b4393
The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3612
diff
changeset
|
2416 look for "I can't believe it's not a toy RDBMS" in the mysql |
|
a292054b4393
The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3612
diff
changeset
|
2417 backend. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2418 """ |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2419 multilink_table = proptree.propclass.table_name |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2420 nodeid_name = proptree.propclass.nodeid_name |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2421 linkid_name = proptree.propclass.linkid_name |
|
6399
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2422 if parentname is None: |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2423 parentname = '_' + proptree.parent.classname |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2424 |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2425 w = '' |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2426 if proptree.need_retired: |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2427 w = ' where %s.__retired__=0'%(multilink_table) |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2428 if proptree.need_child_retired: |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2429 tn1 = multilink_table |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2430 tn2 = '_' + proptree.classname |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2431 w = ', %s where %s.%s=%s.id and %s.__retired__=0'%(tn2, |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2432 tn1, linkid_name, tn2, tn2) |
|
6399
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2433 return '%s.id not in (select %s from %s%s)'%(parentname, nodeid_name, |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2434 multilink_table, w) |
|
3633
a292054b4393
The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3612
diff
changeset
|
2435 |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2436 def _filter_multilink_expression_fallback(self, proptree, expr): |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2437 '''This is a fallback for database that do not support |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2438 subselects.''' |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2439 classname = proptree.parent.uniqname |
|
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2440 multilink_table = proptree.propclass.table_name |
|
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2441 nid = proptree.propclass.nodeid_name |
|
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2442 lid = proptree.propclass.linkid_name |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2443 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2444 is_valid = expr.evaluate |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2445 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2446 last_id, kws = None, [] |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2447 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2448 ids = IdListOptimizer() |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2449 append = ids.append |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2450 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2451 # This join and the evaluation in program space |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2452 # can be expensive for larger databases! |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2453 # TODO: Find a faster way to collect the data needed |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2454 # to evalute the expression. |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2455 # Moving the expression evaluation into the database |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2456 # would be nice but this tricky: Think about the cases |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2457 # where the multilink table does not have join values |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2458 # needed in evaluation. |
|
6402
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2459 w = j = '' |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2460 s = 'm.%s' % lid |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2461 if proptree.need_retired: |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2462 w = ' and m.__retired__=0' |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2463 elif proptree.need_child_retired: |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2464 tn2 = '_' + proptree.classname |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2465 j = ' LEFT OUTER JOIN %s ON %s.id = m.%s' % (tn2, tn2, lid) |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2466 w = ' and %s.__retired__=0'%(tn2) |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2467 s = '%s.id' % tn2 |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2468 |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2469 stmnt = "SELECT c.id, %s FROM _%s as c " \ |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2470 "LEFT OUTER JOIN %s as m " \ |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2471 "ON c.id = m.%s%s%s ORDER BY c.id" % ( |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2472 s, classname, multilink_table, nid, j, w) |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2473 self.db.sql(stmnt) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2474 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2475 # collect all multilink items for a class item |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2476 for nodeid, kw in self.db.sql_fetchiter(): |
|
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2477 if nodeid != last_id: |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2478 if last_id is None: |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2479 last_id = nodeid |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2480 else: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2481 # we have all multilink items -> evaluate! |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2482 if is_valid(kws): append(last_id) |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2483 last_id, kws = nodeid, [] |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2484 if kw is not None: |
|
6395
8baf81d1cfc1
Make multilink expressions work with mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6392
diff
changeset
|
2485 kws.append(int(kw)) |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2486 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2487 if last_id is not None and is_valid(kws): |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2488 append(last_id) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2489 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2490 # we have ids of the classname table |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2491 return ids.where("_%s.id" % classname, self.db.arg) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2492 |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2493 def _filter_link_expression(self, proptree, v): |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2494 """ Filter elements in the table that match the given expression |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2495 """ |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2496 pln = proptree.parent.uniqname |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2497 prp = proptree.name |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2498 try: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2499 opcodes = [int(x) for x in v] |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2500 if min(opcodes) >= -1: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2501 raise ValueError() |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2502 expr = compile_expression(opcodes) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2503 # NULL doesn't compare to NULL in SQL |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2504 # So not (x = '1') will *not* include NULL values for x |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2505 # That's why we need that and clause: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2506 atom = "_%s._%s = %s and _%s._%s is not NULL" % ( |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2507 pln, prp, self.db.arg, pln, prp) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2508 atom_nil = "_%s._%s is NULL" % (pln, prp) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2509 lambda_atom = lambda n: atom if n.x >= 0 else atom_nil |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2510 values = [] |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2511 w = expr.generate(lambda_atom) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2512 def collect_values(n): |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2513 if n.x >= 0: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2514 values.append(n.x) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2515 expr.visit(collect_values) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2516 return w, values |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2517 except: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2518 pass |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2519 # Fallback to original code |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2520 args = [] |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2521 where = None |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2522 d = {} |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2523 for entry in v: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2524 if entry == '-1': |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2525 entry = None |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2526 d[entry] = entry |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2527 l = [] |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2528 if None in d or not d: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2529 if None in d: del d[None] |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2530 l.append('_%s._%s is NULL'%(pln, prp)) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2531 if d: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2532 v = list(d) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2533 s = ','.join([self.db.arg for x in v]) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2534 l.append('(_%s._%s in (%s))'%(pln, prp, s)) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2535 args = v |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2536 if l: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2537 where = '(' + ' or '.join(l) +')' |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2538 return where, args |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2539 |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2540 def _filter_multilink_expression(self, proptree, v): |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2541 """ Filters out elements of the classname table that do not |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2542 match the given expression. |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2543 Returns tuple of 'WHERE' introns for the overall filter. |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2544 """ |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2545 classname = proptree.parent.uniqname |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2546 multilink_table = proptree.propclass.table_name |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2547 nid = proptree.propclass.nodeid_name |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2548 lid = proptree.propclass.linkid_name |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2549 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2550 try: |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2551 opcodes = [int(x) for x in v] |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2552 if min(opcodes) >= -1: |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2553 raise ValueError() |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2554 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2555 expr = compile_expression(opcodes) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2556 |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2557 if not self.supports_subselects: |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2558 # We heavily rely on subselects. If there is |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2559 # no decent support fall back to slower variant. |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2560 return self._filter_multilink_expression_fallback( |
|
6400
5ce995c33eee
Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6399
diff
changeset
|
2561 proptree, expr) |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2562 |
|
6402
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2563 w = j = '' |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2564 if proptree.need_retired: |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2565 w = ' and %s.__retired__=0'%(multilink_table) |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2566 elif proptree.need_child_retired: |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2567 tn1 = multilink_table |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2568 tn2 = '_' + proptree.classname |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2569 j = ', %s' % tn2 |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2570 w = ' and %s.%s=%s.id and %s.__retired__=0'%(tn1, lid, tn2, tn2) |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2571 |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2572 atom = \ |
|
6402
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2573 "%s IN(SELECT %s FROM %s%s WHERE %s=a.id%s)" % ( |
|
619807d9a2df
Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6401
diff
changeset
|
2574 self.db.arg, lid, multilink_table, j, nid, w) |
|
6399
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2575 atom_nil = self._subselect(proptree, 'a') |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2576 |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2577 lambda_atom = lambda n: atom if n.x >= 0 else atom_nil |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2578 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2579 intron = \ |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2580 "_%(classname)s.id in (SELECT id " \ |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2581 "FROM _%(classname)s AS a WHERE %(condition)s) " % { |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2582 'classname' : classname, |
|
6399
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2583 'condition' : expr.generate(lambda_atom) } |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2584 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2585 values = [] |
|
6399
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2586 def collect_values(n): |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2587 if n.x >= 0: |
|
f3fcd6628c0c
Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6396
diff
changeset
|
2588 values.append(n.x) |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2589 expr.visit(collect_values) |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2590 |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2591 return intron, values |
|
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2592 except: |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2593 # fallback behavior when expression parsing above fails |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2594 orclause = '' |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2595 if '-1' in v : |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2596 v = [x for x in v if int (x) > 0] |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2597 orclause = self._subselect(proptree) |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2598 where = [] |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2599 where.append("%s.%s in (%s)" % (multilink_table, lid, |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2600 ','.join([self.db.arg] * len(v)))) |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2601 where.append('_%s.id=%s.%s'%(classname, multilink_table, nid)) |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2602 where = ' and '.join (where) |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2603 if orclause : |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2604 where = '((' + ' or '.join ((where + ')', orclause)) + ')' |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2605 |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2606 return where, v |
|
4466
f1fe6fd0aa61
Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents:
4462
diff
changeset
|
2607 |
|
5318
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2608 def _filter_sql (self, search_matches, filterspec, srt=[], grp=[], retr=0, |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2609 retired=False, exact_match_spec={}, limit=None, |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2610 offset=None): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2611 """ Compute the proptree and the SQL/ARGS for a filter. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2612 For argument description see filter below. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2613 We return a 3-tuple, the proptree, the sql and the sql-args |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2614 or None if no SQL is necessary. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2615 The flag retr serves to retrieve *all* non-Multilink properties |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2616 (for filling the cache during a filter_iter) |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
2617 """ |
|
2578
7f25486ff85e
fixed RDBMS filter() for no matches from full-text search [SF#990778]
Richard Jones <richard@users.sourceforge.net>
parents:
2546
diff
changeset
|
2618 # we can't match anything if search_matches is empty |
| 4044 | 2619 if not search_matches and search_matches is not None: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2620 return None |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
2621 |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2622 icn = self.classname |
| 1165 | 2623 |
|
2379
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2624 # vars to hold the components of the SQL statement |
|
2586
3c67f4bfa225
*** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents:
2578
diff
changeset
|
2625 frum = [] # FROM clauses |
|
2379
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2626 loj = [] # LEFT OUTER JOIN clauses |
|
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2627 where = [] # WHERE clauses |
|
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2628 args = [] # *any* positional arguments |
|
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2629 a = self.db.arg |
|
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2630 |
| 1165 | 2631 # figure the WHERE clause from the filterspec |
|
6413
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2632 use_distinct = False # Do we need a distinct clause? |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2633 sortattr = self._sortattr (group = grp, sort = srt) |
|
5871
acc4a128ab9b
Backwards-compatible signature of _proptree method
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5869
diff
changeset
|
2634 proptree = self._proptree(filterspec, exact_match_spec, sortattr, retr) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2635 mlseen = 0 |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2636 for pt in reversed(proptree.sortattr): |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2637 p = pt |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2638 while p.parent: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2639 if isinstance (p.propclass, Multilink): |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2640 mlseen = True |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2641 if mlseen: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2642 p.sort_ids_needed = True |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2643 p.tree_sort_done = False |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2644 p = p.parent |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2645 if not mlseen: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2646 pt.attr_sort_done = pt.tree_sort_done = True |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2647 proptree.compute_sort_done() |
|
3693
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2648 |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2649 cols = ['_%s.id'%icn] |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2650 mlsort = [] |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2651 rhsnum = 0 |
|
3635
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
2652 for p in proptree: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2653 rc = ac = oc = None |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2654 cn = p.classname |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2655 ln = p.uniqname |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2656 pln = p.parent.uniqname |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2657 pcn = p.parent.classname |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2658 k = p.name |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2659 v = p.val |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2660 propclass = p.propclass |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2661 if p.parent == proptree and p.name == 'id' \ |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2662 and 'retrieve' in p.need_for: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2663 p.sql_idx = 0 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2664 if 'sort' in p.need_for or 'retrieve' in p.need_for: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2665 rc = oc = ac = '_%s._%s'%(pln, k) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2666 if isinstance(propclass, Multilink): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2667 if 'search' in p.need_for: |
|
6413
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2668 # if we joining with Multilink tables we need distinct |
|
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2669 use_distinct = True |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
2670 tn = propclass.table_name |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
2671 nid = propclass.nodeid_name |
|
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
2672 lid = propclass.linkid_name |
|
6403
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2673 frum.append(tn) |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2674 if p.children or p.need_child_retired: |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2675 frum.append('_%s as _%s' % (cn, ln)) |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2676 where.append('%s.%s=_%s.id'%(tn, lid, ln)) |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2677 if p.need_child_retired: |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2678 where.append('_%s.__retired__=0'%(ln)) |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2679 # Note: need the where-clause if p has |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2680 # children that compute additional restrictions |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2681 if (not p.has_values |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2682 or (not isinstance(v, type([])) and v != '-1') |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2683 or p.children): |
|
9957d8d10783
Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6402
diff
changeset
|
2684 where.append('_%s.id=%s.%s'%(pln, tn, nid)) |
|
4061
52431f2ad76b
Correct handling of a Multilink filterspec with an empty list of linked-to items.
Stefan Seefeld <stefan@seefeld.name>
parents:
4060
diff
changeset
|
2685 if v in ('-1', ['-1'], []): |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2686 # only match rows that have count(linkid)=0 in the |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2687 # corresponding multilink table) |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2688 where.append(self._subselect(p)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2689 else: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2690 if p.has_values: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2691 if isinstance(v, type([])): |
|
6396
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2692 # The where-clause above is conditionally |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2693 # created in _filter_multilink_expression |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2694 w, arg = self._filter_multilink_expression(p, v) |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2695 where.append(w) |
|
75a53956cf13
Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6395
diff
changeset
|
2696 args += arg |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2697 else: |
|
6148
8497bf3f23a1
Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6133
diff
changeset
|
2698 where.append('%s.%s=%s'%(tn, lid, a)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2699 args.append(v) |
|
6179
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2700 # Don't match retired nodes if rev_multilink |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2701 if p.need_retired: |
|
a701c9c81597
Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6157
diff
changeset
|
2702 where.append('%s.__retired__=0'%(tn)) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2703 if 'sort' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2704 assert not p.attr_sort_done and not p.sort_ids_needed |
|
1365
4884fb0860f9
fixed rdbms searching by ID [SF#666615]
Richard Jones <richard@users.sourceforge.net>
parents:
1351
diff
changeset
|
2705 elif k == 'id': |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2706 if 'search' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2707 if isinstance(v, type([])): |
|
4462
0f4a8fcb9a1d
fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents:
4448
diff
changeset
|
2708 # If there are no permitted values, then the |
|
0f4a8fcb9a1d
fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents:
4448
diff
changeset
|
2709 # where clause will always be false, and we |
|
0f4a8fcb9a1d
fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents:
4448
diff
changeset
|
2710 # can optimize the query away. |
|
0f4a8fcb9a1d
fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents:
4448
diff
changeset
|
2711 if not v: |
|
4803
5e679e364f9a
Bug-fix: need to return None for empty query
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4787
diff
changeset
|
2712 return None |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2713 s = ','.join([a for x in v]) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2714 where.append('_%s.%s in (%s)'%(pln, k, s)) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2715 args = args + v |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2716 else: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2717 where.append('_%s.%s=%s'%(pln, k, a)) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2718 args.append(v) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2719 if 'sort' in p.need_for or 'retrieve' in p.need_for: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2720 rc = oc = ac = '_%s.id'%pln |
|
3683
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2721 elif isinstance(propclass, String): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2722 if 'search' in p.need_for: |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2723 exact = [] |
|
3683
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2724 if not isinstance(v, type([])): |
|
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2725 v = [v] |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2726 new_v = [] |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2727 for x in v: |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2728 if isinstance(x, hyperdb.Exact_Match): |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2729 exact.append(True) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2730 new_v.append(x.value) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2731 else: |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2732 exact.append(False) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2733 # Quote special search characters '%' and '_' for |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2734 # correct matching with LIKE/ILIKE |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2735 # Note that we now pass the elements of v as query |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2736 # arguments and don't interpolate the quoted string |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2737 # into the sql statement. Should be safer. |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2738 new_v.append(self.db.search_stringquote(x)) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2739 v = new_v |
|
3683
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2740 |
|
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2741 # now add to the where clause |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2742 w = [] |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2743 for vv, ex in zip(v, exact): |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2744 if ex: |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2745 w.append("_%s._%s %s %s"%( |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2746 pln, k, self.case_sensitive_equal, a)) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2747 args.append(vv) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2748 else: |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2749 w.append("_%s._%s %s %s ESCAPE %s"%( |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2750 pln, k, self.case_insensitive_like, a, a)) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2751 args.extend((vv, '\\')) |
|
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2752 where.append ('(' + ' and '.join(w) + ')') |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2753 if 'sort' in p.need_for: |
|
3683
ad5e1d75e028
Fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3682
diff
changeset
|
2754 oc = ac = 'lower(_%s._%s)'%(pln, k) |
|
1170
af104fa52746
Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1168
diff
changeset
|
2755 elif isinstance(propclass, Link): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2756 if 'search' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2757 if p.children: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2758 if 'sort' not in p.need_for: |
|
3694
595041d78104
Fix a peculiarity in sql generation.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3693
diff
changeset
|
2759 frum.append('_%s as _%s' % (cn, ln)) |
|
6413
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2760 c = [x for x in p.children if 'search' in x.need_for] |
|
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2761 if c: |
|
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2762 where.append('_%s._%s=_%s.id'%(pln, k, ln)) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2763 if p.has_values: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2764 if isinstance(v, type([])): |
|
6415
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2765 w, arg = self._filter_link_expression(p, v) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2766 if w: |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2767 where.append(w) |
|
dbacf6bf2a2f
Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6413
diff
changeset
|
2768 args += arg |
|
3636
fa7becc62534
Of course for Links there is the same bug as I fixed before for multilinks.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3635
diff
changeset
|
2769 else: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2770 if v in ('-1', None): |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2771 v = None |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2772 where.append('_%s._%s is NULL'%(pln, k)) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2773 else: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2774 where.append('_%s._%s=%s'%(pln, k, a)) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2775 args.append(v) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2776 if 'sort' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2777 lp = p.cls.labelprop() |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2778 oc = ac = '_%s._%s'%(pln, k) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2779 if lp != 'id': |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2780 if p.tree_sort_done: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2781 loj.append( |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2782 'LEFT OUTER JOIN _%s as _%s on _%s._%s=_%s.id'%( |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2783 cn, ln, pln, k, ln)) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2784 oc = '_%s._%s'%(ln, lp) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2785 if 'retrieve' in p.need_for: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2786 rc = '_%s._%s'%(pln, k) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2787 elif isinstance(propclass, Date) and 'search' in p.need_for: |
|
4060
2a68d7494bbc
Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents:
4059
diff
changeset
|
2788 dc = self.db.to_sql_value(hyperdb.Date) |
|
1351
d1bfb479e527
fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents:
1345
diff
changeset
|
2789 if isinstance(v, type([])): |
|
d1bfb479e527
fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents:
1345
diff
changeset
|
2790 s = ','.join([a for x in v]) |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2791 where.append('_%s._%s in (%s)'%(pln, k, s)) |
|
3633
a292054b4393
The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3612
diff
changeset
|
2792 args = args + [dc(date.Date(x)) for x in v] |
|
1351
d1bfb479e527
fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents:
1345
diff
changeset
|
2793 else: |
|
1499
8ee69708da0c
added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1496
diff
changeset
|
2794 try: |
| 6118 | 2795 wh = [] |
| 2796 ar = [] | |
| 2797 for d in v.split(','): | |
| 2798 w1 = [] | |
| 2799 if d == '-': | |
| 2800 wh.append('_%s._%s is NULL'%(pln, k)) | |
| 2801 continue | |
| 2802 # Try to filter on range of dates | |
| 2803 date_rng = propclass.range_from_raw(d, self.db) | |
| 2804 if date_rng.from_value: | |
| 2805 w1.append('_%s._%s >= %s'%(pln, k, a)) | |
| 2806 ar.append(dc(date_rng.from_value)) | |
| 2807 if date_rng.to_value: | |
| 2808 w1.append('_%s._%s <= %s'%(pln, k, a)) | |
| 2809 ar.append(dc(date_rng.to_value)) | |
| 2810 wh.append (' and '.join (w1)) | |
| 2811 where.append ('(' + ' or '.join (wh) + ')') | |
| 2812 args.extend (ar) | |
|
1499
8ee69708da0c
added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1496
diff
changeset
|
2813 except ValueError: |
|
8ee69708da0c
added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents:
1496
diff
changeset
|
2814 # If range creation fails - ignore that search parameter |
|
2516
125311efe783
fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2514
diff
changeset
|
2815 pass |
|
1351
d1bfb479e527
fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents:
1345
diff
changeset
|
2816 elif isinstance(propclass, Interval): |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2817 # filter/sort using the __<prop>_int__ column |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2818 if 'search' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2819 if isinstance(v, type([])): |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2820 s = ','.join([a for x in v]) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2821 where.append('_%s.__%s_int__ in (%s)'%(pln, k, s)) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2822 args = args + [date.Interval(x).as_seconds() for x in v] |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2823 else: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2824 try: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2825 # Try to filter on range of intervals |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2826 date_rng = Range(v, date.Interval) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2827 if date_rng.from_value: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2828 where.append('_%s.__%s_int__ >= %s'%(pln, k, a)) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2829 args.append(date_rng.from_value.as_seconds()) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2830 if date_rng.to_value: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2831 where.append('_%s.__%s_int__ <= %s'%(pln, k, a)) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2832 args.append(date_rng.to_value.as_seconds()) |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2833 except ValueError: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2834 # If range creation fails - ignore search parameter |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2835 pass |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2836 if 'sort' in p.need_for: |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2837 oc = ac = '_%s.__%s_int__'%(pln,k) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2838 if 'retrieve' in p.need_for: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2839 rc = '_%s._%s'%(pln,k) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2840 elif isinstance(propclass, Boolean) and 'search' in p.need_for: |
|
4365
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2841 if type(v) == type(""): |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2842 v = v.split(',') |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2843 if type(v) != type([]): |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2844 v = [v] |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2845 bv = [] |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2846 for val in v: |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2847 if type(val) is type(''): |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2848 bv.append(propclass.from_raw (val)) |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2849 else: |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2850 bv.append(bool(val)) |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2851 if len(bv) == 1: |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2852 where.append('_%s._%s=%s'%(pln, k, a)) |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2853 args = args + bv |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2854 else: |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2855 s = ','.join([a for x in v]) |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2856 where.append('_%s._%s in (%s)'%(pln, k, s)) |
|
667c818f6a22
- unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4359
diff
changeset
|
2857 args = args + bv |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2858 elif 'search' in p.need_for: |
| 1165 | 2859 if isinstance(v, type([])): |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
2860 s = ','.join([a for x in v]) |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2861 where.append('_%s._%s in (%s)'%(pln, k, s)) |
| 1165 | 2862 args = args + v |
| 2863 else: | |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2864 where.append('_%s._%s=%s'%(pln, k, a)) |
| 1165 | 2865 args.append(v) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2866 if oc: |
|
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2867 if p.sort_ids_needed: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2868 if rc == ac: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2869 p.sql_idx = len(cols) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2870 p.auxcol = len(cols) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2871 cols.append(ac) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2872 if p.tree_sort_done and p.sort_direction: |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2873 # Don't select top-level id or multilink twice |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2874 if (not p.sort_ids_needed or ac != oc) and (p.name != 'id' |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2875 or p.parent != proptree): |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2876 if rc == oc: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2877 p.sql_idx = len(cols) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2878 cols.append(oc) |
|
3685
4d9adb8bc3b1
Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3683
diff
changeset
|
2879 desc = ['', ' desc'][p.sort_direction == '-'] |
|
4d9adb8bc3b1
Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3683
diff
changeset
|
2880 # Some SQL dbs sort NULL values last -- we want them first. |
|
4d9adb8bc3b1
Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3683
diff
changeset
|
2881 if (self.order_by_null_values and p.name != 'id'): |
|
4d9adb8bc3b1
Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3683
diff
changeset
|
2882 nv = self.order_by_null_values % oc |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2883 cols.append(nv) |
|
3693
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2884 p.orderby.append(nv + desc) |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2885 p.orderby.append(oc + desc) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2886 if 'retrieve' in p.need_for and p.sql_idx is None: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2887 assert(rc) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2888 p.sql_idx = len(cols) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2889 cols.append (rc) |
| 1165 | 2890 |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2891 props = self.getprops() |
|
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2892 |
|
1740
5ca448ff8052
don't have RDBMS backends list retired nodes [SF#767319]
Richard Jones <richard@users.sourceforge.net>
parents:
1719
diff
changeset
|
2893 # don't match retired nodes |
|
5318
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2894 if retired is not None: |
|
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2895 op = '=' |
|
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2896 if retired: |
|
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2897 op = '!=' |
|
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2898 where.append('_%s.__retired__%s0'%(icn, op)) |
|
1740
5ca448ff8052
don't have RDBMS backends list retired nodes [SF#767319]
Richard Jones <richard@users.sourceforge.net>
parents:
1719
diff
changeset
|
2899 |
| 1165 | 2900 # add results of full text search |
| 2901 if search_matches is not None: | |
| 4044 | 2902 s = ','.join([a for x in search_matches]) |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2903 where.append('_%s.id in (%s)'%(icn, s)) |
| 4113 | 2904 args = args + [x for x in search_matches] |
| 1165 | 2905 |
| 2906 # construct the SQL | |
|
3634
57c66056ffe4
Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3633
diff
changeset
|
2907 frum.append('_'+icn) |
| 1165 | 2908 frum = ','.join(frum) |
|
1170
af104fa52746
Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1168
diff
changeset
|
2909 if where: |
|
af104fa52746
Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1168
diff
changeset
|
2910 where = ' where ' + (' and '.join(where)) |
|
af104fa52746
Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1168
diff
changeset
|
2911 else: |
|
af104fa52746
Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1168
diff
changeset
|
2912 where = '' |
|
6413
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2913 if use_distinct: |
|
7b1b6dffc7ed
Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6403
diff
changeset
|
2914 # Avoid dupes |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2915 cols[0] = 'distinct(_%s.id)'%icn |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2916 |
|
3693
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2917 order = [] |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2918 # keep correct sequence of order attributes. |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2919 for sa in proptree.sortattr: |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2920 if not sa.attr_sort_done: |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2921 continue |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2922 order.extend(sa.orderby) |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2923 if order: |
|
a775afeeee8b
Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3688
diff
changeset
|
2924 order = ' order by %s'%(','.join(order)) |
| 1165 | 2925 else: |
| 2926 order = '' | |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2927 |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2928 if limit is not None: |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2929 limit = ' LIMIT %s' % limit |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2930 else: |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2931 limit = '' |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2932 if offset is not None: |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2933 offset = ' OFFSET %s' % offset |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2934 else: |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2935 offset = '' |
| 1165 | 2936 cols = ','.join(cols) |
|
2379
a2025bdd1491
fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents:
2374
diff
changeset
|
2937 loj = ' '.join(loj) |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2938 sql = 'select %s from %s %s %s%s%s%s'%( |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2939 cols, frum, loj, where, order, limit, offset) |
| 1165 | 2940 args = tuple(args) |
|
2319
7cf7e3bd1b31
more column uniqueness fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2318
diff
changeset
|
2941 __traceback_info__ = (sql, args) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2942 return proptree, sql, args |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2943 |
|
5318
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2944 def filter(self, search_matches, filterspec, sort=[], group=[], |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2945 retired=False, exact_match_spec={}, limit=None, offset=None): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2946 """Return a list of the ids of the active nodes in this class that |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2947 match the 'filter' spec, sorted by the group spec and then the |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2948 sort spec |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2949 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2950 "filterspec" is {propname: value(s)} |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2951 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2952 "sort" and "group" are [(dir, prop), ...] where dir is '+', '-' |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2953 or None and prop is a prop name or None. Note that for |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2954 backward-compatibility reasons a single (dir, prop) tuple is |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2955 also allowed. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2956 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2957 "search_matches" is a container type or None |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2958 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2959 The filter must match all properties specificed. If the property |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2960 value to match is a list: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2961 |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2962 1. String properties must match all elements in the list, and |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2963 2. Other properties must match any of the elements in the list. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2964 """ |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2965 if __debug__: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2966 start_t = time.time() |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2967 |
|
5318
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
2968 sq = self._filter_sql (search_matches, filterspec, sort, group, |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
2969 retired=retired, |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2970 exact_match_spec=exact_match_spec, |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
2971 limit=limit, offset=offset) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2972 # nothing to match? |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2973 if sq is None: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2974 return [] |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2975 proptree, sql, args = sq |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
2976 |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2977 cursor = self.db.sql_new_cursor(name='filter') |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2978 self.db.sql(sql, args, cursor) |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2979 # Reduce this to only the first row (the ID), this can save a |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2980 # lot of space for large query results (not using fetchall) |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2981 # We cannot do this if sorting by multilink |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2982 if proptree.tree_sort_done: |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2983 l = [str(row[0]) for row in cursor] |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2984 else: |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2985 l = cursor.fetchall() |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2986 cursor.close() |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2987 |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2988 # Multilink sorting |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
2989 # Compute values needed for sorting in proptree.sort |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2990 if not proptree.tree_sort_done: |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2991 for p in proptree: |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2992 if hasattr(p, 'auxcol'): |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2993 p.sort_ids = [row[p.auxcol] for row in l] |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2994 p.sort_result = p._sort_repr \ |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2995 (p.propclass.sort_repr, p.sort_ids) |
|
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
2996 l = proptree.sort ([str(row[0]) for row in l]) |
|
2237
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
2997 |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
2998 if __debug__: |
|
f624fc20f8fe
added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents:
2234
diff
changeset
|
2999 self.db.stats['filtering'] += (time.time() - start_t) |
|
3682
193f316dbbe9
More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3636
diff
changeset
|
3000 return l |
|
1168
94620e088e3a
fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents:
1165
diff
changeset
|
3001 |
|
5318
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
3002 def filter_iter(self, search_matches, filterspec, sort=[], group=[], |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3003 retired=False, exact_match_spec={}, limit=None, |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3004 offset=None): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3005 """Iterator similar to filter above with same args. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3006 Limitation: We don't sort on multilinks. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3007 This uses an optimisation: We put all nodes that are in the |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3008 current row into the node cache. Then we return the node id. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3009 That way a fetch of a node won't create another sql-fetch (with |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3010 a join) from the database because the nodes are already in the |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3011 cache. We're using our own temporary cursor. |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3012 """ |
|
5318
506c7ee9a385
Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5248
diff
changeset
|
3013 sq = self._filter_sql(search_matches, filterspec, sort, group, retr=1, |
|
5867
ee2e8f8d6648
Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5809
diff
changeset
|
3014 retired=retired, |
|
5869
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3015 exact_match_spec=exact_match_spec, |
|
16e1255b16cf
Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5867
diff
changeset
|
3016 limit=limit, offset=offset) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3017 # nothing to match? |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3018 if sq is None: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3019 return |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3020 proptree, sql, args = sq |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3021 cursor = self.db.sql_new_cursor(name='filter_iter') |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3022 self.db.sql(sql, args, cursor) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3023 classes = {} |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3024 for p in proptree: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3025 if 'retrieve' in p.need_for: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3026 cn = p.parent.classname |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3027 ptid = p.parent.id # not the nodeid! |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3028 key = (cn, ptid) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3029 if key not in classes: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3030 classes[key] = {} |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3031 name = p.name |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3032 assert (name) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3033 classes[key][name] = p |
|
4473
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
3034 p.to_hyperdb = self.db.to_hyperdb_value(p.propclass.__class__) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3035 while True: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3036 row = cursor.fetchone() |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3037 if not row: break |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3038 # populate cache with current items |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
3039 for (classname, ptid), pt in classes.items(): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3040 nodeid = str(row[pt['id'].sql_idx]) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3041 key = (classname, nodeid) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3042 if key in self.db.cache: |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3043 self.db._cache_refresh(key) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3044 continue |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3045 node = {} |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
3046 for propname, p in pt.items(): |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3047 value = row[p.sql_idx] |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3048 if value is not None: |
|
4473
fccf7e09af0c
- optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4472
diff
changeset
|
3049 value = p.to_hyperdb(value) |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3050 node[propname] = value |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3051 self.db._cache_save(key, node) |
|
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3052 yield str(row[0]) |
|
6332
6a6b4651be1f
Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6238
diff
changeset
|
3053 cursor.close() |
|
4472
34dce76bb202
Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4466
diff
changeset
|
3054 |
|
3688
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3055 def filter_sql(self, sql): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3056 """Return a list of the ids of the items in this class that match |
|
3688
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3057 the SQL provided. The SQL is a complete "select" statement. |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3058 |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3059 The SQL select must include the item id as the first column. |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3060 |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3061 This function DOES NOT filter out retired items, add on a where |
|
3963
3230f9c88086
Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents:
3933
diff
changeset
|
3062 clause "__retired__=0" if you don't want retired nodes. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3063 """ |
|
3688
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3064 if __debug__: |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3065 start_t = time.time() |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3066 |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3067 self.db.sql(sql) |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3068 l = self.db.sql_fetchall() |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3069 |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3070 if __debug__: |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3071 self.db.stats['filtering'] += (time.time() - start_t) |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3072 return l |
|
722ab52d47fc
added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents:
3687
diff
changeset
|
3073 |
| 1165 | 3074 def count(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3075 """Get the number of nodes in this class. |
| 1165 | 3076 |
| 3077 If the returned integer is 'numnodes', the ids of all the nodes | |
| 3078 in this class run from 1 to numnodes, and numnodes+1 will be the | |
| 3079 id of the next node to be created in this class. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3080 """ |
| 1165 | 3081 return self.db.countnodes(self.classname) |
| 3082 | |
| 3083 # Manipulating properties: | |
| 3084 def getprops(self, protected=1): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3085 """Return a dictionary mapping property names to property objects. |
| 1165 | 3086 If the "protected" flag is true, we include protected properties - |
| 3087 those which may not be modified. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3088 """ |
| 1165 | 3089 d = self.properties.copy() |
| 3090 if protected: | |
| 3091 d['id'] = String() | |
| 3092 d['creation'] = hyperdb.Date() | |
| 3093 d['activity'] = hyperdb.Date() | |
|
1176
bd3b57859c37
On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents:
1175
diff
changeset
|
3094 d['creator'] = hyperdb.Link('user') |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
3095 d['actor'] = hyperdb.Link('user') |
| 1165 | 3096 return d |
| 3097 | |
| 3098 def addprop(self, **properties): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3099 """Add properties to this class. |
| 1165 | 3100 |
| 3101 The keyword arguments in 'properties' must map names to property | |
| 3102 objects, or a TypeError is raised. None of the keys in 'properties' | |
| 3103 may collide with the names of existing properties, or a ValueError | |
| 3104 is raised before any properties have been added. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3105 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3106 for key in properties: |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3107 if key in self.properties: |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3108 raise ValueError(key) |
| 1165 | 3109 self.properties.update(properties) |
| 3110 | |
| 3111 def index(self, nodeid): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3112 """Add (or refresh) the node to search indexes |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3113 """ |
| 1165 | 3114 # find all the String properties that have indexme |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
3115 for prop, propclass in self.getprops().items(): |
| 1165 | 3116 if isinstance(propclass, String) and propclass.indexme: |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3117 self.db.indexer.add_text((self.classname, nodeid, prop), |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3118 str(self.get(nodeid, prop))) |
| 1165 | 3119 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3120 # |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3121 # import / export support |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3122 # |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3123 def export_list(self, propnames, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3124 """ Export a node - generate a list of CSV-able data in the order |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3125 specified by propnames for the given node. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3126 """ |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3127 properties = self.getprops() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3128 l = [] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3129 for prop in propnames: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3130 proptype = properties[prop] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3131 value = self.get(nodeid, prop) |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3132 # "marshal" data where needed |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3133 if value is None: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3134 pass |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3135 elif isinstance(proptype, hyperdb.Date): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3136 value = value.get_tuple() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3137 elif isinstance(proptype, hyperdb.Interval): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3138 value = value.get_tuple() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3139 elif isinstance(proptype, hyperdb.Password): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3140 value = str(value) |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3141 l.append(repr_export(value)) |
|
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3142 l.append(repr_export(self.is_retired(nodeid))) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3143 return l |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3144 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3145 def import_list(self, propnames, proplist): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3146 """ Import a node - all information including "id" is present and |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3147 should not be sanity checked. Triggers are not triggered. The |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3148 journal should be initialised using the "creator" and "created" |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3149 information. |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3150 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3151 Return the nodeid of the node imported. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3152 """ |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3153 |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3154 logger = logging.getLogger('roundup.hyperdb.backend') |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3155 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3156 if self.db.journaltag is None: |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3157 raise DatabaseError(_('Database open read-only')) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3158 properties = self.getprops() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3159 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3160 # make the new node's property map |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3161 d = {} |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3162 retire = 0 |
|
2505
bdd112cf61ba
rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
3163 if not "id" in propnames: |
|
bdd112cf61ba
rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
3164 newid = self.db.newid(self.classname) |
|
bdd112cf61ba
rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
3165 else: |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3166 newid = eval_import(proplist[propnames.index("id")]) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3167 for i in range(len(propnames)): |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3168 # Use eval_import to reverse the repr_export() used to |
|
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3169 # output the CSV |
|
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3170 value = eval_import(proplist[i]) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3171 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3172 # Figure the property for this column |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3173 propname = propnames[i] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3174 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3175 # "unmarshal" where necessary |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3176 if propname == 'id': |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3177 continue |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3178 elif propname == 'is retired': |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3179 # is the item retired? |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3180 if int(value): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3181 retire = 1 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3182 continue |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3183 elif value is None: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3184 d[propname] = None |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3185 continue |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3186 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3187 prop = properties[propname] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3188 if value is None: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3189 # don't set Nones |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3190 continue |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3191 elif isinstance(prop, hyperdb.Date): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3192 value = date.Date(value) |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3193 elif isinstance(prop, hyperdb.Interval): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3194 value = date.Interval(value) |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3195 elif isinstance(prop, hyperdb.Password): |
|
4480
1613754d2646
Fix first part of Password handling security issue2550688
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
4476
diff
changeset
|
3196 value = password.Password(encrypted=value) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
3197 elif isinstance(prop, String): |
|
5416
56c9bcdea47f
Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5395
diff
changeset
|
3198 value = us2s(value) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
3199 if not isinstance(value, str): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3200 raise TypeError('new property "%(propname)s" not a ' |
|
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3201 'string: %(value)r'%locals()) |
|
2892
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
3202 if prop.indexme: |
|
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
3203 self.db.indexer.add_text((self.classname, newid, propname), |
|
2eae5848912d
always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents:
2887
diff
changeset
|
3204 value) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3525
diff
changeset
|
3205 d[propname] = value |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3206 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3207 # get a new id if necessary |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
3208 if newid is None: |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3209 newid = self.db.newid(self.classname) |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
3210 |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3211 activeid = None |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3212 has_node = False |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3213 |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3214 # use the arg for __retired__ to cope with any odd database type |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3215 # conversion (hello, sqlite) |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3216 retired_sql = 'update _%s set __retired__=%s where id=%s'%( |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3217 self.classname, self.db.arg, self.db.arg) |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3218 |
|
2217
98d3bf8ffb19
store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents:
2196
diff
changeset
|
3219 # insert new node or update existing? |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3220 # if integrity error raised try to recover |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3221 try: |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3222 has_node = self.hasnode(newid) |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3223 if not has_node: |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3224 self.db.addnode(self.classname, newid, d) # insert |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3225 else: |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3226 self.db.setnode(self.classname, newid, d) # update |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
3227 self.db.checkpoint_data() |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3228 # Blech, different db's return different exceptions |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3229 # so I can't list them here as some might not be defined |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3230 # on a given system. So capture all exceptions from the |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3231 # code above and try to correct it. If it's correctable its |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3232 # some form of Uniqueness Failure/Integrity Error otherwise |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3233 # undo the fixup and pass on the error. |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
3234 except Exception as e: # nosec |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3235 logger.info('Attempting to handle import exception ' |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3236 'for id %s: %s' % (newid,e)) |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3237 |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3238 keyname = self.db.user.getkey() |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3239 if has_node or not keyname: # Not an integrity error |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3240 raise |
|
6433
c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6431
diff
changeset
|
3241 self.db.restore_connection_on_error() |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3242 activeid = self.db.user.lookup(d[keyname]) |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3243 self.db.sql(retired_sql, (-1, activeid)) # clear the active node |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3244 # this can only happen on an addnode, so retry |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3245 try: |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3246 # if this raises an error, let it propagate upward |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3247 self.db.addnode(self.classname, newid, d) # insert |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3248 except Exception: |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3249 # undo the database change |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3250 self.db.sql(retired_sql, (0, activeid)) # clear the active node |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3251 raise # propagate |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3252 logger.info('Successfully handled import exception ' |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3253 'for id %s which conflicted with %s' % ( |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3254 newid, activeid)) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3255 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3256 # retire? |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3257 if retire: |
|
6431
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3258 self.db.sql(retired_sql, (newid, newid)) |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3259 |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3260 if activeid: |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3261 # unretire the active node |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3262 self.db.sql(retired_sql, ('0', activeid)) |
|
ada1edcc9132
issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents:
6415
diff
changeset
|
3263 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3264 return newid |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3265 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3266 def export_journals(self): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3267 """Export a class's journal - generate a list of lists of |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3268 CSV-able data: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3269 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3270 nodeid, date, user, action, params |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3271 |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3272 No heading here - the columns are fixed. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3273 """ |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3274 properties = self.getprops() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3275 r = [] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3276 for nodeid in self.getnodeids(): |
|
5232
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
3277 for nodeid, date, user, action, params in self.history(nodeid, |
|
462b0f76fce8
issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents:
5175
diff
changeset
|
3278 enforceperm=False, skipquiet=False): |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3279 date = date.get_tuple() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3280 if action == 'set': |
|
3365
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3281 export_data = {} |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
3282 for propname, value in params.items(): |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3283 if propname not in properties: |
|
3365
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3284 # property no longer in the schema |
|
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3285 continue |
|
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3286 |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3287 prop = properties[propname] |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3288 # make sure the params are eval()'able |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3289 if value is None: |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3290 pass |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3291 elif isinstance(prop, Date): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3292 value = value.get_tuple() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3293 elif isinstance(prop, Interval): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3294 value = value.get_tuple() |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3295 elif isinstance(prop, Password): |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3296 value = str(value) |
|
3365
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3297 export_data[propname] = value |
|
e2d65f6c8d83
handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents:
3328
diff
changeset
|
3298 params = export_data |
|
3933
9095a4da67f9
Handle export and import of old trackers that have data attached to...
Richard Jones <richard@users.sourceforge.net>
parents:
3925
diff
changeset
|
3299 elif action == 'create' and params: |
|
9095a4da67f9
Handle export and import of old trackers that have data attached to...
Richard Jones <richard@users.sourceforge.net>
parents:
3925
diff
changeset
|
3300 # old tracker with data stored in the create! |
|
9095a4da67f9
Handle export and import of old trackers that have data attached to...
Richard Jones <richard@users.sourceforge.net>
parents:
3925
diff
changeset
|
3301 params = {} |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3302 l = [nodeid, date, user, action, params] |
|
5525
bb7865241f8a
Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents:
5492
diff
changeset
|
3303 r.append(list(map(repr_export, l))) |
|
2175
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3304 return r |
|
723098a10677
Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents:
2166
diff
changeset
|
3305 |
|
2597
c86b2179085b
fix journal export of files to remove content from CSV files
Richard Jones <richard@users.sourceforge.net>
parents:
2586
diff
changeset
|
3306 class FileClass(hyperdb.FileClass, Class): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3307 """This class defines a large chunk of data. To support this, it has a |
| 1165 | 3308 mandatory String property "content" which is typically saved off |
| 3309 externally to the hyperdb. | |
| 3310 | |
| 3311 The default MIME type of this data is defined by the | |
| 3312 "default_mime_type" class attribute, which may be overridden by each | |
| 3313 node if the class defines a "type" String property. | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3314 """ |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3315 def __init__(self, db, classname, **properties): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3316 """The newly-created class automatically includes the "content" |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3317 and "type" properties. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3318 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3319 if 'content' not in properties: |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3320 properties['content'] = hyperdb.String(indexme='yes') |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3321 if 'type' not in properties: |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3322 properties['type'] = hyperdb.String() |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3323 Class.__init__(self, db, classname, **properties) |
| 1165 | 3324 |
| 3325 def create(self, **propvalues): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3326 """ snaffle the file propvalue and store in a file |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3327 """ |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3328 # we need to fire the auditors now, or the content property won't |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3329 # be in propvalues for the auditors to play with |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3330 self.fireAuditors('create', None, propvalues) |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3331 |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3332 # now remove the content property so it's not stored in the db |
| 1165 | 3333 content = propvalues['content'] |
| 3334 del propvalues['content'] | |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3335 |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3336 # do the database create |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3337 newid = self.create_inner(**propvalues) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3338 |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3339 # figure the mime type |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3340 mime_type = propvalues.get('type', self.default_mime_type) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3341 |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3342 # and index! |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3343 if self.properties['content'].indexme: |
|
5544
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3344 index_content = content |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3345 if bytes != str and isinstance(content, bytes): |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3346 index_content = content.decode('utf-8', errors='ignore') |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3347 self.db.indexer.add_text((self.classname, newid, 'content'), |
|
5544
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3348 index_content, mime_type) |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3349 |
|
3979
954971d612ee
Fire reactors after file storage is all done (patch [SF#2001243])
Richard Jones <richard@users.sourceforge.net>
parents:
3977
diff
changeset
|
3350 # store off the content as a file |
|
5492
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3351 self.db.storefile(self.classname, newid, None, bs2b(content)) |
|
3979
954971d612ee
Fire reactors after file storage is all done (patch [SF#2001243])
Richard Jones <richard@users.sourceforge.net>
parents:
3977
diff
changeset
|
3352 |
|
1431
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3353 # fire reactors |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3354 self.fireReactors('create', newid, None) |
|
c70068162e64
Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents:
1417
diff
changeset
|
3355 |
| 1165 | 3356 return newid |
| 3357 | |
| 3358 def get(self, nodeid, propname, default=_marker, cache=1): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3359 """ Trap the content propname and get it from the file |
|
1780
d2801a2b0a77
Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents:
1751
diff
changeset
|
3360 |
|
d2801a2b0a77
Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents:
1751
diff
changeset
|
3361 'cache' exists for backwards compatibility, and is not used. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3362 """ |
| 1165 | 3363 poss_msg = 'Possibly a access right configuration problem.' |
| 3364 if propname == 'content': | |
| 3365 try: | |
|
5492
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3366 return b2s(self.db.getfile(self.classname, nodeid, None)) |
|
5248
198b6e810c67
Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents:
5236
diff
changeset
|
3367 except IOError as strerror: |
| 1165 | 3368 # BUG: by catching this we donot see an error in the log. |
| 3369 return 'ERROR reading file: %s%s\n%s\n%s'%( | |
| 3370 self.classname, nodeid, poss_msg, strerror) | |
|
5725
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3371 except UnicodeDecodeError as e: |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3372 # if content is not text (e.g. jpeg file) we get |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3373 # unicode error trying to convert to string in python 3. |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3374 # trap it and supply an error message. Include md5sum |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3375 # of content as this string is included in the etag |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3376 # calculation of the object. |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3377 return ('%s%s is not text, retrieve using ' |
|
6923225fd781
Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents:
5544
diff
changeset
|
3378 'binary_content property. mdsum: %s')%(self.classname, |
| 6002 | 3379 nodeid, md5(self.db.getfile(self.classname, nodeid, None)).hexdigest()) # nosec - bandit md5 use ok |
|
5492
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3380 elif propname == 'binary_content': |
|
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3381 return self.db.getfile(self.classname, nodeid, None) |
|
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3382 |
|
2692
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
3383 if default is not _marker: |
|
1780
d2801a2b0a77
Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents:
1751
diff
changeset
|
3384 return Class.get(self, nodeid, propname, default) |
| 1165 | 3385 else: |
|
1780
d2801a2b0a77
Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents:
1751
diff
changeset
|
3386 return Class.get(self, nodeid, propname) |
| 1165 | 3387 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3388 def set(self, itemid, **propvalues): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3389 """ Snarf the "content" propvalue and update it in a file |
|
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3390 """ |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3391 self.fireAuditors('set', itemid, propvalues) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3392 oldvalues = copy.deepcopy(self.db.getnode(self.classname, itemid)) |
| 1165 | 3393 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3394 # now remove the content property so it's not stored in the db |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3395 content = None |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3396 if 'content' in propvalues: |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3397 content = propvalues['content'] |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3398 del propvalues['content'] |
| 1165 | 3399 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3400 # do the database create |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3401 propvalues = self.set_inner(itemid, **propvalues) |
| 1165 | 3402 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3403 # do content? |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3404 if content: |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3405 # store and possibly index |
|
5492
6b0c542642be
blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5467
diff
changeset
|
3406 self.db.storefile(self.classname, itemid, None, bs2b(content)) |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3407 if self.properties['content'].indexme: |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3408 mime_type = self.get(itemid, 'type', self.default_mime_type) |
|
5544
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3409 index_content = content |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3410 if bytes != str and isinstance(content, bytes): |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3411 index_content = content.decode('utf-8', errors='ignore') |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3412 self.db.indexer.add_text((self.classname, itemid, 'content'), |
|
5544
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3413 index_content, mime_type) |
|
2640
b01eca163779
The "type" parameter is supposed to be optional
Richard Jones <richard@users.sourceforge.net>
parents:
2634
diff
changeset
|
3414 propvalues['content'] = content |
|
b01eca163779
The "type" parameter is supposed to be optional
Richard Jones <richard@users.sourceforge.net>
parents:
2634
diff
changeset
|
3415 |
|
2089
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3416 # fire reactors |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3417 self.fireReactors('set', itemid, oldvalues) |
|
93f03c6714d8
A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
3418 return propvalues |
| 1165 | 3419 |
|
2505
bdd112cf61ba
rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
3420 def index(self, nodeid): |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3421 """ Add (or refresh) the node to search indexes. |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3422 |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3423 Use the content-type property for the content property. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3424 """ |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3425 # find all the String properties that have indexme |
|
5395
23b8e6067f7c
Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5387
diff
changeset
|
3426 for prop, propclass in self.getprops().items(): |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3427 if prop == 'content' and propclass.indexme: |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3428 mime_type = self.get(nodeid, 'type', self.default_mime_type) |
|
5544
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3429 index_content = self.get(nodeid, 'binary_content') |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3430 if bytes != str and isinstance(index_content, bytes): |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3431 index_content = index_content.decode('utf-8', |
|
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3432 errors='ignore') |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3433 self.db.indexer.add_text((self.classname, nodeid, 'content'), |
|
5544
1a0498c1ed90
Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5525
diff
changeset
|
3434 index_content, mime_type) |
|
3601
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3435 elif isinstance(propclass, hyperdb.String) and propclass.indexme: |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3436 # index them under (classname, nodeid, property) |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3437 try: |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3438 value = str(self.get(nodeid, prop)) |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3439 except IndexError: |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3440 # node has been destroyed |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3441 continue |
|
7b25567f0f54
indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents:
3586
diff
changeset
|
3442 self.db.indexer.add_text((self.classname, nodeid, prop), value) |
|
2505
bdd112cf61ba
rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents:
2496
diff
changeset
|
3443 |
| 1165 | 3444 # XXX deviation from spec - was called ItemClass |
| 3445 class IssueClass(Class, roundupdb.IssueClass): | |
| 3446 # Overridden methods: | |
| 3447 def __init__(self, db, classname, **properties): | |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3448 """The newly-created class automatically includes the "messages", |
| 1165 | 3449 "files", "nosy", and "superseder" properties. If the 'properties' |
| 3450 dictionary attempts to specify any of these properties or a | |
|
2077
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
3451 "creation", "creator", "activity" or "actor" property, a ValueError |
|
3e0961d6d44d
Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents:
2076
diff
changeset
|
3452 is raised. |
|
3834
8068eb6c704e
Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents:
3821
diff
changeset
|
3453 """ |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3454 if 'title' not in properties: |
| 1165 | 3455 properties['title'] = hyperdb.String(indexme='yes') |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3456 if 'messages' not in properties: |
| 1165 | 3457 properties['messages'] = hyperdb.Multilink("msg") |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3458 if 'files' not in properties: |
| 1165 | 3459 properties['files'] = hyperdb.Multilink("file") |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3460 if 'nosy' not in properties: |
| 1165 | 3461 # note: journalling is turned off as it really just wastes |
| 3462 # space. this behaviour may be overridden in an instance | |
| 3463 properties['nosy'] = hyperdb.Multilink("user", do_journal="no") | |
|
4359
b9abbdd15259
another module modernised
Richard Jones <richard@users.sourceforge.net>
parents:
4113
diff
changeset
|
3464 if 'superseder' not in properties: |
| 1165 | 3465 properties['superseder'] = hyperdb.Multilink(classname) |
| 3466 Class.__init__(self, db, classname, **properties) | |
| 3467 | |
|
2692
f1c9873496f0
fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
2691
diff
changeset
|
3468 # vim: set et sts=4 sw=4 : |
