Mercurial > p > roundup > code

annotate roundup/backends/rdbms_common.py @ 8302:82a26ea1afdf

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
issue2551376: Fix tracebacks in item templates
author Ralf Schlatterbeck <rsc@runtux.com>
date Wed, 26 Feb 2025 11:24:13 +0100
parents 741ea8a86012
children 24549122f9b1
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3554
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
1 #
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/)
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
3 # This module is free software, and you may redistribute it and/or modify
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
4 # under the same terms as Python, so long as this copyright message and
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
5 # disclaimer are retained in their original form.
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
6 #
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
10 # POSSIBILITY OF SUCH DAMAGE.
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
11 #
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS"
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
17 #
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
18 """ Relational database (SQL) backend common code.
1244
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
19
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
20 Basics:
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
21
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
22 - map roundup classes to relational tables
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
23 - automatically detect schema changes and modify the table schemas
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
24 appropriately (we store the "database version" of the schema in the
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
25 database itself as the only row of the "schema" table)
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
26 - multilinks (which represent a many-to-many relationship) are handled through
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
27 intermediate tables
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
28 - journals are stored adjunct to the per-class tables
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
29 - table names and columns have "_" prepended so the names can't clash with
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
30 restricted names (like "order")
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
31 - retirement is determined by the __retired__ column being > 0
1244
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
32
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
33 Database-specific changes may generally be pushed out to the overridable
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
34 sql_* methods, since everything else should be fairly generic. There's
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
35 probably a bit of work to be done if a database is used that actually
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
36 honors column typing, since the initial databases don't (sqlite stores
1528
96cd422532ef bye bye gadfly - you served your purpose well [SF#701127]
Richard Jones <richard@users.sourceforge.net>
parents: 1523
diff changeset
37 everything as a string.)
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
38
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
39 The schema of the hyperdb being mapped to the database is stored in the
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
40 database itself as a repr()'ed dictionary of information about each Class
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
41 that maps to a table. If that information differs from the hyperdb schema,
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
42 then we update it. We also store in the schema dict a version which
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
43 allows us to upgrade the database schema when necessary. See upgrade_db().
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
44
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
45 To force a unqiueness constraint on the key properties we put the item
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
46 id into the __retired__ column duing retirement (so it's 0 for "active"
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
47 items) and place a unqiueness constraint on key + __retired__. This is
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
48 particularly important for the users class where multiple users may
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
49 try to have the same username, with potentially many retired users with
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
50 the same name.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
51 """
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1988
diff changeset
52 __docformat__ = 'restructuredtext'
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
53
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
54 # standard python modules
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
55 import copy
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
56 import datetime
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
57 import logging
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
58 import os
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
59 import re
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
60 import time
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
61
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
62 from hashlib import md5
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
63
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
64 # roundup modules
3544
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
65 from roundup import hyperdb, date, password, roundupdb, security, support
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
66 from roundup.anypy.strings import b2s, bs2b, us2s, repr_export, eval_import
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
67 from roundup.backends.blobfiles import FileStorage
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5086
diff changeset
68 from roundup.backends.indexer_common import get_indexer
6915
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
69 from roundup.backends.indexer_common import Indexer as CommonIndexer
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
70 from roundup.backends.sessions_rdbms import Sessions, OneTimeKeys
1499
8ee69708da0c added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1496
diff changeset
71 from roundup.date import Range
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
72 from roundup.hyperdb import String, Password, Date, Interval, Link, \
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
73 Multilink, DatabaseError, Boolean, Number, Integer
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
74 from roundup.i18n import _
8241
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 7936
diff changeset
75 from roundup.mlink_expr import compile_expression, ExpressionError
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
76
2692
f1c9873496f0 fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2691
diff changeset
77 # dummy value meaning "argument not passed"
f1c9873496f0 fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2691
diff changeset
78 _marker = []
f1c9873496f0 fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2691
diff changeset
79
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
80 # python 3 doesn't have a unicode type
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
81 try:
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
82 unicode # noqa: F821
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
83 except NameError:
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
84 unicode = str
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
85
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
86
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
87 def _num_cvt(num):
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
88 num = str(num)
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
89 try:
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
90 return int(num)
6002
3175bb92ca28 Cleanups for bandit
John Rouillard <rouilj@ieee.org>
parents: 5871
diff changeset
91 except ValueError:
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
92 return float(num)
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
93
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
94
2472
f41539b3c486 fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents: 2456
diff changeset
95 def _bool_cvt(value):
f41539b3c486 fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents: 2456
diff changeset
96 if value in ('TRUE', 'FALSE'):
f41539b3c486 fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents: 2456
diff changeset
97 return {'TRUE': 1, 'FALSE': 0}[value]
f41539b3c486 fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents: 2456
diff changeset
98 # assume it's a number returned from the db API
f41539b3c486 fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents: 2456
diff changeset
99 return int(value)
f41539b3c486 fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents: 2456
diff changeset
100
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
101
4473
fccf7e09af0c - optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4472
diff changeset
102 def date_to_hyperdb_value(d):
fccf7e09af0c - optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4472
diff changeset
103 """ convert date d to a roundup date """
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
104 if isinstance(d, datetime.datetime):
4473
fccf7e09af0c - optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4472
diff changeset
105 return date.Date(d)
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
106 return date.Date(str(d).replace(' ', '.'))
4473
fccf7e09af0c - optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4472
diff changeset
107
fccf7e09af0c - optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4472
diff changeset
108
2634
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
109 def connection_dict(config, dbnamestr=None):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
110 """ Used by Postgresql and MySQL to detemine the keyword args for
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
111 opening the database connection."""
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
112 d = {}
2634
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
113 if dbnamestr:
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
114 d[dbnamestr] = config.RDBMS_NAME
7696
4af0d235b570 feat(db): support using postgresql service connection file
John Rouillard <rouilj@ieee.org>
parents: 7665
diff changeset
115 for name in ('host', 'port', 'password', 'user', 'service',
4af0d235b570 feat(db): support using postgresql service connection file
John Rouillard <rouilj@ieee.org>
parents: 7665
diff changeset
116 'read_default_group', 'read_default_file'):
2634
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
117 cvar = 'RDBMS_'+name.upper()
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
118 if config[cvar] is not None:
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
119 d[name] = config[cvar]
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
120 return d
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
121
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
122
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
123 class IdListOptimizer:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
124 """ To prevent flooding the SQL parser of the underlaying
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
125 db engine with "x IN (1, 2, 3, ..., <large number>)" collapses
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
126 these cases to "x BETWEEN 1 AND <large number>".
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
127 """
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
128
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
129 def __init__(self):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
130 self.ranges = []
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
131 self.singles = []
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
132
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
133 def append(self, nid):
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
134 """ Invariant: nid are ordered ascending """
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
135 if self.ranges:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
136 last = self.ranges[-1]
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
137 if last[1] == nid-1:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
138 last[1] = nid
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
139 return
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
140 if self.singles:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
141 last = self.singles[-1]
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
142 if last == nid-1:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
143 self.singles.pop()
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
144 self.ranges.append([last, nid])
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
145 return
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
146 self.singles.append(nid)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
147
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
148 def where(self, field, placeholder):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
149 ranges = self.ranges
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
150 singles = self.singles
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
151
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
152 if not singles and not ranges: return "(1=0)", [] # noqa: E701
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
153
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
154 if ranges:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
155 between = '%s BETWEEN %s AND %s' % (
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
156 field, placeholder, placeholder)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
157 stmnt = [between] * len(ranges)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
158 else:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
159 stmnt = []
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
160 if singles:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
161 stmnt.append('%s in (%s)' % (
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
162 field, ','.join([placeholder]*len(singles))))
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
163
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
164 return '(%s)' % ' OR '.join(stmnt), sum(ranges, []) + singles
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
165
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
166 def __str__(self):
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
167 return "ranges: %r / singles: %r" % (self.ranges, self.singles)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
168
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
169
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
170 class Database(FileStorage, hyperdb.Database, roundupdb.Database):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
171 """ Wrapper around an SQL database that presents a hyperdb interface.
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
172
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
173 - some functionality is specific to the actual SQL database, hence
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
174 the sql_* methods that are NotImplemented
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
175 - we keep a cache of the latest N row fetches (where N is
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
176 configurable).
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
177 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
178 def __init__(self, config, journaltag=None):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
179 """ Open the database and load the schema from it.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
180 """
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3872
diff changeset
181 FileStorage.__init__(self, config.UMASK)
6658
408fd477761f Add i18n object to roundupdb.Database
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6599
diff changeset
182 roundupdb.Database.__init__(self)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
183 self.config, self.journaltag = config, journaltag
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
184 self.dir = config.DATABASE
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
185 self.classes = {}
6915
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
186 # Assign the indexer base class here. During schema
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
187 # generation in open_connection, the min/max size for FTS
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
188 # tokens is used when creating the database tables for
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
189 # indexer=native full text search. These tables are always
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
190 # created as part of the schema so that the admin can choose
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
191 # indexer=native at some later date and "things will just
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
192 # work" (TM).
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
193 #
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
194 # We would like to use get_indexer() to return the real
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
195 # indexer class. However indexer=native-fts for postgres
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
196 # requires a database connection (conn) to be defined when
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
197 # calling get_indexer. The call to open_connection creates the
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
198 # conn but also creates the schema if it is missing.
6918
cb2ed1e8c852 Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents: 6915
diff changeset
199 if self.config['INDEXER'] != 'native-fts':
cb2ed1e8c852 Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents: 6915
diff changeset
200 self.indexer = get_indexer(config, self)
cb2ed1e8c852 Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents: 6915
diff changeset
201 else:
cb2ed1e8c852 Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents: 6915
diff changeset
202 self.indexer = CommonIndexer(self)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
203 self.security = security.Security(self)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
204
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
205 # additional transaction support for external files and the like
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
206 self.transactions = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
207
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
208 # keep a cache of the N most recently retrieved rows of any kind
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
209 # (classname, nodeid) = row
4074
e039f3cbbb96 Make RDBMS cache-size configurable.
Stefan Seefeld <stefan@seefeld.name>
parents: 4062
diff changeset
210 self.cache_size = config.RDBMS_CACHE_SIZE
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
211 self.clearCache()
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
212 self.stats = {'cache_hits': 0, 'cache_misses': 0, 'get_items': 0,
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
213 'filtering': 0}
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
214
5041
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
215 # make sure the database directory exists
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
216 if not os.path.isdir(self.config.DATABASE):
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
217 os.makedirs(self.config.DATABASE)
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
218
1333
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1304
diff changeset
219 # database lock
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1304
diff changeset
220 self.lockfile = None
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1304
diff changeset
221
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
222 # Uppercase to not collide with Class names
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
223 self.Session = None
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
224 self.Otk = None
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
225
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
226 # open a connection to the database, creating the "conn" attribute
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2081
diff changeset
227 self.open_connection()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
228
6915
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
229 # If indexer is native-fts, conn to db must be available.
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
230 # so we set the real self.indexer value here, after db is open.
6918
cb2ed1e8c852 Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents: 6915
diff changeset
231 if self.config['INDEXER'] == 'native-fts':
cb2ed1e8c852 Change method for settin indexer; have test_livetest for pg cleanup
John Rouillard <rouilj@ieee.org>
parents: 6915
diff changeset
232 self.indexer = get_indexer(config, self)
6915
9ff091537f43 postgresql native-fts; more indexer tests
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
233
1181
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
234 def clearCache(self):
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
235 self.cache = {}
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
236 self.cache_lru = []
4652
dfbc0cfa9811 Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4517
diff changeset
237 # upcall is necessary!
dfbc0cfa9811 Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4517
diff changeset
238 roundupdb.Database.clearCache(self)
1181
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
239
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2081
diff changeset
240 def getSessionManager(self):
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
241 if not self.Session:
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
242 self.Session = Sessions(self)
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
243 return self.Session
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2081
diff changeset
244
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2081
diff changeset
245 def getOTKManager(self):
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
246 if not self.Otk:
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
247 self.Otk = OneTimeKeys(self)
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
248 return self.Otk
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2081
diff changeset
249
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2081
diff changeset
250 def open_connection(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
251 """ Open a connection to the database, creating it if necessary.
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
252
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
253 Must call self.load_dbschema()
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
254 """
5236
0a20e1965f4b Pylint checks bark about:
John Rouillard <rouilj@ieee.org>
parents: 5232
diff changeset
255 raise NotImplementedError
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
256
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
257 def sql(self, sql, args=None, cursor=None):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
258 """ Execute the sql with the optional args.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
259 """
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
260 self.log_debug('SQL %r %r' % (sql, args))
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
261 if not cursor:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
262 cursor = self.cursor
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
263 if args:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
264 cursor.execute(sql, args)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
265 else:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
266 cursor.execute(sql)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
267
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
268 def sql_fetchone(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
269 """ Fetch a single row. If there's nothing to fetch, return None.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
270 """
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
271 return self.cursor.fetchone()
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
272
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
273 def sql_fetchall(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
274 """ Fetch all rows. If there's nothing to fetch, return [].
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
275 """
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
276 return self.cursor.fetchall()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
277
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
278 def sql_fetchiter(self):
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
279 """ Fetch all row as a generator
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
280 """
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
281 while True:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
282 row = self.cursor.fetchone()
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
283 if not row: break # noqa: E701
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
284 yield row
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
285
4849
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
286 def search_stringquote(self, value):
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
287 """ Quote a search string to escape magic search characters
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
288 '%' and '_', also need to quote '\' (first)
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
289 Then put '%' around resulting string for LIKE (or ILIKE) operator
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
290 """
4849
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
291 v = value.replace('\\', '\\\\')
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
292 v = v.replace('%', '\\%')
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
293 v = v.replace('_', '\\_')
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
294 return '%' + v + '%'
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
295
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
296 def init_dbschema(self):
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
297 self.database_schema = {
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
298 'version': self.current_db_version,
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
299 'tables': {}
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
300 }
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
301
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
302 def load_dbschema(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
303 """ Load the schema definition that the database currently implements
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
304 """
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
305 self.cursor.execute('select schema from schema')
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
306 schema = self.cursor.fetchone()
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
307 if schema:
6002
3175bb92ca28 Cleanups for bandit
John Rouillard <rouilj@ieee.org>
parents: 5871
diff changeset
308 # bandit - schema is trusted
3175bb92ca28 Cleanups for bandit
John Rouillard <rouilj@ieee.org>
parents: 5871
diff changeset
309 self.database_schema = eval(schema[0]) # nosec
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
310 else:
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
311 self.database_schema = {}
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
312
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
313 def save_dbschema(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
314 """ Save the schema definition that the database currently implements
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
315 """
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
316 s = repr(self.database_schema)
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
317 self.sql('delete from schema')
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
318 self.sql('insert into schema values (%s)' % self.arg, (s,))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
319
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
320 def post_init(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
321 """ Called once the schema initialisation has finished.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
322
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
323 We should now confirm that the schema defined by our "classes"
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
324 attribute actually matches the schema in the database.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
325 """
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
326 super(Database, self).post_init()
4774
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
327
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
328 # upgrade the database for column type changes, new internal
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
329 # tables, etc.
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
330 save = self.upgrade_db()
3553
Richard Jones <richard@users.sourceforge.net>
parents: 3544
diff changeset
331
Richard Jones <richard@users.sourceforge.net>
parents: 3544
diff changeset
332 # handle changes in the schema
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
333 tables = self.database_schema['tables']
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
334 for classname, spec in self.classes.items():
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
335 if classname in tables:
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
336 dbspec = tables[classname]
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
337 if self.update_class(spec, dbspec):
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
338 tables[classname] = spec.schema()
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
339 save = 1
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
340 else:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
341 self.create_class(spec)
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
342 tables[classname] = spec.schema()
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
343 save = 1
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
344
7038
f524ddc27af8 replace for a,b in x.items() with for a in x.keys()
John Rouillard <rouilj@ieee.org>
parents: 7007
diff changeset
345 for classname in list(tables.keys()):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
346 if classname not in self.classes:
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
347 self.drop_class(classname, tables[classname])
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
348 del tables[classname]
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
349 save = 1
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
350
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
351 # update the database version of the schema
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
352 if save:
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
353 self.save_dbschema()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
354
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
355 # reindex the db if necessary
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
356 if self.indexer.should_reindex():
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
357 self.reindex()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
358
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
359 # commit
2093
3f6024ab2c7a That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
360 self.sql_commit()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
361
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
362 # update this number when we need to make changes to the SQL structure
6588
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6433
diff changeset
363 # of the backend database
6806
bdd28b244839 - issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
364 current_db_version = 8
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
365 db_version_updated = False
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
366
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
367 def upgrade_db(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
368 """ Update the SQL database to reflect changes in the backend code.
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
369
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
370 Return boolean whether we need to save the schema.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
371 """
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
372 version = self.database_schema.get('version', 1)
3993
1eba65a4b0aa Don't run old code on newer database
Richard Jones <richard@users.sourceforge.net>
parents: 3979
diff changeset
373 if version > self.current_db_version:
1eba65a4b0aa Don't run old code on newer database
Richard Jones <richard@users.sourceforge.net>
parents: 3979
diff changeset
374 raise DatabaseError('attempting to run rev %d DATABASE with rev '
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
375 '%d CODE!' % (version,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
376 self.current_db_version))
2081
fb4bf55b94d7 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
377 if version == self.current_db_version:
fb4bf55b94d7 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
378 # nothing to do
fb4bf55b94d7 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
379 return 0
fb4bf55b94d7 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
380
2429
e28a5d5d95a7 better upgrading
Richard Jones <richard@users.sourceforge.net>
parents: 2424
diff changeset
381 if version < 2:
4062
6ea417bafd9c Fix typo.
Stefan Seefeld <stefan@seefeld.name>
parents: 4061
diff changeset
382 self.log_info('upgrade to version 2')
2100
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
383 # change the schema structure
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
384 self.database_schema = {'tables': self.database_schema}
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
385
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
386 # version 1 didn't have the actor column (note that in
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
387 # MySQL this will also transition the tables to typed columns)
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
388 self.add_new_columns_v2()
2100
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
389
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
390 # version 1 doesn't have the OTK, session and indexing in the
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
391 # database
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
392 self.create_version_2_tables()
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
393
2429
e28a5d5d95a7 better upgrading
Richard Jones <richard@users.sourceforge.net>
parents: 2424
diff changeset
394 if version < 3:
4062
6ea417bafd9c Fix typo.
Stefan Seefeld <stefan@seefeld.name>
parents: 4061
diff changeset
395 self.log_info('upgrade to version 3')
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
396 self.fix_version_2_tables()
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
397
2721
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2720
diff changeset
398 if version < 4:
4774
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
399 self.log_info('upgrade to version 4')
2721
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2720
diff changeset
400 self.fix_version_3_tables()
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2720
diff changeset
401
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
402 if version < 5:
4774
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
403 self.log_info('upgrade to version 5')
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
404 self.fix_version_4_tables()
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
405
6433
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
406 if version < 6:
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
407 self.log_info('upgrade to version 6')
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
408 self.fix_version_5_tables()
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
409
6588
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6433
diff changeset
410 if version < 7:
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6433
diff changeset
411 self.log_info('upgrade to version 7')
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6433
diff changeset
412 self.fix_version_6_tables()
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6433
diff changeset
413
6806
bdd28b244839 - issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
414 if version < 8:
bdd28b244839 - issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
415 self.log_info('upgrade to version 8')
bdd28b244839 - issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
416 self.fix_version_7_tables()
bdd28b244839 - issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
417
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
418 self.database_schema['version'] = self.current_db_version
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
419 self.db_version_updated = True
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
420 return 1
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
421
6588
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6433
diff changeset
422 def fix_version_2_tables(self):
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6433
diff changeset
423 # Default (used by sqlite): NOOP
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6433
diff changeset
424 pass
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6433
diff changeset
425
2721
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2720
diff changeset
426 def fix_version_3_tables(self):
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2720
diff changeset
427 # drop the shorter VARCHAR OTK column and add a new TEXT one
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2720
diff changeset
428 for name in ('otk', 'session'):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
429 self.sql('DELETE FROM %ss' % name)
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
430 self.sql('ALTER TABLE %ss DROP %s_value' % (name, name))
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
431 self.sql('ALTER TABLE %ss ADD %s_value TEXT' % (name, name))
2721
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2720
diff changeset
432
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
433 def fix_version_4_tables(self):
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
434 # note this is an explicit call now
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
435 c = self.cursor
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
436 for cn, klass in self.classes.items():
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
437 c.execute('select id from _%s where __retired__<>0' % (cn,))
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
438 for (id,) in c.fetchall():
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
439 c.execute('update _%s set __retired__=%s where id=%s' % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
440 cn,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
441 self.arg, self.arg), (id, id))
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
442
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
443 if klass.key:
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
444 self.add_class_key_required_unique_constraint(cn, klass.key)
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
445
6433
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
446 def fix_version_5_tables(self):
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
447 # Default (used by sqlite, postgres): NOOP
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
448 # mysql overrides this because it is missing
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
449 # _<class>_key_retired_idx index used to make
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
450 # sure that the key is unique if it was created
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
451 # as version 5.
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
452 pass
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
453
6588
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6433
diff changeset
454 def fix_version_6_tables(self):
6599
39189dd94f2c issue2551189 - increase size of words in full text index.
John Rouillard <rouilj@ieee.org>
parents: 6588
diff changeset
455 # Default (used by nobody): NOOP
39189dd94f2c issue2551189 - increase size of words in full text index.
John Rouillard <rouilj@ieee.org>
parents: 6588
diff changeset
456 # Each backend mysql, postgres, sqlite overrides this
39189dd94f2c issue2551189 - increase size of words in full text index.
John Rouillard <rouilj@ieee.org>
parents: 6588
diff changeset
457 # You would think ALTER commands would be the same but nooo.
6588
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6433
diff changeset
458 pass
91ab3e0ffcd0 Summary: Add test cases for sqlite fts
John Rouillard <rouilj@ieee.org>
parents: 6433
diff changeset
459
6806
bdd28b244839 - issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
460 def fix_version_7_tables(self):
bdd28b244839 - issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
461 # Default (used by sqlite): NOOP
bdd28b244839 - issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
462 # Each backend mysql, postgres overrides this
bdd28b244839 - issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
463 # You would think ALTER commands would be the same but nooo.
bdd28b244839 - issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
464 pass
bdd28b244839 - issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
465
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
466 def _convert_journal_tables(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
467 """Get current journal table contents, drop the table and re-create"""
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
468 c = self.cursor
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
469 cols = ','.join('nodeid date tag action params'.split())
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
470 for klass in self.classes.values():
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
471 # slurp and drop
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
472 sql = 'select %s from %s__journal order by date' % \
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
473 (cols, klass.classname)
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
474 c.execute(sql)
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
475 contents = c.fetchall()
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
476 self.drop_journal_table_indexes(klass.classname)
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
477 c.execute('drop table %s__journal' % klass.classname)
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
478
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
479 # re-create and re-populate
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
480 self.create_journal_table(klass)
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
481 a = self.arg
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
482 sql = 'insert into %s__journal (%s) values (%s,%s,%s,%s,%s)' % (
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
483 klass.classname, cols, a, a, a, a, a)
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
484 for row in contents:
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
485 # no data conversion needed
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
486 self.cursor.execute(sql, row)
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
487
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
488 def _convert_string_properties(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
489 """Get current Class tables that contain String properties, and
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
490 convert the VARCHAR columns to TEXT"""
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
491 c = self.cursor
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
492 for klass in self.classes.values():
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
493 # slurp and drop
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
494 cols, mls = self.determine_columns(list(klass.properties.items()))
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
495 scols = ','.join([i[0] for i in cols])
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
496 sql = 'select id,%s from _%s' % (scols, klass.classname)
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
497 c.execute(sql)
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
498 contents = c.fetchall()
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
499 self.drop_class_table_indexes(klass.classname, klass.getkey())
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
500 c.execute('drop table _%s' % klass.classname)
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
501
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
502 # re-create and re-populate
2424
74474ec41050 argh! backwards compat
Richard Jones <richard@users.sourceforge.net>
parents: 2416
diff changeset
503 self.create_class_table(klass, create_sequence=0)
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
504 a = ','.join([self.arg for i in range(len(cols)+1)])
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
505 sql = 'insert into _%s (id,%s) values (%s)' % (klass.classname,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
506 scols, a)
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
507 for row in contents:
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
508 l = []
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
509 for entry in row:
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
510 # mysql will already be a string - psql needs "help"
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
511 if entry is not None and not isinstance(entry, type('')):
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
512 entry = str(entry)
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
513 l.append(entry)
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
514 self.cursor.execute(sql, l)
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
515
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1838
diff changeset
516 def refresh_database(self):
1951
767ff2a03eee more unit tests to improve coverage (up from 85% to 88% for anydbm! :)
Richard Jones <richard@users.sourceforge.net>
parents: 1926
diff changeset
517 self.post_init()
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1838
diff changeset
518
3544
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
519 def reindex(self, classname=None, show_progress=False):
2650
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
520 if classname:
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
521 classes = [self.getclass(classname)]
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
522 else:
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
523 classes = list(self.classes.values())
2650
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
524 for klass in classes:
3544
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
525 if show_progress:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
526 for nodeid in support.Progress('Reindex %s' % klass.classname,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
527 klass.list()):
3544
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
528 klass.index(nodeid)
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
529 else:
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
530 for nodeid in klass.list():
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
531 klass.index(nodeid)
3295
a615cc230160 added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents: 3239
diff changeset
532 self.indexer.save_index()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
533
6433
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
534 def checkpoint_data(self):
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
535 """Call if you need to commit the state of the database
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
536 so you can try to fix the error rather than rolling back
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
537
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
538 Needed for postgres when importing data.
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
539 """
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
540 pass
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
541
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
542 def restore_connection_on_error(self):
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
543 """on a database error/exception recover the db connection
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
544 if left in an unusable state (e.g. postgres requires
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
545 a rollback).
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
546 """
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
547 pass
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
548
5175
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
549 # Used here in the generic backend to determine if the database
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
550 # supports 'DOUBLE PRECISION' for floating point numbers.
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
551 implements_double_precision = True
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
552
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
553 hyperdb_to_sql_datatypes = {
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
554 hyperdb.String : 'TEXT', # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
555 hyperdb.Date : 'TIMESTAMP', # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
556 hyperdb.Link : 'INTEGER', # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
557 hyperdb.Interval : 'VARCHAR(255)', # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
558 hyperdb.Password : 'VARCHAR(255)', # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
559 hyperdb.Boolean : 'BOOLEAN', # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
560 hyperdb.Number : 'REAL', # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
561 hyperdb.Integer : 'INTEGER', # noqa: E203
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
562 }
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
563
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
564 def hyperdb_to_sql_datatype(self, propclass, prop=None):
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
565
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
566 datatype = self.hyperdb_to_sql_datatypes.get(propclass)
5175
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
567 if self.implements_double_precision and prop and \
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
568 isinstance(prop, Number) and prop.use_double:
5175
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
569 datatype = 'DOUBLE PRECISION'
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
570 if datatype:
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
571 return datatype
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
572
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
573 for k, v in self.hyperdb_to_sql_datatypes.items():
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
574 if issubclass(propclass, k):
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
575 return v
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
576
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
577 raise ValueError('%r is not a hyperdb property class' % propclass)
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
578
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
579 def determine_columns(self, properties):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
580 """ Figure the column names and multilink properties from the spec
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
581
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
582 "properties" is a list of (name, prop) where prop may be an
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
583 instance of a hyperdb "type" _or_ a string repr of that type.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
584 """
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
585 cols = [
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
586 ('_actor', self.hyperdb_to_sql_datatype(hyperdb.Link)),
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
587 ('_activity', self.hyperdb_to_sql_datatype(hyperdb.Date)),
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
588 ('_creator', self.hyperdb_to_sql_datatype(hyperdb.Link)),
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
589 ('_creation', self.hyperdb_to_sql_datatype(hyperdb.Date)),
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
590 ]
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
591 mls = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
592 # add the multilinks separately
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
593 for col, prop in properties:
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
594 # Computed props are not in the db
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
595 if prop.computed:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
596 continue
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
597
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
598 if isinstance(prop, Multilink):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
599 mls.append(col)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
600 continue
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
601
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
602 if isinstance(prop, type('')):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
603 raise ValueError("string property spec!")
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
604 # and prop.find('Multilink') != -1:
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
605 # mls.append(col)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
606
5175
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
607 datatype = self.hyperdb_to_sql_datatype(prop.__class__, prop)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
608 cols.append(('_'+col, datatype))
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
609
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
610 # Intervals stored as two columns
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
611 if isinstance(prop, Interval):
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
612 cols.append(('__'+col+'_int__', 'BIGINT'))
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
613
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
614 cols.sort()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
615 return cols, mls
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
616
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1838
diff changeset
617 def update_class(self, spec, old_spec, force=0):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
618 """ Determine the differences between the current spec and the
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1838
diff changeset
619 database version of the spec, and update where necessary.
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
620
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1838
diff changeset
621 If 'force' is true, update the database anyway.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
622 """
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
623 new_spec = spec.schema()
1515
a516bbb9896b fixed rdbms table update detection logic [SF#703297]
Richard Jones <richard@users.sourceforge.net>
parents: 1500
diff changeset
624 new_spec[1].sort()
a516bbb9896b fixed rdbms table update detection logic [SF#703297]
Richard Jones <richard@users.sourceforge.net>
parents: 1500
diff changeset
625 old_spec[1].sort()
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1838
diff changeset
626 if not force and new_spec == old_spec:
1479
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
627 # no changes
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
628 return 0
1479
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
629
4474
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
630 if not self.config.RDBMS_ALLOW_ALTER:
6133
e7cb0147e6fe Make format strings use named placeholders rather than %s/%r
John Rouillard <rouilj@ieee.org>
parents: 6118
diff changeset
631 raise DatabaseError(_(
7750
216662fbaaee fix(i18n): fix incorrect lookup of some translations
John Rouillard <rouilj@ieee.org>
parents: 7696
diff changeset
632 'ALTER operation disallowed: %(old)r -> %(new)r.') % {
216662fbaaee fix(i18n): fix incorrect lookup of some translations
John Rouillard <rouilj@ieee.org>
parents: 7696
diff changeset
633 'old': old_spec, 'new': new_spec})
4474
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
634
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
635 logger = logging.getLogger('roundup.hyperdb.backend')
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
636 logger.info('update_class %s' % spec.classname)
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
637
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
638 logger.debug('old_spec %r' % (old_spec,))
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
639 logger.debug('new_spec %r' % (new_spec,))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
640
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
641 # detect key prop change for potential index change
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
642 keyprop_changes = {}
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
643 if new_spec[0] != old_spec[0]:
3554
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
644 if old_spec[0]:
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
645 keyprop_changes['remove'] = old_spec[0]
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
646 if new_spec[0]:
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
647 keyprop_changes['add'] = new_spec[0]
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
648
1479
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
649 # detect multilinks that have been removed, and drop their table
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
650 old_has = {}
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
651 for name, prop in old_spec[1]:
1479
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
652 old_has[name] = 1
6392
99455aeec1ae Fix dropping of multilinks on schema change
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6332
diff changeset
653 if name in spec.properties and not spec.properties[name].computed:
1479
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
654 continue
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
655
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
656 if prop.find('Multilink to') != -1:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
657 # first drop indexes.
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
658 self.drop_multilink_table_indexes(spec.classname, name)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
659
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
660 # now the multilink table itself
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
661 sql = 'drop table %s_%s' % (spec.classname, name)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
662 else:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
663 # if this is the key prop, drop the index first
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
664 if old_spec[0] == prop:
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
665 self.drop_class_table_key_index(spec.classname, name)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
666 del keyprop_changes['remove']
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
667
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
668 # drop the column
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
669 sql = 'alter table _%s drop column _%s' % (spec.classname,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
670 name)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
671
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
672 self.sql(sql)
1479
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
673
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
674 # if we didn't remove the key prop just then, but the key prop has
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
675 # changed, we still need to remove the old index
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
676 if 'remove' in keyprop_changes:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
677 self.drop_class_table_key_index(spec.classname,
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
678 keyprop_changes['remove'])
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
679
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
680 # add new columns
2196
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
681 for propname, prop in new_spec[1]:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
682 if propname in old_has:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
683 continue
2196
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
684 prop = spec.properties[propname]
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
685 if isinstance(prop, Multilink):
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
686 self.create_multilink_table(spec, propname)
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
687 else:
2727
93e2e5b55a3c new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
688 # add the column
5175
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
689 coltype = self.hyperdb_to_sql_datatype(prop.__class__, prop)
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
690 sql = 'alter table _%s add column _%s %s' % (
2727
93e2e5b55a3c new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
691 spec.classname, propname, coltype)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
692 self.sql(sql)
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
693
2727
93e2e5b55a3c new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
694 # extra Interval column
93e2e5b55a3c new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
695 if isinstance(prop, Interval):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
696 sql = 'alter table _%s add column __%s_int__ BIGINT' % (
2727
93e2e5b55a3c new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
697 spec.classname, propname)
93e2e5b55a3c new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
698 self.sql(sql)
93e2e5b55a3c new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
699
2196
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
700 # if the new column is a key prop, we need an index!
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
701 if new_spec[0] == propname:
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
702 self.create_class_table_key_index(spec.classname, propname)
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
703 del keyprop_changes['add']
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
704
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
705 # if we didn't add the key prop just then, but the key prop has
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
706 # changed, we still need to add the new index
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
707 if 'add' in keyprop_changes:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
708 self.create_class_table_key_index(spec.classname,
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
709 keyprop_changes['add'])
1479
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
710
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
711 return 1
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
712
3016
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
713 def determine_all_columns(self, spec):
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
714 """Figure out the columns from the spec and also add internal columns
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
715
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
716 """
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
717 cols, mls = self.determine_columns(list(spec.properties.items()))
3082
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
718
3016
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
719 # add on our special columns
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
720 cols.append(('id', 'INTEGER PRIMARY KEY'))
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
721 cols.append(('__retired__', 'INTEGER DEFAULT 0'))
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
722 return cols, mls
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
723
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
724 def create_class_table(self, spec):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
725 """Create the class table for the given Class "spec". Creates the
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
726 indexes too."""
3016
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
727 cols, mls = self.determine_all_columns(spec)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
728
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
729 # create the base table
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
730 scols = ','.join(['%s %s' % x for x in cols])
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
731 sql = 'create table _%s (%s)' % (spec.classname, scols)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
732 self.sql(sql)
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
733
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
734 self.create_class_table_indexes(spec)
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
735
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
736 return cols, mls
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
737
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
738 def create_class_table_indexes(self, spec):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
739 """ create the class table for the given spec
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
740 """
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
741 # create __retired__ index
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
742 index_sql2 = 'create index _%s_retired_idx on _%s(__retired__)' % (
1836
94e430ad4fdb make the RDBMS common backend and the SQLite and MYsql backend create...
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1800
diff changeset
743 spec.classname, spec.classname)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
744 self.sql(index_sql2)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
745
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
746 # create index for key property
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
747 if spec.key:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
748 index_sql3 = 'create index _%s_%s_idx on _%s(_%s)' % (
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
749 spec.classname, spec.key,
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
750 spec.classname, spec.key)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
751 self.sql(index_sql3)
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
752
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
753 # and the unique index for key / retired(id)
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
754 self.add_class_key_required_unique_constraint(spec.classname,
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
755 spec.key)
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
756
2228
1d1362c54c94 Some doc / comment fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 2217
diff changeset
757 # TODO: create indexes on (selected?) Link property columns, as
1d1362c54c94 Some doc / comment fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 2217
diff changeset
758 # they're more likely to be used for lookup
1d1362c54c94 Some doc / comment fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 2217
diff changeset
759
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
760 def add_class_key_required_unique_constraint(self, cn, key):
3969
905faf52a51f fix mysql breakage in 1.4.2
Richard Jones <richard@users.sourceforge.net>
parents: 3967
diff changeset
761 sql = '''create unique index _%s_key_retired_idx
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
762 on _%s(__retired__, _%s)''' % (cn, cn, key)
4774
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
763 try:
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
764 self.sql(sql)
6002
3175bb92ca28 Cleanups for bandit
John Rouillard <rouilj@ieee.org>
parents: 5871
diff changeset
765 except Exception: # nosec
4774
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
766 # XXX catch e.g.:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
767 # _sqlite.DatabaseError: index _status_key_retired_idx
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
768 # already exists
4774
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
769 pass
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
770
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
771 def drop_class_table_indexes(self, cn, key):
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
772 # drop the old table indexes first
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
773 l = ['_%s_id_idx' % cn, '_%s_retired_idx' % cn]
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
774 if key:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
775 l.append('_%s_%s_idx' % (cn, key))
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
776
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
777 table_name = '_%s' % cn
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
778 for index_name in l:
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
779 if not self.sql_index_exists(table_name, index_name):
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
780 continue
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
781 index_sql = 'drop index '+index_name
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
782 self.sql(index_sql)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
783
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
784 def create_class_table_key_index(self, cn, key):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
785 """ create the class table for the given spec
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
786 """
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
787 sql = 'create index _%s_%s_idx on _%s(_%s)' % (cn, key, cn, key)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
788 self.sql(sql)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
789
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
790 def drop_class_table_key_index(self, cn, key):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
791 table_name = '_%s' % cn
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
792 index_name = '_%s_%s_idx' % (cn, key)
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
793 if self.sql_index_exists(table_name, index_name):
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
794 sql = 'drop index '+index_name
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
795 self.sql(sql)
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
796
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
797 # and now the retired unique index too
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
798 index_name = '_%s_key_retired_idx' % cn
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
799 if self.sql_index_exists(table_name, index_name):
3967
b1e81ad3fa6a don't recompute the index name
Richard Jones <richard@users.sourceforge.net>
parents: 3963
diff changeset
800 sql = 'drop index '+index_name
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
801 self.sql(sql)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
802
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
803 def create_journal_table(self, spec):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
804 """ create the journal table for a class given the spec and
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
805 already-determined cols
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
806 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
807 # journal table
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
808 sql = """create table %s__journal (
3224
1fd11a9803bb use backend datatype for journal timestamps.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3155
diff changeset
809 nodeid integer, date %s, tag varchar(255),
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
810 action varchar(255), params text)""" % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
811 spec.classname,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
812 self.hyperdb_to_sql_datatype(hyperdb.Date))
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
813 self.sql(sql)
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
814 self.create_journal_table_indexes(spec)
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
815
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
816 def create_journal_table_indexes(self, spec):
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
817 # index on nodeid
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
818 sql = 'create index %s_journ_idx on %s__journal(nodeid)' % (
1836
94e430ad4fdb make the RDBMS common backend and the SQLite and MYsql backend create...
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1800
diff changeset
819 spec.classname, spec.classname)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
820 self.sql(sql)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
821
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
822 def drop_journal_table_indexes(self, classname):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
823 index_name = '%s_journ_idx' % classname
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
824 if not self.sql_index_exists('%s__journal' % classname, index_name):
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
825 return
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
826 index_sql = 'drop index '+index_name
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
827 self.sql(index_sql)
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
828
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
829 def create_multilink_table(self, spec, ml):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
830 """ Create a multilink table for the "ml" property of the class
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
831 given by the spec
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
832 """
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
833 # create the table
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
834 sql = 'create table %s_%s (linkid INTEGER, nodeid INTEGER)' % (
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
835 spec.classname, ml)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
836 self.sql(sql)
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
837 self.create_multilink_table_indexes(spec, ml)
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
838
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
839 def create_multilink_table_indexes(self, spec, ml):
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
840 # create index on linkid
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
841 index_sql = 'create index %s_%s_l_idx on %s_%s(linkid)' % (
2100
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
842 spec.classname, ml, spec.classname, ml)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
843 self.sql(index_sql)
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
844
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
845 # create index on nodeid
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
846 index_sql = 'create index %s_%s_n_idx on %s_%s(nodeid)' % (
2100
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
847 spec.classname, ml, spec.classname, ml)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
848 self.sql(index_sql)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
849
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
850 def drop_multilink_table_indexes(self, classname, ml):
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
851 l = [
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
852 '%s_%s_l_idx' % (classname, ml),
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
853 '%s_%s_n_idx' % (classname, ml)
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
854 ]
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
855 table_name = '%s_%s' % (classname, ml)
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
856 for index_name in l:
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
857 if not self.sql_index_exists(table_name, index_name):
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
858 continue
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
859 index_sql = 'drop index %s' % index_name
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
860 self.sql(index_sql)
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
861
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
862 def create_class(self, spec):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
863 """ Create a database table according to the given spec.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
864 """
4474
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
865
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
866 if not self.config.RDBMS_ALLOW_CREATE:
7750
216662fbaaee fix(i18n): fix incorrect lookup of some translations
John Rouillard <rouilj@ieee.org>
parents: 7696
diff changeset
867 raise DatabaseError(_('CREATE operation disallowed: "%s".') %
216662fbaaee fix(i18n): fix incorrect lookup of some translations
John Rouillard <rouilj@ieee.org>
parents: 7696
diff changeset
868 spec.classname)
4474
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
869
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
870 cols, mls = self.create_class_table(spec)
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
871 self.create_journal_table(spec)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
872
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
873 # now create the multilink tables
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
874 for ml in mls:
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
875 self.create_multilink_table(spec, ml)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
876
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
877 def drop_class(self, cn, spec):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
878 """ Drop the given table from the database.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
879
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
880 Drop the journal and multilink tables too.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
881 """
4474
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
882
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
883 if not self.config.RDBMS_ALLOW_DROP:
7750
216662fbaaee fix(i18n): fix incorrect lookup of some translations
John Rouillard <rouilj@ieee.org>
parents: 7696
diff changeset
884 raise DatabaseError(_('DROP operation disallowed: "%s".') % cn)
4474
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
885
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
886 properties = spec[1]
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
887 # figure the multilinks
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
888 mls = []
3924
21d3d7eeea8c assorted pyflakes fixes
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3906
diff changeset
889 for propname, prop in properties:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
890 if isinstance(prop, Multilink):
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
891 mls.append(propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
892
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
893 # drop class table and indexes
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
894 self.drop_class_table_indexes(cn, spec[0])
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
895
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
896 self.drop_class_table(cn)
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
897
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
898 # drop journal table and indexes
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
899 self.drop_journal_table_indexes(cn)
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
900 sql = 'drop table %s__journal' % cn
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
901 self.sql(sql)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
902
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
903 for ml in mls:
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
904 # drop multilink table and indexes
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
905 self.drop_multilink_table_indexes(cn, ml)
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
906 sql = 'drop table %s_%s' % (spec.classname, ml)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
907 self.sql(sql)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
908
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
909 def drop_class_table(self, cn):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
910 sql = 'drop table _%s' % cn
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
911 self.sql(sql)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
912
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
913 #
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
914 # Classes
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
915 #
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
916 def __getattr__(self, classname):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
917 """ A convenient way of calling self.getclass(classname).
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
918 """
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
919 if classname in self.classes:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
920 return self.classes[classname]
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
921 raise AttributeError(classname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
922
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
923 def addclass(self, cl):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
924 """ Add a Class to the hyperdatabase.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
925 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
926 cn = cl.classname
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
927 if cn in self.classes:
7750
216662fbaaee fix(i18n): fix incorrect lookup of some translations
John Rouillard <rouilj@ieee.org>
parents: 7696
diff changeset
928 raise ValueError(_('Class "%s" already defined.') % cn)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
929 self.classes[cn] = cl
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
930
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
931 # add default Edit and View permissions
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
932 self.security.addPermission(
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
933 name="Create", klass=cn,
2649
1df7d4a41da4 Buncha stuff (sorry about the large checkin):
Richard Jones <richard@users.sourceforge.net>
parents: 2640
diff changeset
934 description="User is allowed to create "+cn)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
935 self.security.addPermission(
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
936 name="Edit", klass=cn,
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
937 description="User is allowed to edit "+cn)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
938 self.security.addPermission(
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
939 name="View", klass=cn,
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
940 description="User is allowed to access "+cn)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
941 self.security.addPermission(
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
942 name="Retire", klass=cn,
4517
f8e85cf5f0fe how odd, the Retire permission wasn't being registered;
Richard Jones <richard@users.sourceforge.net>
parents: 4490
diff changeset
943 description="User is allowed to retire "+cn)
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
944
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
945 def getclasses(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
946 """ Return a list of the names of all existing classes.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
947 """
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
948 return sorted(self.classes)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
949
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
950 def getclass(self, classname):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
951 """Get the Class object representing a particular class.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
952
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
953 If 'classname' is not a valid class name, a KeyError is raised.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
954 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
955 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
956 return self.classes[classname]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
957 except KeyError:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
958 raise KeyError('There is no class called "%s"' % classname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
959
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
960 def clear(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
961 """Delete all database contents.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
962
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1988
diff changeset
963 Note: I don't commit here, which is different behaviour to the
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1988
diff changeset
964 "nuke from orbit" behaviour in the dbs.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
965 """
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
966 logging.getLogger('roundup.hyperdb.backend').info('clear')
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
967 for cn in self.classes:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
968 sql = 'delete from _%s' % cn
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
969 self.sql(sql)
7007
05d81de0d92d in clear() clear journal tables as well.
John Rouillard <rouilj@ieee.org>
parents: 6965
diff changeset
970 sql = 'delete from %s__journal' % cn
05d81de0d92d in clear() clear journal tables as well.
John Rouillard <rouilj@ieee.org>
parents: 6965
diff changeset
971 self.sql(sql)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
972
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
973 #
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
974 # Nodes
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
975 #
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
976
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
977 hyperdb_to_sql_value = {
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
978 hyperdb.String : str, # noqa: E203
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
979 # fractional seconds by default
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
980 hyperdb.Date : lambda x: x.formal(sep=' ', # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
981 sec='%06.3f'),
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
982 hyperdb.Link : int, # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
983 hyperdb.Interval : str, # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
984 hyperdb.Password : str, # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
985 hyperdb.Boolean : lambda x: x and 'TRUE' or 'FALSE', # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
986 hyperdb.Number : lambda x: x, # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
987 hyperdb.Integer : lambda x: x, # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
988 # used in journal marshalling
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
989 hyperdb.Multilink : lambda x: x, # noqa: E203
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
990 }
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
991
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
992 def to_sql_value(self, propklass):
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
993
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
994 fn = self.hyperdb_to_sql_value.get(propklass)
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
995 if fn:
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
996 return fn
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
997
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
998 for k, v in self.hyperdb_to_sql_value.items():
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
999 if issubclass(propklass, k):
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1000 return v
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1001
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1002 raise ValueError('%r is not a hyperdb property class' % propklass)
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1003
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1004 def _cache_del(self, key):
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1005 del self.cache[key]
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1006 self.cache_lru.remove(key)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1007
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1008 def _cache_refresh(self, key):
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1009 self.cache_lru.remove(key)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1010 self.cache_lru.insert(0, key)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1011
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1012 def _cache_save(self, key, node):
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1013 self.cache[key] = node
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1014 # update the LRU
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1015 self.cache_lru.insert(0, key)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1016 if len(self.cache_lru) > self.cache_size:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1017 del self.cache[self.cache_lru.pop()]
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1018
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1019 def addnode(self, classname, nodeid, node):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1020 """ Add the specified node to its class's db.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1021 """
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1022 self.log_debug('addnode %s%s %r' % (classname,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1023 nodeid, node))
1528
96cd422532ef bye bye gadfly - you served your purpose well [SF#701127]
Richard Jones <richard@users.sourceforge.net>
parents: 1523
diff changeset
1024
96cd422532ef bye bye gadfly - you served your purpose well [SF#701127]
Richard Jones <richard@users.sourceforge.net>
parents: 1523
diff changeset
1025 # determine the column definitions and multilink tables
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1026 cl = self.classes[classname]
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1027 cols, mls = self.determine_columns(list(cl.properties.items()))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1028
1189
23b8d1e87fe3 import fixes
Richard Jones <richard@users.sourceforge.net>
parents: 1187
diff changeset
1029 # we'll be supplied these props if we're doing an import
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1030 values = node.copy()
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1031 if 'creator' not in values:
1189
23b8d1e87fe3 import fixes
Richard Jones <richard@users.sourceforge.net>
parents: 1187
diff changeset
1032 # add in the "calculated" properties (dupe so we don't affect
23b8d1e87fe3 import fixes
Richard Jones <richard@users.sourceforge.net>
parents: 1187
diff changeset
1033 # calling code's node assumptions)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1034 values['creation'] = values['activity'] = date.Date()
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1035 values['actor'] = values['creator'] = self.getuid()
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1036
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1037 cl = self.classes[classname]
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1038 props = cl.getprops(protected=1)
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1039 del props['id']
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1040
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1041 # default the non-multilink columns
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1042 for col, prop in props.items():
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1043 if col not in values:
1496
e6ac4e074acb relaxed CVS importing (feature [SF#693277])
Richard Jones <richard@users.sourceforge.net>
parents: 1492
diff changeset
1044 if isinstance(prop, Multilink):
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1045 values[col] = []
1496
e6ac4e074acb relaxed CVS importing (feature [SF#693277])
Richard Jones <richard@users.sourceforge.net>
parents: 1492
diff changeset
1046 else:
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1047 values[col] = None
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1048
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1049 # clear this node out of the cache if it's in there
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1050 key = (classname, nodeid)
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1051 if key in self.cache:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1052 self._cache_del(key)
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1053
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1054 # figure the values to insert
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1055 vals = []
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1056 for col, _dt in cols:
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1057 # this is somewhat dodgy....
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1058 if col.endswith('_int__'):
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1059 # XXX eugh, this test suxxors
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1060 value = values[col[2:-6]]
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1061 # this is an Interval special "int" column
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1062 if value is not None:
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1063 vals.append(value.as_seconds())
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1064 else:
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1065 vals.append(value)
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1066 continue
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1067
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1068 prop = props[col[1:]]
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1069 value = values[col[1:]]
2472
f41539b3c486 fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents: 2456
diff changeset
1070 if value is not None:
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1071 value = self.to_sql_value(prop.__class__)(value)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1072 vals.append(value)
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1073 vals.append(nodeid)
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1074 vals = tuple(vals)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1075
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1076 # make sure the ordering is correct for column name -> column value
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1077 s = ','.join([self.arg for x in cols]) + ',%s' % self.arg
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1078 cols = ','.join([col for col, dt in cols]) + ',id'
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1079
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1080 # perform the inserts
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1081 sql = 'insert into _%s (%s) values (%s)' % (classname, cols, s)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
1082 self.sql(sql, vals)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1083
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1084 # insert the multilink rows
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1085 for col in mls:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1086 t = '%s_%s' % (classname, col)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1087 for entry in node[col]:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1088 sql = 'insert into %s (linkid, nodeid) values (%s,%s)' % (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1089 t, self.arg, self.arg)
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1090 self.sql(sql, (entry, nodeid))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1091
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1092 def setnode(self, classname, nodeid, values, multilink_changes=None):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1093 """ Change the specified node.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1094 """
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1095 if multilink_changes is None: multilink_changes = {} # noqa: E701
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1096 self.log_debug('setnode %s%s %r' % (classname, nodeid, values))
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1097
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1098 # clear this node out of the cache if it's in there
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1099 key = (classname, nodeid)
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1100 if key in self.cache:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1101 self._cache_del(key)
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1102
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1103 cl = self.classes[classname]
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1104 props = cl.getprops()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1105
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1106 cols = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1107 mls = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1108 # add the multilinks separately
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1109 for col in values:
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1110 prop = props[col]
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1111 if isinstance(prop, Multilink):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1112 mls.append(col)
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1113 elif isinstance(prop, Interval):
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1114 # Intervals store the seconds value too
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1115 cols.append(col)
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1116 # extra leading '_' added by code below
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1117 cols.append('_' + col + '_int__')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1118 else:
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1119 cols.append(col)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1120 cols.sort()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1121
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1122 # figure the values to insert
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1123 vals = []
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1124 for col in cols:
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1125 if col.endswith('_int__'):
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1126 # XXX eugh, this test suxxors
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1127 # Intervals store the seconds value too
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1128 col = col[1:-6]
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1129 prop = props[col]
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1130 value = values[col]
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1131 if value is None:
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1132 vals.append(None)
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1133 else:
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1134 vals.append(value.as_seconds())
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1135 else:
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1136 prop = props[col]
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1137 value = values[col]
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1138 if value is None:
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1139 e = None
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1140 else:
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1141 e = self.to_sql_value(prop.__class__)(value)
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1142 vals.append(e)
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1143
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1144 vals.append(int(nodeid))
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1145 vals = tuple(vals)
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1146
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1147 # if there's any updates to regular columns, do them
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1148 if cols:
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1149 # make sure the ordering is correct for column name -> column value
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1150 s = ','.join(['_%s=%s' % (x, self.arg) for x in cols])
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1151 cols = ','.join(cols)
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1152
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1153 # perform the update
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1154 sql = 'update _%s set %s where id=%s' % (classname, s, self.arg)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
1155 self.sql(sql, vals)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1156
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1157 # we're probably coming from an import, not a change
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1158 if not multilink_changes:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1159 for name in mls:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1160 prop = props[name]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1161 value = values[name]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1162
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1163 t = '%s_%s' % (classname, name)
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1164
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1165 # clear out previous values for this node
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1166 # XXX numeric ids
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1167 self.sql('delete from %s where nodeid=%s' % (t, self.arg),
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1168 (nodeid,))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1169
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1170 # insert the values for this node
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1171 for entry in values[name]:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1172 sql = 'insert into %s (linkid, nodeid) values (%s,%s)' % (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1173 t, self.arg, self.arg)
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1174 # XXX numeric ids
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1175 self.sql(sql, (entry, nodeid))
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1176
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1177 # we have multilink changes to apply
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1178 for col, (add, remove) in multilink_changes.items():
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1179 tn = '%s_%s' % (classname, col)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1180 if add:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1181 sql = 'insert into %s (nodeid, linkid) values (%s,%s)' % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1182 tn, self.arg, self.arg)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1183 for addid in add:
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1184 # XXX numeric ids
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1185 self.sql(sql, (int(nodeid), int(addid)))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1186 if remove:
4039
f44ec45cedb4 Fuse multiple DELETE calls into one for multilinks.
Stefan Seefeld <stefan@seefeld.name>
parents: 4038
diff changeset
1187 s = ','.join([self.arg]*len(remove))
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1188 sql = 'delete from %s where nodeid=%s and linkid in (%s)' % (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1189 tn, self.arg, s)
4039
f44ec45cedb4 Fuse multiple DELETE calls into one for multilinks.
Stefan Seefeld <stefan@seefeld.name>
parents: 4038
diff changeset
1190 # XXX numeric ids
f44ec45cedb4 Fuse multiple DELETE calls into one for multilinks.
Stefan Seefeld <stefan@seefeld.name>
parents: 4038
diff changeset
1191 self.sql(sql, [int(nodeid)] + remove)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1192
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1193 sql_to_hyperdb_value = {
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1194 hyperdb.String : us2s, # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1195 hyperdb.Date : date_to_hyperdb_value, # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1196 # hyperdb.Link : int, # XXX numeric ids
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1197 hyperdb.Link : str, # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1198 hyperdb.Interval : date.Interval, # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1199 hyperdb.Password : lambda x: password.Password(encrypted=x), # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1200 hyperdb.Boolean : _bool_cvt, # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1201 hyperdb.Number : _num_cvt, # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1202 hyperdb.Integer : int, # noqa: E203
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1203 # used in journal marshalling
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1204 hyperdb.Multilink : lambda x: x, # noqa: E203
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1205 }
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1206
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1207 def to_hyperdb_value(self, propklass):
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1208
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1209 fn = self.sql_to_hyperdb_value.get(propklass)
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1210 if fn:
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1211 return fn
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1212
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1213 for k, v in self.sql_to_hyperdb_value.items():
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1214 if issubclass(propklass, k):
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1215 return v
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1216
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1217 raise ValueError('%r is not a hyperdb property class' % propklass)
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1218
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1219 def _materialize_multilink(self, classname, nodeid, node, propname):
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1220 """ evaluation of single Multilink (lazy eval may have skipped this)
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1221 """
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1222 if propname not in node:
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
1223 prop = self.getclass(classname).properties[propname]
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1224 tn = prop.table_name
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
1225 lid = prop.linkid_name
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
1226 nid = prop.nodeid_name
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1227 w = ''
6179
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6157
diff changeset
1228 joi = ''
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6157
diff changeset
1229 if prop.computed:
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6157
diff changeset
1230 if isinstance(prop.rev_property, Link):
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1231 w = ' and %s.__retired__=0' % tn
6179
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6157
diff changeset
1232 else:
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6157
diff changeset
1233 tn2 = '_' + prop.classname
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6157
diff changeset
1234 joi = ', %s' % tn2
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1235 w = ' and %s.%s=%s.id and %s.__retired__=0' % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1236 tn, lid, tn2, tn2)
6332
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
1237 cursor = self.sql_new_cursor(name='_materialize_multilink')
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1238 sql = 'select %s from %s%s where %s=%s%s' % (lid, tn, joi, nid,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1239 self.arg, w)
6332
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
1240 self.sql(sql, (nodeid,), cursor)
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
1241 # Reduce this to only the first row (the ID), this can save a
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
1242 # lot of space for large query results (not using fetchall)
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
1243 node[propname] = [str(x) for x in sorted(int(r[0]) for r in cursor)]
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
1244 cursor.close()
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1245
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1246 def _materialize_multilinks(self, classname, nodeid, node, props=None):
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1247 """ get all Multilinks of a node (lazy eval may have skipped this)
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1248 """
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1249 cl = self.classes[classname]
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1250 props = props or [pn for (pn, p) in cl.properties.items()
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1251 if isinstance(p, Multilink)]
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1252 for propname in props:
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1253 if propname not in node:
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1254 self._materialize_multilink(classname, nodeid, node, propname)
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1255
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1256 def getnode(self, classname, nodeid, fetch_multilinks=True):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1257 """ Get a node from the database.
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1258 For optimisation optionally we don't fetch multilinks
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1259 (lazy Multilinks).
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1260 But for internal database operations we need them.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1261 """
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1262 # see if we have this node cached
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1263 key = (classname, nodeid)
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1264 if key in self.cache:
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1265 # push us back to the top of the LRU
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1266 self._cache_refresh(key)
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1267 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1268 self.stats['cache_hits'] += 1
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1269 # return the cached information
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1270 if fetch_multilinks:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1271 self._materialize_multilinks(classname, nodeid,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1272 self.cache[key])
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1273 return self.cache[key]
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1274
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1275 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1276 self.stats['cache_misses'] += 1
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1277 start_t = time.time()
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1278
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1279 # figure the columns we're fetching
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1280 cl = self.classes[classname]
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1281 cols, mls = self.determine_columns(list(cl.properties.items()))
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1282 scols = ','.join([col for col, dt in cols])
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1283
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1284 # perform the basic property fetch
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1285 sql = 'select %s from _%s where id=%s' % (scols, classname, self.arg)
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1286 self.sql(sql, (nodeid,))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1287
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1288 values = self.sql_fetchone()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1289 if values is None:
6197
2492e2e17371 Change text from no such <class> node <node id> to drop "node"
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
1290 raise IndexError('no such %s %s' % (classname, nodeid))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1291
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1292 # make up the node
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1293 node = {}
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1294 props = cl.getprops(protected=1)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1295 for col in range(len(cols)):
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1296 name = cols[col][0][1:]
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1297 if name.endswith('_int__'):
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1298 # XXX eugh, this test suxxors
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1299 # ignore the special Interval-as-seconds column
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1300 continue
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1301 value = values[col]
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1302 if value is not None:
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1303 value = self.to_hyperdb_value(props[name].__class__)(value)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1304 node[name] = value
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1305
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1306 if fetch_multilinks and mls:
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1307 self._materialize_multilinks(classname, nodeid, node, mls)
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1308
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1309 # save off in the cache
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1310 key = (classname, nodeid)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1311 self._cache_save(key, node)
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1312
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1313 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1314 self.stats['get_items'] += (time.time() - start_t)
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1315
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1316 return node
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1317
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1318 def destroynode(self, classname, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1319 """Remove a node from the database. Called exclusively by the
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1320 destroy() method on Class.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1321 """
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1322 message = 'destroynode %s%s' % (classname, nodeid)
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1323 logging.getLogger('roundup.hyperdb.backend').info(message)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1324
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1325 # make sure the node exists
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1326 if not self.hasnode(classname, nodeid):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1327 raise IndexError('%s has no node %s' % (classname, nodeid))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1328
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1329 # see if we have this node cached
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1330 if (classname, nodeid) in self.cache:
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1331 del self.cache[(classname, nodeid)]
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1332
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1333 # see if there's any obvious commit actions that we should get rid of
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1334 for entry in self.transactions[:]:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1335 if entry[1][:2] == (classname, nodeid):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1336 self.transactions.remove(entry)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1337
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1338 # now do the SQL
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1339 sql = 'delete from _%s where id=%s' % (classname, self.arg)
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1340 self.sql(sql, (nodeid,))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1341
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1342 # remove from multilnks
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1343 cl = self.getclass(classname)
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1344 x, mls = self.determine_columns(list(cl.properties.items()))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1345 for col in mls:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1346 # get the link ids
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1347 sql = 'delete from %s_%s where nodeid=%s' % (classname, col,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1348 self.arg)
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
1349 self.sql(sql, (nodeid,))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1350
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1351 # remove journal entries
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1352 sql = 'delete from %s__journal where nodeid=%s' % (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1353 classname, self.arg)
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1354 self.sql(sql, (nodeid,))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1355
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
1356 # cleanup any blob filestorage when we commit
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1357 self.transactions.append((FileStorage.destroy, (self,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1358 classname, nodeid)))
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
1359
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1360 def hasnode(self, classname, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1361 """ Determine if the database has a given node.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1362 """
5086
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1363 # nodeid (aka link) is sql type INTEGER. max positive value
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1364 # for INTEGER is 2^31-1 for Postgres and MySQL. The max
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1365 # positive value for SqLite is 2^63 -1, so arguably this check
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1366 # needs to adapt for the type of the RDBMS. For right now,
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1367 # choose lowest common denominator.
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1368 if int(nodeid) >= 2**31:
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1369 # value out of range return false
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1370 return 0
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1371
4031
6e2dec4f9836 Fix issue2550501
Stefan Seefeld <stefan@seefeld.name>
parents: 3993
diff changeset
1372 # If this node is in the cache, then we do not need to go to
6e2dec4f9836 Fix issue2550501
Stefan Seefeld <stefan@seefeld.name>
parents: 3993
diff changeset
1373 # the database. (We don't consider this an LRU hit, though.)
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1374 if (classname, nodeid) in self.cache:
4031
6e2dec4f9836 Fix issue2550501
Stefan Seefeld <stefan@seefeld.name>
parents: 3993
diff changeset
1375 # Return 1, not True, to match the type of the result of
6e2dec4f9836 Fix issue2550501
Stefan Seefeld <stefan@seefeld.name>
parents: 3993
diff changeset
1376 # the SQL operation below.
6e2dec4f9836 Fix issue2550501
Stefan Seefeld <stefan@seefeld.name>
parents: 3993
diff changeset
1377 return 1
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1378 sql = 'select count(*) from _%s where id=%s' % (classname, self.arg)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
1379 self.sql(sql, (nodeid,))
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1380 return int(self.cursor.fetchone()[0])
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1381
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1382 def countnodes(self, classname):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1383 """ Count the number of nodes that exist for a particular Class.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1384 """
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1385 sql = 'select count(*) from _%s' % classname
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
1386 self.sql(sql)
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1387 return self.cursor.fetchone()[0]
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1388
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1389 def addjournal(self, classname, nodeid, action, params, creator=None,
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1390 creation=None):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1391 """ Journal the Action
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1392 'action' may be:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1393
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1394 'set' -- 'params' is a dictionary of property values
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1395 'create' -- 'params' is an empty dictionary as of
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1396 Wed Nov 06 11:38:43 2002 +0000
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1397 'link' or 'unlink' -- 'params' is (classname, nodeid, propname)
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1398 'retired' or 'restored' -- 'params' is None
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1399
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1400 'creator' -- the user performing the action, which defaults to
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1401 the current user.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1402 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1403 # handle supply of the special journalling parameters (usually
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1404 # supplied on importing an existing database)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1405 if creator:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1406 journaltag = creator
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1407 else:
1800
a3b1b1dcf639 Use getuid(), not figure_curuserid()
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 1794
diff changeset
1408 journaltag = self.getuid()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1409 if creation:
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1410 journaldate = creation
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1411 else:
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1412 journaldate = date.Date()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1413
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1414 # create the journal entry
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1415 cols = 'nodeid,date,tag,action,params'
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1416
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1417 self.log_debug('addjournal %s%s %r %s %s %r' % (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1418 classname, nodeid, journaldate, journaltag, action, params))
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1419
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1420 # make the journalled data marshallable
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1421 if isinstance(params, type({})):
2275
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1422 self._journal_marshal(params, classname)
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1423
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
1424 params = repr_export(params)
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1425
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1426 dc = self.to_sql_value(hyperdb.Date)
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1427 journaldate = dc(journaldate)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1428
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1429 self.save_journal(classname, cols, nodeid, journaldate,
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1430 journaltag, action, params)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1431
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1432 def setjournal(self, classname, nodeid, journal):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1433 """Set the journal to the "journal" list."""
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1434 # clear out any existing entries
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1435 self.sql('delete from %s__journal where nodeid=%s' % (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1436 classname, self.arg), (nodeid,))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1437
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1438 # create the journal entry
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1439 cols = 'nodeid,date,tag,action,params'
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1440
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1441 dc = self.to_sql_value(hyperdb.Date)
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1442 for nodeid, journaldate, journaltag, action, params in journal:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1443 self.log_debug('addjournal %s%s %r %s %s %r' % (
4059
ef0b4396888a Enhance and simplify logging.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1444 classname, nodeid, journaldate, journaltag, action,
ef0b4396888a Enhance and simplify logging.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1445 params))
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1446
2275
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1447 # make the journalled data marshallable
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1448 if isinstance(params, type({})):
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1449 self._journal_marshal(params, classname)
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
1450 params = repr_export(params)
2275
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1451
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1452 self.save_journal(classname, cols, nodeid, dc(journaldate),
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1453 journaltag, action, params)
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1454
2275
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1455 def _journal_marshal(self, params, classname):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1456 """Convert the journal params values into safely repr'able and
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1457 eval'able values."""
2275
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1458 properties = self.getclass(classname).getprops()
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1459 for param, value in params.items():
2275
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1460 if not value:
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1461 continue
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1462 property = properties[param]
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1463 cvt = self.to_sql_value(property.__class__)
2275
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1464 if isinstance(property, Password):
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1465 params[param] = cvt(value)
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1466 elif isinstance(property, Date):
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1467 params[param] = cvt(value)
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1468 elif isinstance(property, Interval):
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1469 params[param] = cvt(value)
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1470 elif isinstance(property, Boolean):
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1471 params[param] = cvt(value)
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1472
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1473 def getjournal(self, classname, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1474 """ get the journal for id
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1475 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1476 # make sure the node exists
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1477 if not self.hasnode(classname, nodeid):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1478 raise IndexError('%s has no node %s' % (classname, nodeid))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1479
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1480 cols = ','.join('nodeid date tag action params'.split())
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1481 journal = self.load_journal(classname, cols, nodeid)
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1482
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1483 # now unmarshal the data
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1484 dc = self.to_hyperdb_value(hyperdb.Date)
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1485 res = []
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1486 properties = self.getclass(classname).getprops()
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1487 for nodeid, date_stamp, user, action, params in journal:
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
1488 params = eval_import(params)
2248
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1489 if isinstance(params, type({})):
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1490 for param, value in params.items():
2248
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1491 if not value:
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1492 continue
2731
7c472ed6babf handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents: 2727
diff changeset
1493 property = properties.get(param, None)
7c472ed6babf handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents: 2727
diff changeset
1494 if property is None:
7c472ed6babf handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents: 2727
diff changeset
1495 # deleted property
7c472ed6babf handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents: 2727
diff changeset
1496 continue
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1497 cvt = self.to_hyperdb_value(property.__class__)
2248
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1498 if isinstance(property, Password):
4483
22bc0426e348 Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4480
diff changeset
1499 params[param] = password.JournalPassword(value)
2248
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1500 elif isinstance(property, Date):
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1501 params[param] = cvt(value)
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1502 elif isinstance(property, Interval):
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1503 params[param] = cvt(value)
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1504 elif isinstance(property, Boolean):
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1505 params[param] = cvt(value)
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1506 # XXX numeric ids
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1507 res.append((str(nodeid), dc(date_stamp), user, action, params))
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1508 return res
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1509
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1510 def save_journal(self, classname, cols, nodeid, journaldate,
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1511 journaltag, action, params):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1512 """ Save the journal entry to the database
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1513 """
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1514 entry = (nodeid, journaldate, journaltag, action, params)
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1515
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1516 # do the insert
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1517 a = self.arg
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1518 sql = 'insert into %s__journal (%s) values (%s,%s,%s,%s,%s)' % (
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1519 classname, cols, a, a, a, a, a)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
1520 self.sql(sql, entry)
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1521
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1522 def load_journal(self, classname, cols, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1523 """ Load the journal from the database
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1524 """
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1525 # now get the journal entries
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1526 sql = 'select %s from %s__journal where nodeid=%s order by date' % (
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1527 cols, classname, self.arg)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
1528 self.sql(sql, (nodeid,))
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1529 return self.cursor.fetchall()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1530
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1531 def pack(self, pack_before):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1532 """ Delete all journal entries except "create" before 'pack_before'.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1533 """
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1534 date_stamp = self.to_sql_value(Date)(pack_before)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1535
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1536 # do the delete
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1537 for classname in self.classes:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1538 sql = "delete from %s__journal where date<%s and "\
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1539 "action<>'create'" % (classname, self.arg)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
1540 self.sql(sql, (date_stamp,))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1541
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1542 def sql_commit(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1543 """ Actually commit to the database.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1544 """
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1545 logging.getLogger('roundup.hyperdb.backend').info('commit')
3687
ff9f4ca42454 Postgres backend allows transaction collisions to be ignored when...
Richard Jones <richard@users.sourceforge.net>
parents: 3685
diff changeset
1546
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1547 self.conn.commit()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1548
2374
31cb1014300c Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents: 2362
diff changeset
1549 # open a new cursor for subsequent work
31cb1014300c Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents: 2362
diff changeset
1550 self.cursor = self.conn.cursor()
31cb1014300c Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents: 2362
diff changeset
1551
6332
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
1552 def sql_new_cursor(self, conn=None, *args, **kw):
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
1553 """ Create new cursor, this may need additional parameters for
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
1554 performance optimization for different backends.
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
1555 """
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
1556 if conn is None:
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
1557 conn = self.conn
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
1558 return conn.cursor()
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
1559
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1560 def commit(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1561 """ Commit the current transactions.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1562
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1563 Save all data changed since the database was opened or since the
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1564 last commit() or rollback().
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1565 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1566 # commit the database
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1567 self.sql_commit()
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1568
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1569 # session and otk are committed with the db but not the other
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1570 # way round
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1571 if self.Session:
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1572 self.Session.commit()
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1573 if self.Otk:
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1574 self.Otk.commit()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1575
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1576 # now, do all the other transaction stuff
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1577 for method, args in self.transactions:
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1578 method(*args)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1579
3295
a615cc230160 added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents: 3239
diff changeset
1580 # save the indexer
a615cc230160 added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents: 3239
diff changeset
1581 self.indexer.save_index()
a615cc230160 added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents: 3239
diff changeset
1582
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1583 # clear out the transactions
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1584 self.transactions = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1585
4448
2784c239e6c8 clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4430
diff changeset
1586 # clear the cache: Don't carry over cached values from one
2784c239e6c8 clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4430
diff changeset
1587 # transaction to the next (there may be other changes from other
2784c239e6c8 clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4430
diff changeset
1588 # transactions)
2784c239e6c8 clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4430
diff changeset
1589 self.clearCache()
2784c239e6c8 clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4430
diff changeset
1590
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1591 def sql_rollback(self):
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1592 self.conn.rollback()
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1593
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1594 def rollback(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1595 """ Reverse all actions from the current transaction.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1596
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1597 Undo all the changes made since the database was opened or the last
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1598 commit() or rollback() was performed.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1599 """
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1600 logging.getLogger('roundup.hyperdb.backend').info('rollback')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1601
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1602 self.sql_rollback()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1603
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1604 # roll back "other" transaction stuff
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1605 for method, args in self.transactions:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1606 # delete temporary files
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1607 if method == self.doStoreFile:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1608 self.rollbackStoreFile(*args)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1609 self.transactions = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1610
1492
2fc7d4a8c9e7 fixed sqlite rollback/caching bug [SF#689383]
Richard Jones <richard@users.sourceforge.net>
parents: 1484
diff changeset
1611 # clear the cache
2fc7d4a8c9e7 fixed sqlite rollback/caching bug [SF#689383]
Richard Jones <richard@users.sourceforge.net>
parents: 1484
diff changeset
1612 self.clearCache()
2fc7d4a8c9e7 fixed sqlite rollback/caching bug [SF#689383]
Richard Jones <richard@users.sourceforge.net>
parents: 1484
diff changeset
1613
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1614 def sql_close(self):
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1615 logging.getLogger('roundup.hyperdb.backend').info('close')
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1616 self.conn.close()
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1617
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1618 def close(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1619 """ Close off the connection.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1620 """
2093
3f6024ab2c7a That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
1621 self.indexer.close()
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1622 self.sql_close()
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1623 if self.Session:
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1624 self.Session.close()
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1625 self.Session = None
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1626 if self.Otk:
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1627 self.Otk.close()
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1628 self.Otk = None
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1629
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1630
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1631 #
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1632 # The base Class class
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1633 #
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1634 class Class(hyperdb.Class):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1635 """ The handle to a particular class of nodes in a hyperdatabase.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1636
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1637 All methods except __repr__ and getnode must be implemented by a
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1638 concrete backend Class.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1639 """
4849
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
1640 # For many databases the LIKE operator ignores case.
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
1641 # Postgres and Oracle have an ILIKE operator to support this.
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
1642 # We define the default here, can be changed in derivative class
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
1643 case_insensitive_like = 'LIKE'
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1644
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1645 # For some databases (mysql) the = operator for strings ignores case.
7860
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
1646 # We define the default here, can be changed in derivative class.
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
1647 # If set to any false value, self.get_case_sensitive_equal() is
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
1648 # called to set its value.
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1649 case_sensitive_equal = '='
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1650
6396
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
1651 # Some DBs order NULL values last. Set this variable in the backend
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
1652 # for prepending an order by clause for each attribute that causes
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
1653 # correct sort order for NULLs. Examples:
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
1654 # order_by_null_values = '(%s is not NULL)'
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
1655 # order_by_null_values = 'notnull(%s)'
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
1656 # The format parameter is replaced with the attribute.
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
1657 order_by_null_values = None
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
1658
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
1659 # Assuming DBs can do subselects, overwrite if they cannot.
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
1660 supports_subselects = True
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
1661
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1662 def schema(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1663 """ A dumpable version of the schema that we can store in the
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1664 database
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1665 """
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
1666 return (self.key,
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1667 [(x, repr(y)) for x, y in
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1668 self.properties.items() if not y.computed])
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1669
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1670 def enableJournalling(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1671 """Turn journalling on for this class
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1672 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1673 self.do_journal = 1
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1674
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1675 def disableJournalling(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1676 """Turn journalling off for this class
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1677 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1678 self.do_journal = 0
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1679
7860
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
1680 def get_case_sensitive_equal(self):
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
1681 """ For some databases (mysql) the = operator for strings ignores
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
1682 case. We define the default here, can be changed in derivative class.
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
1683
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
1684 It takes config as an argument because mysql has multiple collations.
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
1685 The admin sets both the primary and case sensitive collation in
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
1686 config.ini for mysql.
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
1687 """
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
1688 raise ValueError("get_case_sensitive_equal called in error")
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
1689
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1690 # Editing nodes:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1691 def create(self, **propvalues):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1692 """ Create a new node of this class and return its id.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1693
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1694 The keyword arguments in 'propvalues' map property names to values.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1695
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1696 The values of arguments must be acceptable for the types of their
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1697 corresponding properties or a TypeError is raised.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1698
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1699 If this class has a key property, it must be present and its value
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1700 must not collide with other key strings or a ValueError is raised.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1701
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1702 Any other properties on this class that are missing from the
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1703 'propvalues' dictionary are set to None.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1704
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1705 If an id in a link or multilink property does not refer to a valid
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1706 node, an IndexError is raised.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1707 """
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
1708 self.fireAuditors('create', None, propvalues)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
1709 newid = self.create_inner(**propvalues)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
1710 self.fireReactors('create', newid, None)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
1711 return newid
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1712
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
1713 def create_inner(self, **propvalues):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1714 """ Called by create, in-between the audit and react calls.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1715 """
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1716 if 'id' in propvalues:
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1717 raise KeyError('"id" is reserved')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1718
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1719 if self.db.journaltag is None:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1720 raise DatabaseError(_('Database open read-only'))
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1721
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1722 if ('creator' in propvalues or 'actor' in propvalues or
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1723 'creation' in propvalues or 'activity' in propvalues):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1724 raise KeyError('"creator", "actor", "creation" and '
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1725 '"activity" are reserved')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1726
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
1727 for p in propvalues:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
1728 prop = self.properties[p]
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
1729 if prop.computed:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1730 raise KeyError('"%s" is a computed property' % p)
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
1731
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1732 # new node's id
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1733 newid = self.db.newid(self.classname)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1734
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1735 # validate propvalues
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
1736 num_re = re.compile(r'^\d+$')
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1737 for key, value in propvalues.items():
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1738 if key == self.key:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1739 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1740 self.lookup(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1741 except KeyError:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1742 pass
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1743 else:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1744 raise ValueError('node with key "%s" exists' % value)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1745
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1746 # try to handle this property
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1747 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1748 prop = self.properties[key]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1749 except KeyError:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1750 raise KeyError('"%s" has no property "%s"' % (self.classname,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1751 key))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1752
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1753 if value is not None and isinstance(prop, Link):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1754 if not isinstance(value, type('')):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1755 raise ValueError('link value must be String')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1756 link_class = self.properties[key].classname
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1757 # if it isn't a number, it's a key
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1758 if not num_re.match(value):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1759 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1760 value = self.db.classes[link_class].lookup(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1761 except (TypeError, KeyError):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1762 raise IndexError('new property "%s": %s not a %s' % (
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1763 key, value, link_class))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1764 elif not self.db.getclass(link_class).hasnode(value):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1765 raise IndexError('%s has no node %s' % (link_class,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1766 value))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1767
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1768 # save off the value
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1769 propvalues[key] = value
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1770
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1771 # register the link with the newly linked node
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1772 if self.do_journal and self.properties[key].do_journal:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1773 self.db.addjournal(link_class, value, 'link',
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1774 (self.classname, newid, key))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1775
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1776 elif isinstance(prop, Multilink):
3872
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
1777 if value is None:
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
1778 value = []
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1779 if not hasattr(value, '__iter__') or \
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1780 isinstance(value, type('')):
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1781 raise TypeError('new property "%s" not an iterable of ids'
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1782 % key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1783 # clean up and validate the list of links
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1784 link_class = self.properties[key].classname
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1785 l = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1786 for entry in value:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1787 if not isinstance(entry, type('')):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1788 raise ValueError('"%s" multilink value (%r) '
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1789 'must contain Strings' % (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1790 key, value))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1791 # if it isn't a number, it's a key
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1792 if not num_re.match(entry):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1793 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1794 entry = self.db.classes[link_class].lookup(entry)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1795 except (TypeError, KeyError):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1796 raise IndexError('new property "%s": %s not a %s' % (
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1797 key, entry, self.properties[key].classname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1798 l.append(entry)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1799 value = l
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1800 propvalues[key] = value
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1801
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1802 # handle additions
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1803 for nodeid in value:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1804 if not self.db.getclass(link_class).hasnode(nodeid):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1805 raise IndexError('%s has no node %s' % (link_class,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1806 nodeid))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1807 # register the link with the newly linked node
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1808 if self.do_journal and self.properties[key].do_journal:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1809 self.db.addjournal(link_class, nodeid, 'link',
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1810 (self.classname, newid, key))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1811
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1812 elif isinstance(prop, String):
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1813 if not isinstance(value, (str, unicode)):
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1814 raise TypeError('new property "%s" not a string' % key)
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2887
diff changeset
1815 if prop.indexme:
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2887
diff changeset
1816 self.db.indexer.add_text((self.classname, newid, key),
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1817 value)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1818
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1819 elif isinstance(prop, Password):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1820 if not isinstance(value, password.Password):
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1821 raise TypeError('new property "%s" not a Password' % key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1822
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1823 elif isinstance(prop, Date):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1824 if value is not None and not isinstance(value, date.Date):
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1825 raise TypeError('new property "%s" not a Date' % key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1826
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1827 elif isinstance(prop, Interval):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1828 if value is not None and not isinstance(value, date.Interval):
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1829 raise TypeError('new property "%s" not an Interval' % key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1830
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1831 elif value is not None and isinstance(prop, Number):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1832 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1833 float(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1834 except ValueError:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1835 raise TypeError('new property "%s" not numeric' % key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1836
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1837 elif value is not None and isinstance(prop, Integer):
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1838 try:
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1839 int(value)
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1840 except ValueError:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1841 raise TypeError('new property "%s" not integer' % key)
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1842
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1843 elif value is not None and isinstance(prop, Boolean):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1844 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1845 int(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1846 except ValueError:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1847 raise TypeError('new property "%s" not boolean' % key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1848
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1849 # make sure there's data where there needs to be
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1850 for key, prop in self.properties.items():
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1851 if key in propvalues:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1852 continue
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1853 if key == self.key:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1854 raise ValueError('key property "%s" is required' % key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1855 if isinstance(prop, Multilink):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1856 propvalues[key] = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1857 else:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1858 propvalues[key] = None
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1859
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1860 # done
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1861 self.db.addnode(self.classname, newid, propvalues)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1862 if self.do_journal:
2546
1b07c9740bba mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2539
diff changeset
1863 self.db.addjournal(self.classname, newid, ''"create", {})
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1864
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1865 # XXX numeric ids
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1866 return str(newid)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1867
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1868 def get(self, nodeid, propname, default=_marker, cache=1):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1869 """Get the value of a property on an existing node of this class.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1870
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1871 'nodeid' must be the id of an existing node of this class or an
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1872 IndexError is raised. 'propname' must be the name of a property
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1873 of this class or a KeyError is raised.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1874
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
1875 'cache' exists for backwards compatibility, and is not used.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1876 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1877 if propname == 'id':
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1878 return nodeid
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1879
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1880 # get the node's dict
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1881 d = self.db.getnode(self.classname, nodeid, fetch_multilinks=False)
4476
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1882 # handle common case -- that property is in dict -- first
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1883 # if None and one of creator/creation actor/activity return None
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1884 if propname in d:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1885 r = d[propname]
4476
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1886 # return copy of our list
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1887 if isinstance(r, list):
4476
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1888 return r[:]
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1889 if r is not None:
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1890 return r
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1891 elif propname in ('creation', 'activity', 'creator', 'actor'):
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1892 return r
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1893
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1894 # propname not in d:
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1895 if propname == 'creation' or propname == 'activity':
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1896 return date.Date()
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1897 if propname == 'creator' or propname == 'actor':
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1898 return self.db.getuid()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1899
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1900 # get the property (raises KeyError if invalid)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1901 prop = self.properties[propname]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1902
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1903 # lazy evaluation of Multilink
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1904 if propname not in d and isinstance(prop, Multilink):
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1905 self.db._materialize_multilink(self.classname, nodeid, d, propname)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1906
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1907 # handle there being no value in the table for the property
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1908 if propname not in d or d[propname] is None:
2692
f1c9873496f0 fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2691
diff changeset
1909 if default is _marker:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1910 if isinstance(prop, Multilink):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1911 return []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1912 else:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1913 return None
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1914 else:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1915 return default
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1916
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1917 # don't pass our list to other code
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1918 if isinstance(prop, Multilink):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1919 return d[propname][:]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1920
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1921 return d[propname]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1922
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1923 def set(self, nodeid, **propvalues):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1924 """Modify a property on an existing node of this class.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1925
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1926 'nodeid' must be the id of an existing node of this class or an
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1927 IndexError is raised.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1928
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1929 Each key in 'propvalues' must be the name of a property of this
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1930 class or a KeyError is raised.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1931
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1932 All values in 'propvalues' must be acceptable types for their
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1933 corresponding properties or a TypeError is raised.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1934
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1935 If the value of the key property is set, it must not collide with
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1936 other key strings or a ValueError is raised.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1937
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1938 If the value of a Link or Multilink property contains an invalid
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1939 node id, a ValueError is raised.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1940 """
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1941 self.fireAuditors('set', nodeid, propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1942 oldvalues = copy.deepcopy(self.db.getnode(self.classname, nodeid))
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1943 propvalues = self.set_inner(nodeid, **propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1944 self.fireReactors('set', nodeid, oldvalues)
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1945 return propvalues
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1946
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1947 def set_inner(self, nodeid, **propvalues):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1948 """ Called by set, in-between the audit and react calls.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1949 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1950 if not propvalues:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1951 return propvalues
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1952
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1953 if ('creator' in propvalues or 'actor' in propvalues or
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1954 'creation' in propvalues or 'activity' in propvalues):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1955 raise KeyError('"creator", "actor", "creation" and '
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1956 '"activity" are reserved')
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1957
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1958 if 'id' in propvalues:
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1959 raise KeyError('"id" is reserved')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1960
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
1961 for p in propvalues:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
1962 prop = self.properties[p]
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
1963 if prop.computed:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1964 raise KeyError('"%s" is a computed property' % p)
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
1965
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1966 if self.db.journaltag is None:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1967 raise DatabaseError(_('Database open read-only'))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1968
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1969 node = self.db.getnode(self.classname, nodeid)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1970 if self.is_retired(nodeid):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1971 raise IndexError('Requested item is retired')
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
1972 num_re = re.compile(r'^\d+$')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1973
3082
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
1974 # make a copy of the values dictionary - we'll modify the contents
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
1975 propvalues = propvalues.copy()
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
1976
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1977 # if the journal value is to be different, store it in here
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1978 journalvalues = {}
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1979
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1980 # remember the add/remove stuff for multilinks, making it easier
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1981 # for the Database layer to do its stuff
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1982 multilink_changes = {}
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1983
5112
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
1984 # omit quiet properties from history/changelog
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
1985 quiet_props = []
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
1986
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1987 for propname, value in list(propvalues.items()):
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1988 # check to make sure we're not duplicating an existing key
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1989 if propname == self.key and node[propname] != value:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1990 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1991 self.lookup(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1992 except KeyError:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1993 pass
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1994 else:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
1995 raise ValueError('node with key "%s" exists' % value)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1996
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1997 # this will raise the KeyError if the property isn't valid
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1998 # ... we don't use getprops() here because we only care about
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1999 # the writeable properties.
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
2000 try:
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
2001 prop = self.properties[propname]
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
2002 except KeyError:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2003 raise KeyError('"%s" has no property named "%s"' % (
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2004 self.classname, propname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2005
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2006 # if the value's the same as the existing value, no sense in
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2007 # doing anything
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1295
diff changeset
2008 current = node.get(propname, None)
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1295
diff changeset
2009 if value == current:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2010 del propvalues[propname]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2011 continue
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1295
diff changeset
2012 journalvalues[propname] = current
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2013
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2014 # do stuff based on the prop type
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2015 if isinstance(prop, Link):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2016 link_class = prop.classname
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2017 # if it isn't a number, it's a key
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2018 if value is not None and not isinstance(value, str):
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2019 raise ValueError('property "%s" link value be a string' % (
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2020 propname))
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2021 if isinstance(value, str) and not num_re.match(value):
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2022 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2023 value = self.db.classes[link_class].lookup(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2024 except (TypeError, KeyError):
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2025 raise IndexError('new property "%s": %s not a %s' % (
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2026 propname, value, prop.classname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2027
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2028 if (value is not None and
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2029 not self.db.getclass(link_class).hasnode(value)):
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2030 raise IndexError('%s has no node %s' % (link_class,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2031 value))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2032
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2033 if self.do_journal and prop.do_journal:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2034 # register the unlink with the old linked node
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2035 if node[propname] is not None:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2036 self.db.addjournal(
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2037 link_class, node[propname],
2546
1b07c9740bba mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2539
diff changeset
2038 ''"unlink", (self.classname, nodeid, propname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2039
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2040 # register the link with the newly linked node
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2041 if value is not None:
2546
1b07c9740bba mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2539
diff changeset
2042 self.db.addjournal(link_class, value, ''"link",
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2043 (self.classname, nodeid, propname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2044
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2045 elif isinstance(prop, Multilink):
3872
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
2046 if value is None:
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
2047 value = []
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2048 if not hasattr(value, '__iter__') or isinstance(value, str):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2049 raise TypeError('new property "%s" not an iterable of'
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2050 ' ids' % propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2051 link_class = self.properties[propname].classname
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2052 l = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2053 for entry in value:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2054 # if it isn't a number, it's a key
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2055 if not isinstance(entry, str):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2056 raise ValueError('new property "%s" link value '
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2057 'must be a string' % propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2058 if not num_re.match(entry):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2059 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2060 entry = self.db.classes[link_class].lookup(entry)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2061 except (TypeError, KeyError):
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2062 raise IndexError(
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2063 'new property "%s": %s not a %s' % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2064 propname, entry,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2065 self.properties[propname].classname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2066 l.append(entry)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2067 value = l
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2068 propvalues[propname] = value
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2069
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2070 # figure the journal entry for this property
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2071 add = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2072 remove = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2073
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2074 # handle removals
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2075 if propname in node:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2076 l = node[propname]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2077 else:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2078 l = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2079 for id in l[:]:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2080 if id in value:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2081 continue
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2082 # register the unlink with the old linked node
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2083 if self.do_journal and self.properties[propname].do_journal:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2084 self.db.addjournal(link_class, id, 'unlink',
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2085 (self.classname, nodeid, propname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2086 l.remove(id)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2087 remove.append(id)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2088
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2089 # handle additions
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2090 for id in value:
4095
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
2091 if id in l:
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
2092 continue
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
2093 # We can safely check this condition after
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
2094 # checking that this is an addition to the
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
2095 # multilink since the condition was checked for
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
2096 # existing entries at the point they were added to
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
2097 # the multilink. Since the hasnode call will
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
2098 # result in a SQL query, it is more efficient to
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
2099 # avoid the check if possible.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2100 if not self.db.getclass(link_class).hasnode(id):
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2101 raise IndexError('%s has no node %s' % (link_class,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2102 id))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2103 # register the link with the newly linked node
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2104 if self.do_journal and self.properties[propname].do_journal:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2105 self.db.addjournal(link_class, id, 'link',
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2106 (self.classname, nodeid, propname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2107 l.append(id)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2108 add.append(id)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2109
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2110 # figure the journal entry
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2111 l = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2112 if add:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2113 l.append(('+', add))
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2114 if remove:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2115 l.append(('-', remove))
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2116 multilink_changes[propname] = (add, remove)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2117 if l:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2118 journalvalues[propname] = tuple(l)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2119
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2120 elif isinstance(prop, String):
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2121 if value is not None and not isinstance(value, (str, unicode)):
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2122 raise TypeError(
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2123 'new property "%s" not a string' % propname)
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2887
diff changeset
2124 if prop.indexme:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2125 if value is None: value = '' # noqa: E701
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2126 self.db.indexer.add_text(
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2127 (self.classname, nodeid, propname), value)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2128
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2129 elif isinstance(prop, Password):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2130 if not isinstance(value, password.Password):
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2131 raise TypeError(
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2132 'new property "%s" not a Password' % propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2133 propvalues[propname] = value
4483
22bc0426e348 Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4480
diff changeset
2134 journalvalues[propname] = \
22bc0426e348 Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4480
diff changeset
2135 current and password.JournalPassword(current)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2136
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2137 elif value is not None and isinstance(prop, Date):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2138 if not isinstance(value, date.Date):
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2139 raise TypeError('new property "%s" not a Date' % propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2140 propvalues[propname] = value
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2141
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2142 elif value is not None and isinstance(prop, Interval):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2143 if not isinstance(value, date.Interval):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2144 raise TypeError('new property "%s" not an '
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2145 'Interval' % propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2146 propvalues[propname] = value
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2147
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2148 elif value is not None and isinstance(prop, Number):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2149 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2150 float(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2151 except ValueError:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2152 raise TypeError('new property "%s" not numeric' % propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2153
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
2154 elif value is not None and isinstance(prop, Integer):
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
2155 try:
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
2156 int(value)
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
2157 except ValueError:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2158 raise TypeError('new property "%s" not integer' % propname)
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
2159
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2160 elif value is not None and isinstance(prop, Boolean):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2161 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2162 int(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2163 except ValueError:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2164 raise TypeError('new property "%s" not boolean' % propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2165
5112
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2166 # record quiet properties to omit from history/changelog
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2167 if prop.quiet:
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2168 quiet_props.append(propname)
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2169
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2170 # nothing to do?
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2171 if not propvalues:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2172 return propvalues
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2173
3082
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
2174 # update the activity time
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
2175 propvalues['activity'] = date.Date()
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
2176 propvalues['actor'] = self.db.getuid()
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
2177
3328
475c8560ef9b merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3310
diff changeset
2178 # do the set
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
2179 self.db.setnode(self.classname, nodeid, propvalues, multilink_changes)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2180
3328
475c8560ef9b merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3310
diff changeset
2181 # remove the activity props now they're handled
475c8560ef9b merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3310
diff changeset
2182 del propvalues['activity']
475c8560ef9b merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3310
diff changeset
2183 del propvalues['actor']
475c8560ef9b merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3310
diff changeset
2184
475c8560ef9b merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3310
diff changeset
2185 # journal the set
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2186 if self.do_journal:
2546
1b07c9740bba mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2539
diff changeset
2187 self.db.addjournal(self.classname, nodeid, ''"set", journalvalues)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2188
5112
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2189 # remove quiet properties from output
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2190 for propname in quiet_props:
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2191 if propname in propvalues:
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2192 del propvalues[propname]
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2193
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
2194 return propvalues
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2195
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2196 def retire(self, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2197 """Retire a node.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
2198
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2199 The properties on the node remain available from the get() method,
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2200 and the node's id is never reused.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
2201
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2202 Retired nodes are not returned by the find(), list(), or lookup()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2203 methods, and other nodes may reuse the values of their key properties.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2204 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2205 if self.db.journaltag is None:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2206 raise DatabaseError(_('Database open read-only'))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2207
1345
618aa9c37d65 fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents: 1333
diff changeset
2208 self.fireAuditors('retire', nodeid, None)
618aa9c37d65 fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents: 1333
diff changeset
2209
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2210 # use the arg for __retired__ to cope with any odd database type
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2211 # conversion (hello, sqlite)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2212 sql = 'update _%s set __retired__=%s where id=%s' % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2213 self.classname, self.db.arg, self.db.arg)
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2214 self.db.sql(sql, (nodeid, nodeid))
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2215 if self.do_journal:
2546
1b07c9740bba mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2539
diff changeset
2216 self.db.addjournal(self.classname, nodeid, ''"retired", None)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2217
1345
618aa9c37d65 fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents: 1333
diff changeset
2218 self.fireReactors('retire', nodeid, None)
618aa9c37d65 fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents: 1333
diff changeset
2219
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2220 def restore(self, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2221 """Restore a retired node.
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2222
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2223 Make node available for all operations like it was before retirement.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2224 """
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2225 if self.db.journaltag is None:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2226 raise DatabaseError(_('Database open read-only'))
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2227
1523
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2228 node = self.db.getnode(self.classname, nodeid)
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2229 # check if key property was overrided
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2230 key = self.getkey()
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2231 try:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2232 self.lookup(node[key])
1523
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2233 except KeyError:
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2234 pass
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2235 else:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2236 raise KeyError("Key property (%s) of retired node clashes "
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2237 "with existing one (%s)" % (key, node[key]))
1523
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2238
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2239 self.fireAuditors('restore', nodeid, None)
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2240 # use the arg for __retired__ to cope with any odd database type
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2241 # conversion (hello, sqlite)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2242 sql = 'update _%s set __retired__=%s where id=%s' % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2243 self.classname, self.db.arg, self.db.arg)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
2244 self.db.sql(sql, (0, nodeid))
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2245 if self.do_journal:
2546
1b07c9740bba mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2539
diff changeset
2246 self.db.addjournal(self.classname, nodeid, ''"restored", None)
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2247
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2248 self.fireReactors('restore', nodeid, None)
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
2249
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2250 def is_retired(self, nodeid):
8302
82a26ea1afdf issue2551376: Fix tracebacks in item templates
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2251 """Return true if the node is retired
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2252 """
8302
82a26ea1afdf issue2551376: Fix tracebacks in item templates
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2253 # Do not produce invalid sql, the id must be numeric
82a26ea1afdf issue2551376: Fix tracebacks in item templates
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2254 try:
82a26ea1afdf issue2551376: Fix tracebacks in item templates
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2255 id = int(nodeid)
82a26ea1afdf issue2551376: Fix tracebacks in item templates
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2256 except ValueError:
82a26ea1afdf issue2551376: Fix tracebacks in item templates
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2257 raise hyperdb.HyperdbValueError(_(
82a26ea1afdf issue2551376: Fix tracebacks in item templates
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2258 'class %(cls)s: %(value)r is not an id')
82a26ea1afdf issue2551376: Fix tracebacks in item templates
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2259 % {'cls': self.classname, 'value': nodeid})
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2260 sql = 'select __retired__ from _%s where id=%s' % (self.classname,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2261 self.db.arg)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
2262 self.db.sql(sql, (nodeid,))
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2263 return int(self.db.sql_fetchone()[0]) > 0
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2264
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2265 def destroy(self, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2266 """Destroy a node.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
2267
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2268 WARNING: this method should never be used except in extremely rare
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2269 situations where there could never be links to the node being
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2270 deleted
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1988
diff changeset
2271
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2272 WARNING: use retire() instead
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1988
diff changeset
2273
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2274 WARNING: the properties of this node will not be available ever again
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1988
diff changeset
2275
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2276 WARNING: really, use retire() instead
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2277
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2278 Well, I think that's enough warnings. This method exists mostly to
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2279 support the session storage of the cgi interface.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2280
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2281 The node is completely removed from the hyperdb, including all journal
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2282 entries. It will no longer be available, and will generally break code
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2283 if there are any references to the node.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2284 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2285 if self.db.journaltag is None:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2286 raise DatabaseError(_('Database open read-only'))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2287 self.db.destroynode(self.classname, nodeid)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2288
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2289 # Locating nodes:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2290 def hasnode(self, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2291 """Determine if the given nodeid actually exists
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2292 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2293 return self.db.hasnode(self.classname, nodeid)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2294
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2295 def setkey(self, propname):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2296 """Select a String property of this class to be the key property.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2297
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2298 'propname' must be the name of a String property of this class or
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2299 None, or a TypeError is raised. The values of the key property on
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2300 all existing nodes must be unique or a ValueError is raised.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2301 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2302 prop = self.getprops()[propname]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2303 if not isinstance(prop, String):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2304 raise TypeError('key properties must be String')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2305 self.key = propname
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2306
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2307 def getkey(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2308 """Return the name of the key property for this class or None."""
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2309 return self.key
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2310
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2311 def lookup(self, keyvalue):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2312 """Locate a particular node by its key property and return its id.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2313
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2314 If this class has no key property, a TypeError is raised. If the
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2315 'keyvalue' matches one of the values for the key property among
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2316 the nodes in this class, the matching node's id is returned;
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2317 otherwise a KeyError is raised.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2318 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2319 if not self.key:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2320 raise TypeError('No key property set for class %s' %
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2321 self.classname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2322
7936
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7860
diff changeset
2323 # special notation for looking up the current database user
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7860
diff changeset
2324 if keyvalue == '@current_user' and self.classname == 'user':
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7860
diff changeset
2325 keyvalue = self.db.user.get(self.db.getuid(), self.key)
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7860
diff changeset
2326
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2327 # use the arg to handle any odd database type conversion (hello,
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2328 # sqlite)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2329 sql = "select id from _%s where _%s=%s and __retired__=%s" % (
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2330 self.classname, self.key, self.db.arg, self.db.arg)
3977
732a37da3a10 Fix for postgres 8.3 compatibility (and bug) (patch [SF#2030479])
Richard Jones <richard@users.sourceforge.net>
parents: 3969
diff changeset
2331 self.db.sql(sql, (str(keyvalue), 0))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2332
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
2333 # see if there was a result that's not retired
1203
735adcbfc665 fix to SQL lookup() and retirement
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
2334 row = self.db.sql_fetchone()
735adcbfc665 fix to SQL lookup() and retirement
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
2335 if not row:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2336 raise KeyError('No key (%s) value "%s" for "%s"' % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2337 self.key, keyvalue, self.classname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2338
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2339 # return the id
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2340 # XXX numeric ids
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2341 return str(row[0])
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2342
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2343 def find(self, **propspec):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2344 """Get the ids of nodes in this class which link to the given nodes.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2345
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2346 'propspec' consists of keyword args propname=nodeid or
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2347 propname={nodeid:1, }
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2348 'propname' must be the name of a property in this class, or a
1912
2b0ab61db194 fixes for [SF#818339]
Richard Jones <richard@users.sourceforge.net>
parents: 1911
diff changeset
2349 KeyError is raised. That property must be a Link or
2b0ab61db194 fixes for [SF#818339]
Richard Jones <richard@users.sourceforge.net>
parents: 1911
diff changeset
2350 Multilink property, or a TypeError is raised.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2351
3239
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3224
diff changeset
2352 Any node in this class whose 'propname' property links to any of
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3224
diff changeset
2353 the nodeids will be returned. Examples::
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3224
diff changeset
2354
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3224
diff changeset
2355 db.issue.find(messages='1')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2356 db.issue.find(messages={'1':1,'3':1}, files={'7':1})
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2357 """
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2358 # shortcut
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2359 if not propspec:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2360 return []
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2361
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2362 # validate the args
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2363 props = self.getprops()
7038
f524ddc27af8 replace for a,b in x.items() with for a in x.keys()
John Rouillard <rouilj@ieee.org>
parents: 7007
diff changeset
2364 for propname in propspec.keys():
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2365 # check the prop is OK
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2366 prop = props[propname]
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2367 if not isinstance(prop, Link) and not isinstance(prop, Multilink):
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2368 raise TypeError("'%s' not a Link/Multilink property" %
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2369 propname)
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2370
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2371 # first, links
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2372 a = self.db.arg
2733
ef396596a24e more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents: 2731
diff changeset
2373 allvalues = ()
ef396596a24e more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents: 2731
diff changeset
2374 sql = []
1988
5660b89f8903 more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents: 1986
diff changeset
2375 where = []
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
2376 for prop, values in propspec.items():
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2377 if not isinstance(props[prop], hyperdb.Link):
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2378 continue
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2379 if isinstance(values, dict) and len(values) == 1:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2380 values = list(values)[0]
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2381 if isinstance(values, str):
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2382 allvalues += (values,)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2383 where.append('_%s = %s' % (prop, a))
1563
e2a8ce4d2317 Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents: 1557
diff changeset
2384 elif values is None:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2385 where.append('_%s is NULL' % prop)
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2386 else:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2387 values = list(values)
2494
ea7fb2f416db fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents: 2472
diff changeset
2388 s = ''
ea7fb2f416db fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents: 2472
diff changeset
2389 if None in values:
ea7fb2f416db fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents: 2472
diff changeset
2390 values.remove(None)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2391 s = '_%s is NULL or ' % prop
2494
ea7fb2f416db fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents: 2472
diff changeset
2392 allvalues += tuple(values)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2393 s += '_%s in (%s)' % (prop, ','.join([a]*len(values)))
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2394 where.append('(' + s + ')')
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2395 if where:
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2396 allvalues = (0, ) + allvalues
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2397 sql.append("""select id from _%s where __retired__=%s
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2398 and %s""" % (self.classname, a, ' and '.join(where)))
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2399
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2400 # now multilinks
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
2401 for prop, values in propspec.items():
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
2402 p = props[prop]
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
2403 if not isinstance(p, hyperdb.Multilink):
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2404 continue
1988
5660b89f8903 more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents: 1986
diff changeset
2405 if not values:
5660b89f8903 more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents: 1986
diff changeset
2406 continue
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2407 allvalues += (0, )
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
2408 tn = p.table_name
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
2409 ln = p.linkid_name
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
2410 nn = p.nodeid_name
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
2411 cn = '_' + self.classname
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
2412 ret = ''
6155
ba0cfc1a87c9 Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
2413 dis = ''
ba0cfc1a87c9 Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
2414 ord = ''
ba0cfc1a87c9 Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
2415 if p.rev_property:
ba0cfc1a87c9 Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
2416 if isinstance(p.rev_property, Link):
ba0cfc1a87c9 Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
2417 ret = 'and %s.__retired__=%s ' % (tn, a)
ba0cfc1a87c9 Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
2418 allvalues += (0, )
ba0cfc1a87c9 Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
2419 dis = 'distinct '
ba0cfc1a87c9 Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
2420 ord = ' order by %s.id' % cn
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2421 if isinstance(values, str):
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2422 allvalues += (values,)
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2423 s = a
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2424 else:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2425 allvalues += tuple(values)
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2426 s = ','.join([a]*len(values))
6155
ba0cfc1a87c9 Fix Database.find for rev_multilink
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
2427 sql.append("""select %s%s.id from %s, %s where %s.__retired__=%s
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2428 %sand %s.id = %s.%s and %s.%s in (%s)%s""" % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2429 dis, cn, cn, tn, cn, a, ret, cn, tn, nn, tn, ln, s, ord))
2733
ef396596a24e more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents: 2731
diff changeset
2430
ef396596a24e more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents: 2731
diff changeset
2431 if not sql:
1988
5660b89f8903 more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents: 1986
diff changeset
2432 return []
2733
ef396596a24e more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents: 2731
diff changeset
2433 sql = ' union '.join(sql)
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
2434 self.db.sql(sql, allvalues)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2435 # XXX numeric ids
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2436 l = [str(x[0]) for x in self.db.sql_fetchall()]
1195
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2437 return l
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2438
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2439 def stringFind(self, **requirements):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2440 """Locate a particular node by matching a set of its String
1195
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2441 properties in a caseless search.
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2442
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2443 If the property is not a String property, a TypeError is raised.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
2444
1195
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2445 The return is a list of the id of all nodes that match.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2446 """
1195
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2447 where = []
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2448 args = []
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2449 for propname in requirements:
1195
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2450 prop = self.properties[propname]
1951
767ff2a03eee more unit tests to improve coverage (up from 85% to 88% for anydbm! :)
Richard Jones <richard@users.sourceforge.net>
parents: 1926
diff changeset
2451 if not isinstance(prop, String):
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2452 raise TypeError("'%s' not a String property" % propname)
1195
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2453 where.append(propname)
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2454 args.append(requirements[propname].lower())
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2455
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2456 # generate the where clause
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2457 s = ' and '.join(['lower(_%s)=%s' % (col,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2458 self.db.arg) for col in where])
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2459 sql = 'select id from _%s where %s and __retired__=%s' % (
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
2460 self.classname, s, self.db.arg)
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2461 args.append(0)
1195
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2462 self.db.sql(sql, tuple(args))
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2463 # XXX numeric ids
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2464 l = [str(x[0]) for x in self.db.sql_fetchall()]
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2465 return l
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2466
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2467 def list(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2468 """ Return a list of the ids of the active nodes in this class.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2469 """
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1479
diff changeset
2470 return self.getnodeids(retired=0)
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1479
diff changeset
2471
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1479
diff changeset
2472 def getnodeids(self, retired=None):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2473 """ Retrieve all the ids of the nodes for a particular Class.
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1479
diff changeset
2474
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
2475 Set retired=None to get all nodes. Otherwise it'll get all the
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1479
diff changeset
2476 retired or non-retired nodes, depending on the flag.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2477 """
1707
3d627e34f18e sqlite backend now passes all tests under 2.3.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1599
diff changeset
2478 # flip the sense of the 'retired' flag if we don't want all of them
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1479
diff changeset
2479 if retired is not None:
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2480 args = (0, )
1719
eeb167fb8faf *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 1707
diff changeset
2481 if retired:
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2482 compare = '>'
1719
eeb167fb8faf *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 1707
diff changeset
2483 else:
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2484 compare = '='
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2485 sql = 'select id from _%s where __retired__%s%s' % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2486 self.classname, compare, self.db.arg)
1539
800b5896e14a fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents: 1528
diff changeset
2487 else:
800b5896e14a fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents: 1528
diff changeset
2488 args = ()
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2489 sql = 'select id from _%s' % self.classname
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
2490 self.db.sql(sql, args)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2491 # XXX numeric ids
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2492 ids = [str(x[0]) for x in self.db.cursor.fetchall()]
1539
800b5896e14a fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents: 1528
diff changeset
2493 return ids
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2494
6399
f3fcd6628c0c Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6396
diff changeset
2495 def _subselect(self, proptree, parentname=None):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2496 """Create a subselect. This is factored out because some
3633
a292054b4393 The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3612
diff changeset
2497 databases (hmm only one, so far) doesn't support subselects
a292054b4393 The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3612
diff changeset
2498 look for "I can't believe it's not a toy RDBMS" in the mysql
a292054b4393 The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3612
diff changeset
2499 backend.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2500 """
6179
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6157
diff changeset
2501 multilink_table = proptree.propclass.table_name
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2502 nodeid_name = proptree.propclass.nodeid_name
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2503 linkid_name = proptree.propclass.linkid_name
6399
f3fcd6628c0c Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6396
diff changeset
2504 if parentname is None:
f3fcd6628c0c Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6396
diff changeset
2505 parentname = '_' + proptree.parent.classname
f3fcd6628c0c Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6396
diff changeset
2506
6179
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6157
diff changeset
2507 w = ''
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6157
diff changeset
2508 if proptree.need_retired:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2509 w = ' where %s.__retired__=0' % (multilink_table)
6179
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6157
diff changeset
2510 if proptree.need_child_retired:
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6157
diff changeset
2511 tn1 = multilink_table
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6157
diff changeset
2512 tn2 = '_' + proptree.classname
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2513 w = ', %s where %s.%s=%s.id and %s.__retired__=0' % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2514 tn2, tn1, linkid_name, tn2, tn2)
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2515 return '%s.id not in (select %s from %s%s)' % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2516 parentname, nodeid_name, multilink_table, w)
3633
a292054b4393 The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3612
diff changeset
2517
6400
5ce995c33eee Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6399
diff changeset
2518 def _filter_multilink_expression_fallback(self, proptree, expr):
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2519 '''This is a fallback for database that do not support
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2520 subselects.'''
6400
5ce995c33eee Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6399
diff changeset
2521 classname = proptree.parent.uniqname
5ce995c33eee Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6399
diff changeset
2522 multilink_table = proptree.propclass.table_name
5ce995c33eee Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6399
diff changeset
2523 nid = proptree.propclass.nodeid_name
5ce995c33eee Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6399
diff changeset
2524 lid = proptree.propclass.linkid_name
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2525
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2526 is_valid = expr.evaluate
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2527
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2528 last_id, kws = None, []
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2529
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2530 ids = IdListOptimizer()
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2531 append = ids.append
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2532
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2533 # This join and the evaluation in program space
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2534 # can be expensive for larger databases!
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2535 # TODO: Find a faster way to collect the data needed
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2536 # to evalute the expression.
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2537 # Moving the expression evaluation into the database
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2538 # would be nice but this tricky: Think about the cases
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2539 # where the multilink table does not have join values
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2540 # needed in evaluation.
6402
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2541 w = j = ''
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2542 s = 'm.%s' % lid
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2543 if proptree.need_retired:
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2544 w = ' and m.__retired__=0'
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2545 elif proptree.need_child_retired:
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2546 tn2 = '_' + proptree.classname
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2547 j = ' LEFT OUTER JOIN %s ON %s.id = m.%s' % (tn2, tn2, lid)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2548 w = ' and %s.__retired__=0' % (tn2)
6402
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2549 s = '%s.id' % tn2
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2550
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2551 stmnt = "SELECT c.id, %s FROM _%s as c " \
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2552 "LEFT OUTER JOIN %s as m " \
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2553 "ON c.id = m.%s%s%s ORDER BY c.id" % (
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2554 s, classname, multilink_table, nid, j, w)
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2555 self.db.sql(stmnt)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2556
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2557 # collect all multilink items for a class item
6400
5ce995c33eee Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6399
diff changeset
2558 for nodeid, kw in self.db.sql_fetchiter():
5ce995c33eee Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6399
diff changeset
2559 if nodeid != last_id:
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2560 if last_id is None:
6400
5ce995c33eee Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6399
diff changeset
2561 last_id = nodeid
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2562 else:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2563 # we have all multilink items -> evaluate!
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2564 if is_valid(kws): append(last_id) # noqa: E701
6400
5ce995c33eee Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6399
diff changeset
2565 last_id, kws = nodeid, []
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2566 if kw is not None:
6395
8baf81d1cfc1 Make multilink expressions work with mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6392
diff changeset
2567 kws.append(int(kw))
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2568
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2569 if last_id is not None and is_valid(kws):
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2570 append(last_id)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2571
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2572 # we have ids of the classname table
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2573 return ids.where("_%s.id" % classname, self.db.arg)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2574
6415
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2575 def _filter_link_expression(self, proptree, v):
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2576 """ Filter elements in the table that match the given expression
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2577 """
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2578 pln = proptree.parent.uniqname
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2579 prp = proptree.name
7936
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7860
diff changeset
2580
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7860
diff changeset
2581 if proptree.classname == 'user' and '@current_user' in v:
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7860
diff changeset
2582 cu = self.db.getuid()
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7860
diff changeset
2583 v = [x if x != "@current_user" else cu for x in v]
6415
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2584 try:
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2585 opcodes = [int(x) for x in v]
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2586 if min(opcodes) >= -1:
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2587 raise ValueError()
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2588 expr = compile_expression(opcodes)
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2589 # NULL doesn't compare to NULL in SQL
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2590 # So not (x = '1') will *not* include NULL values for x
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2591 # That's why we need that and clause:
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2592 atom = "_%s._%s = %s and _%s._%s is not NULL" % (
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2593 pln, prp, self.db.arg, pln, prp)
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2594 atom_nil = "_%s._%s is NULL" % (pln, prp)
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2595 lambda_atom = lambda n: atom if n.x >= 0 else atom_nil
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2596 values = []
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2597 w = expr.generate(lambda_atom)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2598
6415
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2599 def collect_values(n):
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2600 if n.x >= 0:
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2601 values.append(n.x)
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2602 expr.visit(collect_values)
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2603 return w, values
8241
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 7936
diff changeset
2604 except ExpressionError as e:
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 7936
diff changeset
2605 e.context['class'] = pln
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 7936
diff changeset
2606 e.context['attr'] = prp
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 7936
diff changeset
2607 raise
6415
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2608 except:
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2609 pass
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2610 # Fallback to original code
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2611 args = []
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2612 where = None
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2613 d = {}
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2614 for entry in v:
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2615 if entry == '-1':
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2616 entry = None
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2617 d[entry] = entry
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2618 l = []
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2619 if None in d or not d:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2620 if None in d: del d[None] # noqa: E701
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2621 l.append('_%s._%s is NULL' % (pln, prp))
6415
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2622 if d:
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2623 v = list(d)
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2624 s = ','.join([self.db.arg for x in v])
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2625 l.append('(_%s._%s in (%s))' % (pln, prp, s))
6415
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2626 args = v
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2627 if l:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2628 where = '(' + ' or '.join(l) + ')'
6415
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2629 return where, args
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2630
6396
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2631 def _filter_multilink_expression(self, proptree, v):
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2632 """ Filters out elements of the classname table that do not
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2633 match the given expression.
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2634 Returns tuple of 'WHERE' introns for the overall filter.
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2635 """
6396
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2636 classname = proptree.parent.uniqname
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2637 multilink_table = proptree.propclass.table_name
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2638 nid = proptree.propclass.nodeid_name
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2639 lid = proptree.propclass.linkid_name
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2640
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2641 try:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2642 opcodes = [int(x) for x in v]
6396
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2643 if min(opcodes) >= -1:
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2644 raise ValueError()
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2645
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2646 expr = compile_expression(opcodes)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2647
6396
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2648 if not self.supports_subselects:
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2649 # We heavily rely on subselects. If there is
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2650 # no decent support fall back to slower variant.
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2651 return self._filter_multilink_expression_fallback(
6400
5ce995c33eee Make reverse multilink expressions work for mysql
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6399
diff changeset
2652 proptree, expr)
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2653
6402
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2654 w = j = ''
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2655 if proptree.need_retired:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2656 w = ' and %s.__retired__=0' % (multilink_table)
6402
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2657 elif proptree.need_child_retired:
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2658 tn1 = multilink_table
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2659 tn2 = '_' + proptree.classname
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2660 j = ', %s' % tn2
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2661 w = ' and %s.%s=%s.id and %s.__retired__=0' % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2662 tn1, lid, tn2, tn2)
6402
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2663
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2664 atom = \
6402
619807d9a2df Make rev multilink for Link work
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
2665 "%s IN(SELECT %s FROM %s%s WHERE %s=a.id%s)" % (
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2666 self.db.arg, lid, multilink_table, j, nid, w)
6399
f3fcd6628c0c Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6396
diff changeset
2667 atom_nil = self._subselect(proptree, 'a')
f3fcd6628c0c Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6396
diff changeset
2668
f3fcd6628c0c Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6396
diff changeset
2669 lambda_atom = lambda n: atom if n.x >= 0 else atom_nil
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2670
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2671 intron = \
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2672 "_%(classname)s.id in (SELECT id " \
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2673 "FROM _%(classname)s AS a WHERE %(condition)s) " % {
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2674 'classname': classname,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2675 'condition': expr.generate(lambda_atom)}
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2676
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2677 values = []
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2678
6399
f3fcd6628c0c Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6396
diff changeset
2679 def collect_values(n):
f3fcd6628c0c Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6396
diff changeset
2680 if n.x >= 0:
f3fcd6628c0c Allow '-1' (empty) in multilink expression
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6396
diff changeset
2681 values.append(n.x)
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2682 expr.visit(collect_values)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2683
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2684 return intron, values
8241
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 7936
diff changeset
2685 except ExpressionError as e:
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 7936
diff changeset
2686 e.context['class'] = classname
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 7936
diff changeset
2687 e.context['attr'] = proptree.name
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 7936
diff changeset
2688 raise
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2689 except:
6396
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2690 # fallback behavior when expression parsing above fails
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2691 orclause = ''
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2692 if '-1' in v:
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2693 v = [x for x in v if int(x) > 0]
6396
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2694 orclause = self._subselect(proptree)
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2695 where = []
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2696 where.append("%s.%s in (%s)" % (multilink_table, lid,
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2697 ','.join([self.db.arg] * len(v))))
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2698 where.append('_%s.id=%s.%s' % (classname, multilink_table, nid))
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2699 where = ' and '.join(where)
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2700 if orclause:
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2701 where = '((' + ' or '.join((where + ')', orclause)) + ')'
6396
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2702
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2703 return where, v
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2704
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2705 def _filter_sql(self, search_matches, filterspec, srt=[], grp=[], retr=0,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2706 retired=False, exact_match_spec={}, limit=None,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2707 offset=None):
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2708 """ Compute the proptree and the SQL/ARGS for a filter.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2709 For argument description see filter below.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2710 We return a 3-tuple, the proptree, the sql and the sql-args
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2711 or None if no SQL is necessary.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2712 The flag retr serves to retrieve *all* non-Multilink properties
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2713 (for filling the cache during a filter_iter)
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2714 """
2578
7f25486ff85e fixed RDBMS filter() for no matches from full-text search [SF#990778]
Richard Jones <richard@users.sourceforge.net>
parents: 2546
diff changeset
2715 # we can't match anything if search_matches is empty
4044
27a9906cd8d1 Fix issue2550505.
Stefan Seefeld <stefan@seefeld.name>
parents: 4039
diff changeset
2716 if not search_matches and search_matches is not None:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2717 return None
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
2718
3634
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2719 icn = self.classname
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2720
2379
a2025bdd1491 fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
2721 # vars to hold the components of the SQL statement
2586
3c67f4bfa225 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2578
diff changeset
2722 frum = [] # FROM clauses
2379
a2025bdd1491 fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
2723 loj = [] # LEFT OUTER JOIN clauses
a2025bdd1491 fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
2724 where = [] # WHERE clauses
a2025bdd1491 fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
2725 args = [] # *any* positional arguments
a2025bdd1491 fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
2726 a = self.db.arg
a2025bdd1491 fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
2727
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2728 # figure the WHERE clause from the filterspec
6413
7b1b6dffc7ed Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6403
diff changeset
2729 use_distinct = False # Do we need a distinct clause?
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2730 sortattr = self._sortattr(group=grp, sort=srt)
5871
acc4a128ab9b Backwards-compatible signature of _proptree method
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5869
diff changeset
2731 proptree = self._proptree(filterspec, exact_match_spec, sortattr, retr)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2732 mlseen = 0
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2733 for pt in reversed(proptree.sortattr):
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2734 p = pt
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2735 while p.parent:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2736 if isinstance(p.propclass, Multilink):
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2737 mlseen = True
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2738 if mlseen:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2739 p.sort_ids_needed = True
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2740 p.tree_sort_done = False
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2741 p = p.parent
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2742 if not mlseen:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2743 pt.attr_sort_done = pt.tree_sort_done = True
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2744 proptree.compute_sort_done()
3693
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
2745
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2746 cols = ['_%s.id' % icn]
3635
53987aa153d2 Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2747 for p in proptree:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2748 rc = ac = oc = None
3634
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2749 cn = p.classname
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2750 ln = p.uniqname
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2751 pln = p.parent.uniqname
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2752 pcn = p.parent.classname
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2753 k = p.name
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2754 v = p.val
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2755 propclass = p.propclass
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2756 if p.parent == proptree and p.name == 'id' \
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2757 and 'retrieve' in p.need_for:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2758 p.sql_idx = 0
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2759 if 'sort' in p.need_for or 'retrieve' in p.need_for:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2760 rc = oc = ac = '_%s._%s' % (pln, k)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2761 if isinstance(propclass, Multilink):
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2762 if 'search' in p.need_for:
6413
7b1b6dffc7ed Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6403
diff changeset
2763 # if we joining with Multilink tables we need distinct
7b1b6dffc7ed Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6403
diff changeset
2764 use_distinct = True
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
2765 tn = propclass.table_name
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
2766 nid = propclass.nodeid_name
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6133
diff changeset
2767 lid = propclass.linkid_name
6403
9957d8d10783 Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6402
diff changeset
2768 frum.append(tn)
9957d8d10783 Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6402
diff changeset
2769 if p.children or p.need_child_retired:
9957d8d10783 Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6402
diff changeset
2770 frum.append('_%s as _%s' % (cn, ln))
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2771 where.append('%s.%s=_%s.id' % (tn, lid, ln))
6403
9957d8d10783 Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6402
diff changeset
2772 if p.need_child_retired:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2773 where.append('_%s.__retired__=0' % (ln))
6403
9957d8d10783 Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6402
diff changeset
2774 # Note: need the where-clause if p has
9957d8d10783 Tests and bug-fix for issue2551119
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6402
diff changeset
2775 # children that compute additional restrictions
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2776 if (not p.has_values or
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2777 (not isinstance(v, type([])) and v != '-1') or
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2778 p.children):
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2779 where.append('_%s.id=%s.%s' % (pln, tn, nid))
4061
52431f2ad76b Correct handling of a Multilink filterspec with an empty list of linked-to items.
Stefan Seefeld <stefan@seefeld.name>
parents: 4060
diff changeset
2780 if v in ('-1', ['-1'], []):
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2781 # only match rows that have count(linkid)=0 in the
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2782 # corresponding multilink table)
6179
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6157
diff changeset
2783 where.append(self._subselect(p))
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2784 else:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2785 if p.has_values:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2786 if isinstance(v, type([])):
6396
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2787 # The where-clause above is conditionally
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2788 # created in _filter_multilink_expression
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2789 w, arg = self._filter_multilink_expression(p, v)
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2790 where.append(w)
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6395
diff changeset
2791 args += arg
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2792 else:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2793 where.append('%s.%s=%s' % (tn, lid, a))
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2794 args.append(v)
6179
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6157
diff changeset
2795 # Don't match retired nodes if rev_multilink
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6157
diff changeset
2796 if p.need_retired:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2797 where.append('%s.__retired__=0' % (tn))
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2798 if 'sort' in p.need_for:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2799 assert not p.attr_sort_done and not p.sort_ids_needed
1365
4884fb0860f9 fixed rdbms searching by ID [SF#666615]
Richard Jones <richard@users.sourceforge.net>
parents: 1351
diff changeset
2800 elif k == 'id':
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2801 if 'search' in p.need_for:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2802 if isinstance(v, type([])):
4462
0f4a8fcb9a1d fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents: 4448
diff changeset
2803 # If there are no permitted values, then the
0f4a8fcb9a1d fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents: 4448
diff changeset
2804 # where clause will always be false, and we
0f4a8fcb9a1d fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents: 4448
diff changeset
2805 # can optimize the query away.
0f4a8fcb9a1d fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents: 4448
diff changeset
2806 if not v:
4803
5e679e364f9a Bug-fix: need to return None for empty query
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4787
diff changeset
2807 return None
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2808 s = ','.join([a for x in v])
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2809 where.append('_%s.%s in (%s)' % (pln, k, s))
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2810 args = args + v
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2811 else:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2812 where.append('_%s.%s=%s' % (pln, k, a))
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2813 args.append(v)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2814 if 'sort' in p.need_for or 'retrieve' in p.need_for:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2815 rc = oc = ac = '_%s.id' % pln
3683
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3682
diff changeset
2816 elif isinstance(propclass, String):
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2817 if 'search' in p.need_for:
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2818 exact = []
3683
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3682
diff changeset
2819 if not isinstance(v, type([])):
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3682
diff changeset
2820 v = [v]
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2821 new_v = []
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2822 for x in v:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2823 if isinstance(x, hyperdb.Exact_Match):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2824 exact.append(True)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2825 new_v.append(x.value)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2826 else:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2827 exact.append(False)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2828 # Quote special search characters '%' and '_' for
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2829 # correct matching with LIKE/ILIKE
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2830 # Note that we now pass the elements of v as query
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2831 # arguments and don't interpolate the quoted string
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2832 # into the sql statement. Should be safer.
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2833 new_v.append(self.db.search_stringquote(x))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2834 v = new_v
3683
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3682
diff changeset
2835
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3682
diff changeset
2836 # now add to the where clause
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2837 w = []
7860
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
2838 if not self.case_sensitive_equal:
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
2839 self.case_sensitive_equal = \
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
2840 self.get_case_sensitive_equal()
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
2841 cse = self.case_sensitive_equal
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2842 for vv, ex in zip(v, exact):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2843 if ex:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2844 w.append("_%s._%s %s %s" % (
7860
8b31893f5930 issue2551115/issue2551282 - utf8mb4 support in roundup
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
2845 pln, k, cse, a))
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2846 args.append(vv)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2847 else:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2848 w.append("_%s._%s %s %s ESCAPE %s" % (
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2849 pln, k, self.case_insensitive_like, a, a))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2850 args.extend((vv, '\\'))
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2851 where.append('(' + ' and '.join(w) + ')')
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2852 if 'sort' in p.need_for:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2853 oc = ac = 'lower(_%s._%s)' % (pln, k)
1170
af104fa52746 Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents: 1168
diff changeset
2854 elif isinstance(propclass, Link):
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2855 if 'search' in p.need_for:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2856 if p.children:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2857 if 'sort' not in p.need_for:
3694
595041d78104 Fix a peculiarity in sql generation.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3693
diff changeset
2858 frum.append('_%s as _%s' % (cn, ln))
6413
7b1b6dffc7ed Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6403
diff changeset
2859 c = [x for x in p.children if 'search' in x.need_for]
7b1b6dffc7ed Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6403
diff changeset
2860 if c:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2861 where.append('_%s._%s=_%s.id' % (pln, k, ln))
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2862 if p.has_values:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2863 if isinstance(v, type([])):
6415
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2864 w, arg = self._filter_link_expression(p, v)
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2865 if w:
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2866 where.append(w)
dbacf6bf2a2f Implement expressions for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6413
diff changeset
2867 args += arg
3636
fa7becc62534 Of course for Links there is the same bug as I fixed before for multilinks.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3635
diff changeset
2868 else:
7936
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7860
diff changeset
2869 if v == '@current_user' and \
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7860
diff changeset
2870 propclass.classname == 'user':
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7860
diff changeset
2871 v = self.db.getuid()
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2872 if v in ('-1', None):
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2873 v = None
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2874 where.append('_%s._%s is NULL' % (pln, k))
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2875 else:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2876 where.append('_%s._%s=%s' % (pln, k, a))
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2877 args.append(v)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2878 if 'sort' in p.need_for:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2879 lp = p.cls.labelprop()
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2880 oc = ac = '_%s._%s' % (pln, k)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2881 if lp != 'id':
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2882 if p.tree_sort_done:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2883 loj.append(
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2884 'LEFT OUTER JOIN _%s as _%s on _%s._%s=_%s.id' % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2885 cn, ln, pln, k, ln))
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2886 oc = '_%s._%s' % (ln, lp)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2887 if 'retrieve' in p.need_for:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2888 rc = '_%s._%s' % (pln, k)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2889 elif isinstance(propclass, Date) and 'search' in p.need_for:
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
2890 dc = self.db.to_sql_value(hyperdb.Date)
1351
d1bfb479e527 fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents: 1345
diff changeset
2891 if isinstance(v, type([])):
d1bfb479e527 fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents: 1345
diff changeset
2892 s = ','.join([a for x in v])
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2893 where.append('_%s._%s in (%s)' % (pln, k, s))
3633
a292054b4393 The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3612
diff changeset
2894 args = args + [dc(date.Date(x)) for x in v]
1351
d1bfb479e527 fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents: 1345
diff changeset
2895 else:
1499
8ee69708da0c added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1496
diff changeset
2896 try:
6118
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2897 wh = []
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2898 ar = []
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2899 for d in v.split(','):
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2900 w1 = []
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2901 if d == '-':
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2902 wh.append('_%s._%s is NULL' % (pln, k))
6118
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2903 continue
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2904 # Try to filter on range of dates
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2905 date_rng = propclass.range_from_raw(d, self.db)
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2906 if date_rng.from_value:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2907 w1.append('_%s._%s >= %s' % (pln, k, a))
6118
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2908 ar.append(dc(date_rng.from_value))
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2909 if date_rng.to_value:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2910 w1.append('_%s._%s <= %s' % (pln, k, a))
6118
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2911 ar.append(dc(date_rng.to_value))
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2912 wh.append(' and '.join(w1))
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2913 where.append('(' + ' or '.join(wh) + ')')
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2914 args.extend(ar)
1499
8ee69708da0c added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1496
diff changeset
2915 except ValueError:
8ee69708da0c added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1496
diff changeset
2916 # If range creation fails - ignore that search parameter
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
2917 pass
1351
d1bfb479e527 fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents: 1345
diff changeset
2918 elif isinstance(propclass, Interval):
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2919 # filter/sort using the __<prop>_int__ column
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2920 if 'search' in p.need_for:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2921 if isinstance(v, type([])):
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2922 s = ','.join([a for x in v])
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2923 where.append('_%s.__%s_int__ in (%s)' % (pln, k, s))
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2924 args = args + [date.Interval(x).as_seconds() for x in v]
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2925 else:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2926 try:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2927 # Try to filter on range of intervals
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2928 date_rng = Range(v, date.Interval)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2929 if date_rng.from_value:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2930 where.append('_%s.__%s_int__ >= %s' % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2931 pln, k, a))
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2932 args.append(date_rng.from_value.as_seconds())
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2933 if date_rng.to_value:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2934 where.append('_%s.__%s_int__ <= %s' % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2935 pln, k, a))
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2936 args.append(date_rng.to_value.as_seconds())
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2937 except ValueError:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2938 # If range creation fails - ignore search parameter
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2939 pass
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2940 if 'sort' in p.need_for:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2941 oc = ac = '_%s.__%s_int__' % (pln, k)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2942 if 'retrieve' in p.need_for:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2943 rc = '_%s._%s' % (pln, k)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2944 elif isinstance(propclass, Boolean) and 'search' in p.need_for:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2945 if isinstance(v, str):
4365
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2946 v = v.split(',')
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2947 if not isinstance(v, list):
4365
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2948 v = [v]
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2949 bv = []
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2950 for val in v:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2951 if isinstance(val, str):
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2952 bv.append(propclass.from_raw(val))
4365
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2953 else:
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2954 bv.append(bool(val))
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2955 if len(bv) == 1:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2956 where.append('_%s._%s=%s' % (pln, k, a))
4365
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2957 args = args + bv
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2958 else:
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2959 s = ','.join([a for x in v])
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2960 where.append('_%s._%s in (%s)' % (pln, k, s))
4365
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2961 args = args + bv
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2962 elif 'search' in p.need_for:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2963 if isinstance(v, type([])):
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
2964 s = ','.join([a for x in v])
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2965 where.append('_%s._%s in (%s)' % (pln, k, s))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2966 args = args + v
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2967 else:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2968 where.append('_%s._%s=%s' % (pln, k, a))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2969 args.append(v)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2970 if oc:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2971 if p.sort_ids_needed:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2972 if rc == ac:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2973 p.sql_idx = len(cols)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2974 p.auxcol = len(cols)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2975 cols.append(ac)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2976 if p.tree_sort_done and p.sort_direction:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2977 # Don't select top-level id or multilink twice
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2978 if (not p.sort_ids_needed or ac != oc) and (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2979 p.name != 'id' or p.parent != proptree):
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2980 if rc == oc:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2981 p.sql_idx = len(cols)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2982 cols.append(oc)
3685
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2983 desc = ['', ' desc'][p.sort_direction == '-']
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2984 # Some SQL dbs sort NULL values last -- we want them first.
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2985 if (self.order_by_null_values and p.name != 'id'):
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2986 nv = self.order_by_null_values % oc
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2987 cols.append(nv)
3693
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
2988 p.orderby.append(nv + desc)
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
2989 p.orderby.append(oc + desc)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2990 if 'retrieve' in p.need_for and p.sql_idx is None:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2991 assert(rc)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2992 p.sql_idx = len(cols)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
2993 cols.append(rc)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2994
3634
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2995 props = self.getprops()
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2996
1740
5ca448ff8052 don't have RDBMS backends list retired nodes [SF#767319]
Richard Jones <richard@users.sourceforge.net>
parents: 1719
diff changeset
2997 # don't match retired nodes
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
2998 if retired is not None:
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
2999 op = '='
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
3000 if retired:
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
3001 op = '!='
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3002 where.append('_%s.__retired__%s0' % (icn, op))
1740
5ca448ff8052 don't have RDBMS backends list retired nodes [SF#767319]
Richard Jones <richard@users.sourceforge.net>
parents: 1719
diff changeset
3003
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3004 # add results of full text search
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3005 if search_matches is not None:
4044
27a9906cd8d1 Fix issue2550505.
Stefan Seefeld <stefan@seefeld.name>
parents: 4039
diff changeset
3006 s = ','.join([a for x in search_matches])
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3007 where.append('_%s.id in (%s)' % (icn, s))
4113
f4641359b892 Fix issue2550505
Stefan Seefeld <stefan@seefeld.name>
parents: 4095
diff changeset
3008 args = args + [x for x in search_matches]
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3009
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3010 # construct the SQL
3634
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
3011 frum.append('_'+icn)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3012 frum = ','.join(frum)
1170
af104fa52746 Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents: 1168
diff changeset
3013 if where:
af104fa52746 Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents: 1168
diff changeset
3014 where = ' where ' + (' and '.join(where))
af104fa52746 Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents: 1168
diff changeset
3015 else:
af104fa52746 Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents: 1168
diff changeset
3016 where = ''
6413
7b1b6dffc7ed Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6403
diff changeset
3017 if use_distinct:
7b1b6dffc7ed Fix searching+sorting for Link properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6403
diff changeset
3018 # Avoid dupes
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3019 cols[0] = 'distinct(_%s.id)' % icn
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3020
3693
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
3021 order = []
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
3022 # keep correct sequence of order attributes.
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
3023 for sa in proptree.sortattr:
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
3024 if not sa.attr_sort_done:
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
3025 continue
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
3026 order.extend(sa.orderby)
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
3027 if order:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3028 order = ' order by %s' % (','.join(order))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3029 else:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3030 order = ''
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
3031
5869
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
3032 if limit is not None:
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
3033 limit = ' LIMIT %s' % limit
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
3034 else:
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
3035 limit = ''
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
3036 if offset is not None:
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
3037 offset = ' OFFSET %s' % offset
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
3038 else:
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
3039 offset = ''
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3040 cols = ','.join(cols)
2379
a2025bdd1491 fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
3041 loj = ' '.join(loj)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3042 sql = 'select %s from %s %s %s%s%s%s' % (
5869
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
3043 cols, frum, loj, where, order, limit, offset)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3044 args = tuple(args)
2319
7cf7e3bd1b31 more column uniqueness fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2318
diff changeset
3045 __traceback_info__ = (sql, args)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3046 return proptree, sql, args
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3047
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
3048 def filter(self, search_matches, filterspec, sort=[], group=[],
5869
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
3049 retired=False, exact_match_spec={}, limit=None, offset=None):
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3050 """Return a list of the ids of the active nodes in this class that
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3051 match the 'filter' spec, sorted by the group spec and then the
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3052 sort spec
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3053
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3054 "filterspec" is {propname: value(s)}
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3055
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3056 "sort" and "group" are [(dir, prop), ...] where dir is '+', '-'
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3057 or None and prop is a prop name or None. Note that for
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3058 backward-compatibility reasons a single (dir, prop) tuple is
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3059 also allowed.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3060
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3061 "search_matches" is a container type or None
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3062
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3063 The filter must match all properties specificed. If the property
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3064 value to match is a list:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3065
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3066 1. String properties must match all elements in the list, and
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3067 2. Other properties must match any of the elements in the list.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3068 """
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3069 if __debug__:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3070 start_t = time.time()
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3071
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3072 sq = self._filter_sql(search_matches, filterspec, sort, group,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3073 retired=retired,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3074 exact_match_spec=exact_match_spec,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3075 limit=limit, offset=offset)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3076 # nothing to match?
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3077 if sq is None:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3078 return []
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3079 proptree, sql, args = sq
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3080
6332
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3081 cursor = self.db.sql_new_cursor(name='filter')
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3082 self.db.sql(sql, args, cursor)
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3083 # Reduce this to only the first row (the ID), this can save a
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3084 # lot of space for large query results (not using fetchall)
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3085 # We cannot do this if sorting by multilink
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3086 if proptree.tree_sort_done:
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3087 l = [str(row[0]) for row in cursor]
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3088 else:
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3089 l = cursor.fetchall()
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3090 cursor.close()
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3091
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3092 # Multilink sorting
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
3093 # Compute values needed for sorting in proptree.sort
6332
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3094 if not proptree.tree_sort_done:
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3095 for p in proptree:
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3096 if hasattr(p, 'auxcol'):
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3097 p.sort_ids = [row[p.auxcol] for row in l]
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3098 p.sort_result = p._sort_repr(
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3099 p.propclass.sort_repr, p.sort_ids)
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3100 l = proptree.sort([str(row[0]) for row in l])
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
3101
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
3102 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
3103 self.db.stats['filtering'] += (time.time() - start_t)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
3104 return l
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
3105
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
3106 def filter_iter(self, search_matches, filterspec, sort=[], group=[],
5869
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
3107 retired=False, exact_match_spec={}, limit=None,
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
3108 offset=None):
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3109 """Iterator similar to filter above with same args.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3110 Limitation: We don't sort on multilinks.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3111 This uses an optimisation: We put all nodes that are in the
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3112 current row into the node cache. Then we return the node id.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3113 That way a fetch of a node won't create another sql-fetch (with
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3114 a join) from the database because the nodes are already in the
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3115 cache. We're using our own temporary cursor.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3116 """
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
3117 sq = self._filter_sql(search_matches, filterspec, sort, group, retr=1,
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
3118 retired=retired,
5869
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
3119 exact_match_spec=exact_match_spec,
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
3120 limit=limit, offset=offset)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3121 # nothing to match?
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3122 if sq is None:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3123 return
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3124 proptree, sql, args = sq
6332
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3125 cursor = self.db.sql_new_cursor(name='filter_iter')
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3126 self.db.sql(sql, args, cursor)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3127 classes = {}
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3128 for p in proptree:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3129 if 'retrieve' in p.need_for:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3130 cn = p.parent.classname
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3131 ptid = p.parent.id # not the nodeid!
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3132 key = (cn, ptid)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3133 if key not in classes:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3134 classes[key] = {}
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3135 name = p.name
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3136 assert (name)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3137 classes[key][name] = p
4473
fccf7e09af0c - optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4472
diff changeset
3138 p.to_hyperdb = self.db.to_hyperdb_value(p.propclass.__class__)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3139 while True:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3140 row = cursor.fetchone()
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3141 if not row: break # noqa: E701
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3142 # populate cache with current items
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3143 for (classname, _ptid), pt in classes.items():
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3144 nodeid = str(row[pt['id'].sql_idx])
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3145 key = (classname, nodeid)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3146 if key in self.db.cache:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3147 self.db._cache_refresh(key)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3148 continue
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3149 node = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
3150 for propname, p in pt.items():
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3151 value = row[p.sql_idx]
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3152 if value is not None:
4473
fccf7e09af0c - optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4472
diff changeset
3153 value = p.to_hyperdb(value)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3154 node[propname] = value
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3155 self.db._cache_save(key, node)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3156 yield str(row[0])
6332
6a6b4651be1f Use server-side cursor for postgres in some cases
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6238
diff changeset
3157 cursor.close()
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
3158
3688
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
3159 def filter_sql(self, sql):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3160 """Return a list of the ids of the items in this class that match
3688
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
3161 the SQL provided. The SQL is a complete "select" statement.
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
3162
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
3163 The SQL select must include the item id as the first column.
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
3164
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
3165 This function DOES NOT filter out retired items, add on a where
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
3166 clause "__retired__=0" if you don't want retired nodes.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3167 """
3688
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
3168 if __debug__:
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
3169 start_t = time.time()
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
3170
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
3171 self.db.sql(sql)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3172 results = self.db.sql_fetchall()
3688
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
3173
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
3174 if __debug__:
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
3175 self.db.stats['filtering'] += (time.time() - start_t)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3176 return results
3688
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
3177
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3178 def count(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3179 """Get the number of nodes in this class.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3180
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3181 If the returned integer is 'numnodes', the ids of all the nodes
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3182 in this class run from 1 to numnodes, and numnodes+1 will be the
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3183 id of the next node to be created in this class.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3184 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3185 return self.db.countnodes(self.classname)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3186
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3187 # Manipulating properties:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3188 def getprops(self, protected=1):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3189 """Return a dictionary mapping property names to property objects.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3190 If the "protected" flag is true, we include protected properties -
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3191 those which may not be modified.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3192 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3193 d = self.properties.copy()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3194 if protected:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3195 d['id'] = String()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3196 d['creation'] = hyperdb.Date()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3197 d['activity'] = hyperdb.Date()
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1175
diff changeset
3198 d['creator'] = hyperdb.Link('user')
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
3199 d['actor'] = hyperdb.Link('user')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3200 return d
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3201
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3202 def addprop(self, **properties):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3203 """Add properties to this class.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3204
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3205 The keyword arguments in 'properties' must map names to property
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3206 objects, or a TypeError is raised. None of the keys in 'properties'
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3207 may collide with the names of existing properties, or a ValueError
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3208 is raised before any properties have been added.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3209 """
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3210 for key in properties:
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3211 if key in self.properties:
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3212 raise ValueError(key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3213 self.properties.update(properties)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3214
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3215 def index(self, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3216 """Add (or refresh) the node to search indexes
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3217 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3218 # find all the String properties that have indexme
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
3219 for prop, propclass in self.getprops().items():
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3220 if isinstance(propclass, String) and propclass.indexme:
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3221 self.db.indexer.add_text((self.classname, nodeid, prop),
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3222 str(self.get(nodeid, prop)))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3223
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3224 #
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3225 # import / export support
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3226 #
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3227 def export_list(self, propnames, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3228 """ Export a node - generate a list of CSV-able data in the order
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3229 specified by propnames for the given node.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3230 """
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3231 properties = self.getprops()
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3232 return_list = []
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3233 for prop in propnames:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3234 proptype = properties[prop]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3235 value = self.get(nodeid, prop)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3236 # "marshal" data where needed
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3237 if value is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3238 pass
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3239 elif isinstance(proptype, hyperdb.Date):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3240 value = value.get_tuple()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3241 elif isinstance(proptype, hyperdb.Interval):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3242 value = value.get_tuple()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3243 elif isinstance(proptype, hyperdb.Password):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3244 value = str(value)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3245 return_list.append(repr_export(value))
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3246 return_list.append(repr_export(self.is_retired(nodeid)))
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3247 return return_list
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3248
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3249 def import_list(self, propnames, proplist):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3250 """ Import a node - all information including "id" is present and
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3251 should not be sanity checked. Triggers are not triggered. The
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3252 journal should be initialised using the "creator" and "created"
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3253 information.
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3254
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3255 Return the nodeid of the node imported.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3256 """
6431
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3257
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3258 logger = logging.getLogger('roundup.hyperdb.backend')
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3259
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3260 if self.db.journaltag is None:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3261 raise DatabaseError(_('Database open read-only'))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3262 properties = self.getprops()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3263
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3264 # make the new node's property map
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3265 d = {}
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3266 retire = 0
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3267 if "id" not in propnames:
2505
bdd112cf61ba rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
3268 newid = self.db.newid(self.classname)
bdd112cf61ba rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
3269 else:
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
3270 newid = eval_import(proplist[propnames.index("id")])
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3271 for i in range(len(propnames)):
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
3272 # Use eval_import to reverse the repr_export() used to
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
3273 # output the CSV
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
3274 value = eval_import(proplist[i])
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3275
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3276 # Figure the property for this column
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3277 propname = propnames[i]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3278
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3279 # "unmarshal" where necessary
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3280 if propname == 'id':
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3281 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3282 elif propname == 'is retired':
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3283 # is the item retired?
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3284 if int(value):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3285 retire = 1
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3286 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3287 elif value is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3288 d[propname] = None
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3289 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3290
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3291 prop = properties[propname]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3292 if value is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3293 # don't set Nones
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3294 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3295 elif isinstance(prop, hyperdb.Date):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3296 value = date.Date(value)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3297 elif isinstance(prop, hyperdb.Interval):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3298 value = date.Interval(value)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3299 elif isinstance(prop, hyperdb.Password):
7211
506c86823abb Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents: 7038
diff changeset
3300 value = password.Password(encrypted=value,
506c86823abb Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents: 7038
diff changeset
3301 config=self.db.config)
3544
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
3302 elif isinstance(prop, String):
5416
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5395
diff changeset
3303 value = us2s(value)
3544
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
3304 if not isinstance(value, str):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3305 raise TypeError('new property "%(propname)s" not a '
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3306 'string: %(value)r' % locals())
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2887
diff changeset
3307 if prop.indexme:
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2887
diff changeset
3308 self.db.indexer.add_text((self.classname, newid, propname),
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3309 value)
3544
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
3310 d[propname] = value
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3311
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3312 # get a new id if necessary
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
3313 if newid is None:
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3314 newid = self.db.newid(self.classname)
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
3315
6431
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3316 activeid = None
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3317 has_node = False
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3318
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3319 # use the arg for __retired__ to cope with any odd database type
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3320 # conversion (hello, sqlite)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3321 retired_sql = 'update _%s set __retired__=%s where id=%s' % (
6431
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3322 self.classname, self.db.arg, self.db.arg)
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3323
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
3324 # insert new node or update existing?
6431
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3325 # if integrity error raised try to recover
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3326 try:
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3327 has_node = self.hasnode(newid)
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3328 if not has_node:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3329 self.db.addnode(self.classname, newid, d) # insert
6431
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3330 else:
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3331 self.db.setnode(self.classname, newid, d) # update
6433
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
3332 self.db.checkpoint_data()
6431
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3333 # Blech, different db's return different exceptions
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3334 # so I can't list them here as some might not be defined
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3335 # on a given system. So capture all exceptions from the
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3336 # code above and try to correct it. If it's correctable its
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3337 # some form of Uniqueness Failure/Integrity Error otherwise
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3338 # undo the fixup and pass on the error.
6433
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
3339 except Exception as e: # nosec
6431
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3340 logger.info('Attempting to handle import exception '
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3341 'for id %s: %s' % (newid, e))
6431
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3342
7665
6f841a55eabf fix: database error handling during import of a non-user item
John Rouillard <rouilj@ieee.org>
parents: 7211
diff changeset
3343 classdb = self.db.getclass(self.classname)
6f841a55eabf fix: database error handling during import of a non-user item
John Rouillard <rouilj@ieee.org>
parents: 7211
diff changeset
3344 keyname = classdb.getkey()
6431
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3345 if has_node or not keyname: # Not an integrity error
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3346 raise
6433
c1d3fbcdbfbd issue2551142 - Import of retired node ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6431
diff changeset
3347 self.db.restore_connection_on_error()
7665
6f841a55eabf fix: database error handling during import of a non-user item
John Rouillard <rouilj@ieee.org>
parents: 7211
diff changeset
3348 activeid = classdb.lookup(d[keyname])
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3349 self.db.sql(retired_sql, (-1, activeid)) # clear the active node
6431
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3350 # this can only happen on an addnode, so retry
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3351 try:
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3352 # if this raises an error, let it propagate upward
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3353 self.db.addnode(self.classname, newid, d) # insert
6431
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3354 except Exception:
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3355 # undo the database change
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3356 self.db.sql(retired_sql, (0, activeid)) # clear active node
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3357 raise # propagate
6431
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3358 logger.info('Successfully handled import exception '
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3359 'for id %s which conflicted with %s' % (
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3360 newid, activeid))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3361
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3362 # retire?
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3363 if retire:
6431
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3364 self.db.sql(retired_sql, (newid, newid))
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3365
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3366 if activeid:
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3367 # unretire the active node
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3368 self.db.sql(retired_sql, ('0', activeid))
ada1edcc9132 issue2551142 - Import ... unique constraint failure.
John Rouillard <rouilj@ieee.org>
parents: 6415
diff changeset
3369
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3370 return newid
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3371
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3372 def export_journals(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3373 """Export a class's journal - generate a list of lists of
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3374 CSV-able data:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3375
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3376 nodeid, date, user, action, params
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3377
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3378 No heading here - the columns are fixed.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3379 """
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3380 properties = self.getprops()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3381 r = []
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3382 for nodeid in self.getnodeids():
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3383 for nodeid, date_, user, action, params in self.history(
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3384 nodeid, enforceperm=False, skipquiet=False):
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3385 date_ = date_.get_tuple()
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3386 if action == 'set':
3365
e2d65f6c8d83 handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents: 3328
diff changeset
3387 export_data = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
3388 for propname, value in params.items():
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3389 if propname not in properties:
3365
e2d65f6c8d83 handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents: 3328
diff changeset
3390 # property no longer in the schema
e2d65f6c8d83 handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents: 3328
diff changeset
3391 continue
e2d65f6c8d83 handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents: 3328
diff changeset
3392
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3393 prop = properties[propname]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3394 # make sure the params are eval()'able
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3395 if value is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3396 pass
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3397 elif isinstance(prop, Date):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3398 value = value.get_tuple()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3399 elif isinstance(prop, Interval):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3400 value = value.get_tuple()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3401 elif isinstance(prop, Password):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3402 value = str(value)
3365
e2d65f6c8d83 handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents: 3328
diff changeset
3403 export_data[propname] = value
e2d65f6c8d83 handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents: 3328
diff changeset
3404 params = export_data
3933
9095a4da67f9 Handle export and import of old trackers that have data attached to...
Richard Jones <richard@users.sourceforge.net>
parents: 3925
diff changeset
3405 elif action == 'create' and params:
9095a4da67f9 Handle export and import of old trackers that have data attached to...
Richard Jones <richard@users.sourceforge.net>
parents: 3925
diff changeset
3406 # old tracker with data stored in the create!
9095a4da67f9 Handle export and import of old trackers that have data attached to...
Richard Jones <richard@users.sourceforge.net>
parents: 3925
diff changeset
3407 params = {}
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3408 l = [nodeid, date_, user, action, params]
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
3409 r.append(list(map(repr_export, l)))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3410 return r
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3411
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3412
2597
c86b2179085b fix journal export of files to remove content from CSV files
Richard Jones <richard@users.sourceforge.net>
parents: 2586
diff changeset
3413 class FileClass(hyperdb.FileClass, Class):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3414 """This class defines a large chunk of data. To support this, it has a
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3415 mandatory String property "content" which is typically saved off
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3416 externally to the hyperdb.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3417
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3418 The default MIME type of this data is defined by the
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3419 "default_mime_type" class attribute, which may be overridden by each
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3420 node if the class defines a "type" String property.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3421 """
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3422 def __init__(self, db, classname, **properties):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3423 """The newly-created class automatically includes the "content"
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3424 and "type" properties.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3425 """
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3426 if 'content' not in properties:
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3427 properties['content'] = hyperdb.String(indexme='yes')
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3428 if 'type' not in properties:
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3429 properties['type'] = hyperdb.String()
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3430 Class.__init__(self, db, classname, **properties)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3431
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3432 def create(self, **propvalues):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3433 """ snaffle the file propvalue and store in a file
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3434 """
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3435 # we need to fire the auditors now, or the content property won't
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3436 # be in propvalues for the auditors to play with
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3437 self.fireAuditors('create', None, propvalues)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3438
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3439 # now remove the content property so it's not stored in the db
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3440 content = propvalues['content']
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3441 del propvalues['content']
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3442
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3443 # do the database create
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3444 newid = self.create_inner(**propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3445
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3446 # figure the mime type
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3447 mime_type = propvalues.get('type', self.default_mime_type)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3448
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3449 # and index!
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3450 if self.properties['content'].indexme:
5544
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3451 index_content = content
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3452 if bytes != str and isinstance(content, bytes):
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3453 index_content = content.decode('utf-8', errors='ignore')
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3454 self.db.indexer.add_text((self.classname, newid, 'content'),
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3455 index_content, mime_type)
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3456
3979
954971d612ee Fire reactors after file storage is all done (patch [SF#2001243])
Richard Jones <richard@users.sourceforge.net>
parents: 3977
diff changeset
3457 # store off the content as a file
5492
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
3458 self.db.storefile(self.classname, newid, None, bs2b(content))
3979
954971d612ee Fire reactors after file storage is all done (patch [SF#2001243])
Richard Jones <richard@users.sourceforge.net>
parents: 3977
diff changeset
3459
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3460 # fire reactors
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3461 self.fireReactors('create', newid, None)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3462
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3463 return newid
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3464
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3465 def get(self, nodeid, propname, default=_marker, cache=1):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3466 """ Trap the content propname and get it from the file
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
3467
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
3468 'cache' exists for backwards compatibility, and is not used.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3469 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3470 poss_msg = 'Possibly a access right configuration problem.'
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3471 if propname == 'content':
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3472 try:
5492
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
3473 return b2s(self.db.getfile(self.classname, nodeid, None))
5248
198b6e810c67 Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents: 5236
diff changeset
3474 except IOError as strerror:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3475 # BUG: by catching this we donot see an error in the log.
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3476 return 'ERROR reading file: %s%s\n%s\n%s' % (
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3477 self.classname, nodeid, poss_msg, strerror)
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3478 except UnicodeDecodeError:
5725
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
3479 # if content is not text (e.g. jpeg file) we get
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
3480 # unicode error trying to convert to string in python 3.
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
3481 # trap it and supply an error message. Include md5sum
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
3482 # of content as this string is included in the etag
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
3483 # calculation of the object.
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
3484 return ('%s%s is not text, retrieve using '
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3485 'binary_content property. mdsum: %s') % (
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3486 self.classname, nodeid,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3487 md5(self.db.getfile(
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3488 self.classname,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3489 nodeid,
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3490 None)).hexdigest()) # nosec - bandit md5 use ok
5492
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
3491 elif propname == 'binary_content':
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
3492 return self.db.getfile(self.classname, nodeid, None)
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
3493
2692
f1c9873496f0 fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2691
diff changeset
3494 if default is not _marker:
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
3495 return Class.get(self, nodeid, propname, default)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3496 else:
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
3497 return Class.get(self, nodeid, propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3498
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3499 def set(self, itemid, **propvalues):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3500 """ Snarf the "content" propvalue and update it in a file
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3501 """
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3502 self.fireAuditors('set', itemid, propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3503 oldvalues = copy.deepcopy(self.db.getnode(self.classname, itemid))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3504
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3505 # now remove the content property so it's not stored in the db
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3506 content = None
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3507 if 'content' in propvalues:
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3508 content = propvalues['content']
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3509 del propvalues['content']
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3510
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3511 # do the database create
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3512 propvalues = self.set_inner(itemid, **propvalues)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3513
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3514 # do content?
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3515 if content:
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3516 # store and possibly index
5492
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
3517 self.db.storefile(self.classname, itemid, None, bs2b(content))
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3518 if self.properties['content'].indexme:
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3519 mime_type = self.get(itemid, 'type', self.default_mime_type)
5544
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3520 index_content = content
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3521 if bytes != str and isinstance(content, bytes):
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3522 index_content = content.decode('utf-8', errors='ignore')
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3523 self.db.indexer.add_text((self.classname, itemid, 'content'),
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3524 index_content, mime_type)
2640
b01eca163779 The "type" parameter is supposed to be optional
Richard Jones <richard@users.sourceforge.net>
parents: 2634
diff changeset
3525 propvalues['content'] = content
b01eca163779 The "type" parameter is supposed to be optional
Richard Jones <richard@users.sourceforge.net>
parents: 2634
diff changeset
3526
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3527 # fire reactors
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3528 self.fireReactors('set', itemid, oldvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3529 return propvalues
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3530
2505
bdd112cf61ba rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
3531 def index(self, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3532 """ Add (or refresh) the node to search indexes.
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3533
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3534 Use the content-type property for the content property.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3535 """
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3536 # find all the String properties that have indexme
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
3537 for prop, propclass in self.getprops().items():
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3538 if prop == 'content' and propclass.indexme:
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3539 mime_type = self.get(nodeid, 'type', self.default_mime_type)
5544
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3540 index_content = self.get(nodeid, 'binary_content')
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3541 if bytes != str and isinstance(index_content, bytes):
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3542 index_content = index_content.decode('utf-8',
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3543 errors='ignore')
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3544 self.db.indexer.add_text((self.classname, nodeid, 'content'),
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3545 index_content, mime_type)
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3546 elif isinstance(propclass, hyperdb.String) and propclass.indexme:
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3547 # index them under (classname, nodeid, property)
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3548 try:
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3549 value = str(self.get(nodeid, prop))
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3550 except IndexError:
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3551 # node has been destroyed
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3552 continue
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3553 self.db.indexer.add_text((self.classname, nodeid, prop), value)
2505
bdd112cf61ba rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
3554
6965
98d72d4bb489 flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6918
diff changeset
3555
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3556 # XXX deviation from spec - was called ItemClass
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3557 class IssueClass(Class, roundupdb.IssueClass):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3558 # Overridden methods:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3559 def __init__(self, db, classname, **properties):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3560 """The newly-created class automatically includes the "messages",
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3561 "files", "nosy", and "superseder" properties. If the 'properties'
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3562 dictionary attempts to specify any of these properties or a
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
3563 "creation", "creator", "activity" or "actor" property, a ValueError
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
3564 is raised.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3565 """
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3566 if 'title' not in properties:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3567 properties['title'] = hyperdb.String(indexme='yes')
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3568 if 'messages' not in properties:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3569 properties['messages'] = hyperdb.Multilink("msg")
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3570 if 'files' not in properties:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3571 properties['files'] = hyperdb.Multilink("file")
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3572 if 'nosy' not in properties:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3573 # note: journalling is turned off as it really just wastes
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3574 # space. this behaviour may be overridden in an instance
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3575 properties['nosy'] = hyperdb.Multilink("user", do_journal="no")
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3576 if 'superseder' not in properties:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3577 properties['superseder'] = hyperdb.Multilink(classname)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3578 Class.__init__(self, db, classname, **properties)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3579
2692
f1c9873496f0 fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2691
diff changeset
3580 # vim: set et sts=4 sw=4 :
Roundup Issue Tracker: http://roundup-tracker.org/