Mercurial > p > roundup > code

annotate roundup/backends/rdbms_common.py @ 6118:e6073c2291c6

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Better Date filtering Allow filtering by multiple date ranges or empty date. Date ranges are separated by comma, an empty date is represented by '-'.
author Ralf Schlatterbeck <rsc@runtux.com>
date Mon, 02 Mar 2020 19:02:16 +0100
parents 1ec4aa670b0c
children e7cb0147e6fe
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3554
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
1 #
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/)
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
3 # This module is free software, and you may redistribute it and/or modify
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
4 # under the same terms as Python, so long as this copyright message and
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
5 # disclaimer are retained in their original form.
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
6 #
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
10 # POSSIBILITY OF SUCH DAMAGE.
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
11 #
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS"
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
17 #
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
18 """ Relational database (SQL) backend common code.
1244
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
19
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
20 Basics:
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
21
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
22 - map roundup classes to relational tables
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
23 - automatically detect schema changes and modify the table schemas
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
24 appropriately (we store the "database version" of the schema in the
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
25 database itself as the only row of the "schema" table)
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
26 - multilinks (which represent a many-to-many relationship) are handled through
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
27 intermediate tables
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
28 - journals are stored adjunct to the per-class tables
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
29 - table names and columns have "_" prepended so the names can't clash with
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
30 restricted names (like "order")
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
31 - retirement is determined by the __retired__ column being > 0
1244
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
32
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
33 Database-specific changes may generally be pushed out to the overridable
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
34 sql_* methods, since everything else should be fairly generic. There's
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
35 probably a bit of work to be done if a database is used that actually
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1222
diff changeset
36 honors column typing, since the initial databases don't (sqlite stores
1528
96cd422532ef bye bye gadfly - you served your purpose well [SF#701127]
Richard Jones <richard@users.sourceforge.net>
parents: 1523
diff changeset
37 everything as a string.)
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
38
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
39 The schema of the hyperdb being mapped to the database is stored in the
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
40 database itself as a repr()'ed dictionary of information about each Class
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
41 that maps to a table. If that information differs from the hyperdb schema,
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
42 then we update it. We also store in the schema dict a version which
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
43 allows us to upgrade the database schema when necessary. See upgrade_db().
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
44
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
45 To force a unqiueness constraint on the key properties we put the item
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
46 id into the __retired__ column duing retirement (so it's 0 for "active"
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
47 items) and place a unqiueness constraint on key + __retired__. This is
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
48 particularly important for the users class where multiple users may
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
49 try to have the same username, with potentially many retired users with
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
50 the same name.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
51 """
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1988
diff changeset
52 __docformat__ = 'restructuredtext'
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
53
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
54 # standard python modules
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
55 import os, time, re, weakref, copy, logging, datetime
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
56
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
57 # roundup modules
3544
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
58 from roundup import hyperdb, date, password, roundupdb, security, support
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
59 from roundup.hyperdb import String, Password, Date, Interval, Link, \
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
60 Multilink, DatabaseError, Boolean, Number, Integer
3925
603ec9630b08 i18n for hyperdb and backend errors
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3924
diff changeset
61 from roundup.i18n import _
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
62
4473
fccf7e09af0c - optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4472
diff changeset
63
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
64 # support
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
65 from roundup.backends.blobfiles import FileStorage
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5086
diff changeset
66 from roundup.backends.indexer_common import get_indexer
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
67 from roundup.backends.sessions_rdbms import Sessions, OneTimeKeys
1499
8ee69708da0c added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1496
diff changeset
68 from roundup.date import Range
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
69
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
70 from roundup.backends.back_anydbm import compile_expression
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
71 from roundup.anypy.strings import b2s, bs2b, us2s, repr_export, eval_import
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
72
5725
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
73 from hashlib import md5
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
74
2692
f1c9873496f0 fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2691
diff changeset
75 # dummy value meaning "argument not passed"
f1c9873496f0 fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2691
diff changeset
76 _marker = []
f1c9873496f0 fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2691
diff changeset
77
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
78
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
79 def _num_cvt(num):
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
80 num = str(num)
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
81 try:
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
82 return int(num)
6002
3175bb92ca28 Cleanups for bandit
John Rouillard <rouilj@ieee.org>
parents: 5871
diff changeset
83 except ValueError:
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
84 return float(num)
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
85
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
86
2472
f41539b3c486 fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents: 2456
diff changeset
87 def _bool_cvt(value):
f41539b3c486 fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents: 2456
diff changeset
88 if value in ('TRUE', 'FALSE'):
f41539b3c486 fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents: 2456
diff changeset
89 return {'TRUE': 1, 'FALSE': 0}[value]
f41539b3c486 fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents: 2456
diff changeset
90 # assume it's a number returned from the db API
f41539b3c486 fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents: 2456
diff changeset
91 return int(value)
f41539b3c486 fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents: 2456
diff changeset
92
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
93
4473
fccf7e09af0c - optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4472
diff changeset
94 def date_to_hyperdb_value(d):
fccf7e09af0c - optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4472
diff changeset
95 """ convert date d to a roundup date """
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
96 if isinstance(d, datetime.datetime):
4473
fccf7e09af0c - optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4472
diff changeset
97 return date.Date(d)
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
98 return date.Date(str(d).replace(' ', '.'))
4473
fccf7e09af0c - optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4472
diff changeset
99
fccf7e09af0c - optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4472
diff changeset
100
2634
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
101 def connection_dict(config, dbnamestr=None):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
102 """ Used by Postgresql and MySQL to detemine the keyword args for
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
103 opening the database connection."""
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
104 d = {}
2634
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
105 if dbnamestr:
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
106 d[dbnamestr] = config.RDBMS_NAME
3099
519b92df37dc handle ~/.my.cnf files for MySQL defaults [SF#1096031]
Richard Jones <richard@users.sourceforge.net>
parents: 3092
diff changeset
107 for name in ('host', 'port', 'password', 'user', 'read_default_group',
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
108 'read_default_file'):
2634
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
109 cvar = 'RDBMS_'+name.upper()
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
110 if config[cvar] is not None:
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
111 d[name] = config[cvar]
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
112 return d
f47ca4541770 Both RDBMS backends now use the same config.ini section, [rdbms].
Richard Jones <richard@users.sourceforge.net>
parents: 2597
diff changeset
113
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
114
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
115 class IdListOptimizer:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
116 """ To prevent flooding the SQL parser of the underlaying
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
117 db engine with "x IN (1, 2, 3, ..., <large number>)" collapses
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
118 these cases to "x BETWEEN 1 AND <large number>".
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
119 """
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
120
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
121 def __init__(self):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
122 self.ranges = []
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
123 self.singles = []
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
124
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
125 def append(self, nid):
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
126 """ Invariant: nid are ordered ascending """
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
127 if self.ranges:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
128 last = self.ranges[-1]
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
129 if last[1] == nid-1:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
130 last[1] = nid
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
131 return
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
132 if self.singles:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
133 last = self.singles[-1]
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
134 if last == nid-1:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
135 self.singles.pop()
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
136 self.ranges.append([last, nid])
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
137 return
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
138 self.singles.append(nid)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
139
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
140 def where(self, field, placeholder):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
141 ranges = self.ranges
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
142 singles = self.singles
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
143
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
144 if not singles and not ranges: return "(1=0)", []
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
145
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
146 if ranges:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
147 between = '%s BETWEEN %s AND %s' % (
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
148 field, placeholder, placeholder)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
149 stmnt = [between] * len(ranges)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
150 else:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
151 stmnt = []
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
152 if singles:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
153 stmnt.append('%s in (%s)' % (
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
154 field, ','.join([placeholder]*len(singles))))
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
155
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
156 return '(%s)' % ' OR '.join(stmnt), sum(ranges, []) + singles
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
157
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
158 def __str__(self):
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
159 return "ranges: %r / singles: %r" % (self.ranges, self.singles)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
160
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
161
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
162 class Database(FileStorage, hyperdb.Database, roundupdb.Database):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
163 """ Wrapper around an SQL database that presents a hyperdb interface.
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
164
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
165 - some functionality is specific to the actual SQL database, hence
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
166 the sql_* methods that are NotImplemented
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
167 - we keep a cache of the latest N row fetches (where N is
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
168 configurable).
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
169 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
170 def __init__(self, config, journaltag=None):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
171 """ Open the database and load the schema from it.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
172 """
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3872
diff changeset
173 FileStorage.__init__(self, config.UMASK)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
174 self.config, self.journaltag = config, journaltag
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
175 self.dir = config.DATABASE
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
176 self.classes = {}
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5086
diff changeset
177 self.indexer = get_indexer(config, self)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
178 self.security = security.Security(self)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
179
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
180 # additional transaction support for external files and the like
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
181 self.transactions = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
182
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
183 # keep a cache of the N most recently retrieved rows of any kind
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
184 # (classname, nodeid) = row
4074
e039f3cbbb96 Make RDBMS cache-size configurable.
Stefan Seefeld <stefan@seefeld.name>
parents: 4062
diff changeset
185 self.cache_size = config.RDBMS_CACHE_SIZE
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
186 self.clearCache()
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
187 self.stats = {'cache_hits': 0, 'cache_misses': 0, 'get_items': 0,
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
188 'filtering': 0}
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
189
5041
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
190 # make sure the database directory exists
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
191 if not os.path.isdir(self.config.DATABASE):
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
192 os.makedirs(self.config.DATABASE)
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
193
1333
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1304
diff changeset
194 # database lock
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1304
diff changeset
195 self.lockfile = None
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1304
diff changeset
196
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
197 # Uppercase to not collide with Class names
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
198 self.Session = None
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
199 self.Otk = None
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
200
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
201 # open a connection to the database, creating the "conn" attribute
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2081
diff changeset
202 self.open_connection()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
203
1181
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
204 def clearCache(self):
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
205 self.cache = {}
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
206 self.cache_lru = []
4652
dfbc0cfa9811 Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4517
diff changeset
207 # upcall is necessary!
dfbc0cfa9811 Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4517
diff changeset
208 roundupdb.Database.clearCache(self)
1181
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
209
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2081
diff changeset
210 def getSessionManager(self):
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
211 if not self.Session:
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
212 self.Session = Sessions(self)
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
213 return self.Session
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2081
diff changeset
214
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2081
diff changeset
215 def getOTKManager(self):
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
216 if not self.Otk:
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
217 self.Otk = OneTimeKeys(self)
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
218 return self.Otk
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2081
diff changeset
219
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2081
diff changeset
220 def open_connection(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
221 """ Open a connection to the database, creating it if necessary.
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
222
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
223 Must call self.load_dbschema()
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
224 """
5236
0a20e1965f4b Pylint checks bark about:
John Rouillard <rouilj@ieee.org>
parents: 5232
diff changeset
225 raise NotImplementedError
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
226
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
227 def sql(self, sql, args=None, cursor=None):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
228 """ Execute the sql with the optional args.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
229 """
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
230 self.log_debug('SQL %r %r' % (sql, args))
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
231 if not cursor:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
232 cursor = self.cursor
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
233 if args:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
234 cursor.execute(sql, args)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
235 else:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
236 cursor.execute(sql)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
237
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
238 def sql_fetchone(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
239 """ Fetch a single row. If there's nothing to fetch, return None.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
240 """
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
241 return self.cursor.fetchone()
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
242
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
243 def sql_fetchall(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
244 """ Fetch all rows. If there's nothing to fetch, return [].
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
245 """
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
246 return self.cursor.fetchall()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
247
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
248 def sql_fetchiter(self):
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
249 """ Fetch all row as a generator
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
250 """
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
251 while True:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
252 row = self.cursor.fetchone()
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
253 if not row: break
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
254 yield row
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
255
4849
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
256 def search_stringquote(self, value):
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
257 """ Quote a search string to escape magic search characters
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
258 '%' and '_', also need to quote '\' (first)
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
259 Then put '%' around resulting string for LIKE (or ILIKE) operator
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
260 """
4849
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
261 v = value.replace('\\', '\\\\')
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
262 v = v.replace('%', '\\%')
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
263 v = v.replace('_', '\\_')
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
264 return '%' + v + '%'
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
265
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
266 def init_dbschema(self):
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
267 self.database_schema = {
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
268 'version': self.current_db_version,
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
269 'tables': {}
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
270 }
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
271
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
272 def load_dbschema(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
273 """ Load the schema definition that the database currently implements
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
274 """
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
275 self.cursor.execute('select schema from schema')
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
276 schema = self.cursor.fetchone()
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
277 if schema:
6002
3175bb92ca28 Cleanups for bandit
John Rouillard <rouilj@ieee.org>
parents: 5871
diff changeset
278 # bandit - schema is trusted
3175bb92ca28 Cleanups for bandit
John Rouillard <rouilj@ieee.org>
parents: 5871
diff changeset
279 self.database_schema = eval(schema[0]) # nosec
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
280 else:
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
281 self.database_schema = {}
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
282
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
283 def save_dbschema(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
284 """ Save the schema definition that the database currently implements
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
285 """
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
286 s = repr(self.database_schema)
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
287 self.sql('delete from schema')
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
288 self.sql('insert into schema values (%s)' % self.arg, (s,))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
289
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
290 def post_init(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
291 """ Called once the schema initialisation has finished.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
292
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
293 We should now confirm that the schema defined by our "classes"
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
294 attribute actually matches the schema in the database.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
295 """
4774
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
296
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
297 # upgrade the database for column type changes, new internal
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
298 # tables, etc.
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
299 save = self.upgrade_db()
3553
Richard Jones <richard@users.sourceforge.net>
parents: 3544
diff changeset
300
Richard Jones <richard@users.sourceforge.net>
parents: 3544
diff changeset
301 # handle changes in the schema
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
302 tables = self.database_schema['tables']
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
303 for classname, spec in self.classes.items():
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
304 if classname in tables:
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
305 dbspec = tables[classname]
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
306 if self.update_class(spec, dbspec):
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
307 tables[classname] = spec.schema()
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
308 save = 1
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
309 else:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
310 self.create_class(spec)
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
311 tables[classname] = spec.schema()
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
312 save = 1
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
313
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
314 for classname, _spec in list(tables.items()):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
315 if classname not in self.classes:
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
316 self.drop_class(classname, tables[classname])
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
317 del tables[classname]
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
318 save = 1
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
319
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
320 # update the database version of the schema
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
321 if save:
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
322 self.save_dbschema()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
323
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
324 # reindex the db if necessary
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
325 if self.indexer.should_reindex():
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
326 self.reindex()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
327
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
328 # commit
2093
3f6024ab2c7a That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
329 self.sql_commit()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
330
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
331 # update this number when we need to make changes to the SQL structure
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
332 # of the backen database
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
333 current_db_version = 5
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
334 db_version_updated = False
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
335
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
336 def upgrade_db(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
337 """ Update the SQL database to reflect changes in the backend code.
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
338
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
339 Return boolean whether we need to save the schema.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
340 """
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
341 version = self.database_schema.get('version', 1)
3993
1eba65a4b0aa Don't run old code on newer database
Richard Jones <richard@users.sourceforge.net>
parents: 3979
diff changeset
342 if version > self.current_db_version:
1eba65a4b0aa Don't run old code on newer database
Richard Jones <richard@users.sourceforge.net>
parents: 3979
diff changeset
343 raise DatabaseError('attempting to run rev %d DATABASE with rev '
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
344 '%d CODE!' % (version,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
345 self.current_db_version))
2081
fb4bf55b94d7 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
346 if version == self.current_db_version:
fb4bf55b94d7 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
347 # nothing to do
fb4bf55b94d7 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
348 return 0
fb4bf55b94d7 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
349
2429
e28a5d5d95a7 better upgrading
Richard Jones <richard@users.sourceforge.net>
parents: 2424
diff changeset
350 if version < 2:
4062
6ea417bafd9c Fix typo.
Stefan Seefeld <stefan@seefeld.name>
parents: 4061
diff changeset
351 self.log_info('upgrade to version 2')
2100
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
352 # change the schema structure
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
353 self.database_schema = {'tables': self.database_schema}
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
354
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
355 # version 1 didn't have the actor column (note that in
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
356 # MySQL this will also transition the tables to typed columns)
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
357 self.add_new_columns_v2()
2100
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
358
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
359 # version 1 doesn't have the OTK, session and indexing in the
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
360 # database
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
361 self.create_version_2_tables()
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
362
2429
e28a5d5d95a7 better upgrading
Richard Jones <richard@users.sourceforge.net>
parents: 2424
diff changeset
363 if version < 3:
4062
6ea417bafd9c Fix typo.
Stefan Seefeld <stefan@seefeld.name>
parents: 4061
diff changeset
364 self.log_info('upgrade to version 3')
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
365 self.fix_version_2_tables()
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
366
2721
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2720
diff changeset
367 if version < 4:
4774
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
368 self.log_info('upgrade to version 4')
2721
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2720
diff changeset
369 self.fix_version_3_tables()
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2720
diff changeset
370
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
371 if version < 5:
4774
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
372 self.log_info('upgrade to version 5')
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
373 self.fix_version_4_tables()
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
374
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
375 self.database_schema['version'] = self.current_db_version
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
376 self.db_version_updated = True
2075
b1704ba7be41 make mysql / postgresql work again. beginnings of otk/session store in rdbmses
Richard Jones <richard@users.sourceforge.net>
parents: 2073
diff changeset
377 return 1
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
378
2721
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2720
diff changeset
379 def fix_version_3_tables(self):
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2720
diff changeset
380 # drop the shorter VARCHAR OTK column and add a new TEXT one
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2720
diff changeset
381 for name in ('otk', 'session'):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
382 self.sql('DELETE FROM %ss' % name)
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
383 self.sql('ALTER TABLE %ss DROP %s_value' % (name, name))
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
384 self.sql('ALTER TABLE %ss ADD %s_value TEXT' % (name, name))
2721
1cd01cf106e1 extend OTK and session table value cols to TEXT [SF#1031271]
Richard Jones <richard@users.sourceforge.net>
parents: 2720
diff changeset
385
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
386 def fix_version_2_tables(self):
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
387 # Default (used by sqlite): NOOP
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
388 pass
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
389
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
390 def fix_version_4_tables(self):
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
391 # note this is an explicit call now
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
392 c = self.cursor
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
393 for cn, klass in self.classes.items():
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
394 c.execute('select id from _%s where __retired__<>0' % (cn,))
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
395 for (id,) in c.fetchall():
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
396 c.execute('update _%s set __retired__=%s where id=%s' % (cn,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
397 self.arg, self.arg), (id, id))
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
398
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
399 if klass.key:
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
400 self.add_class_key_required_unique_constraint(cn, klass.key)
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
401
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
402 def _convert_journal_tables(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
403 """Get current journal table contents, drop the table and re-create"""
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
404 c = self.cursor
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
405 cols = ','.join('nodeid date tag action params'.split())
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
406 for klass in self.classes.values():
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
407 # slurp and drop
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
408 sql = 'select %s from %s__journal order by date' % \
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
409 (cols, klass.classname)
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
410 c.execute(sql)
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
411 contents = c.fetchall()
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
412 self.drop_journal_table_indexes(klass.classname)
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
413 c.execute('drop table %s__journal' % klass.classname)
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
414
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
415 # re-create and re-populate
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
416 self.create_journal_table(klass)
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
417 a = self.arg
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
418 sql = 'insert into %s__journal (%s) values (%s,%s,%s,%s,%s)' % (
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
419 klass.classname, cols, a, a, a, a, a)
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
420 for row in contents:
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
421 # no data conversion needed
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
422 self.cursor.execute(sql, row)
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
423
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
424 def _convert_string_properties(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
425 """Get current Class tables that contain String properties, and
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
426 convert the VARCHAR columns to TEXT"""
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
427 c = self.cursor
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
428 for klass in self.classes.values():
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
429 # slurp and drop
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
430 cols, mls = self.determine_columns(list(klass.properties.items()))
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
431 scols = ','.join([i[0] for i in cols])
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
432 sql = 'select id,%s from _%s' % (scols, klass.classname)
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
433 c.execute(sql)
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
434 contents = c.fetchall()
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
435 self.drop_class_table_indexes(klass.classname, klass.getkey())
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
436 c.execute('drop table _%s' % klass.classname)
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
437
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
438 # re-create and re-populate
2424
74474ec41050 argh! backwards compat
Richard Jones <richard@users.sourceforge.net>
parents: 2416
diff changeset
439 self.create_class_table(klass, create_sequence=0)
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
440 a = ','.join([self.arg for i in range(len(cols)+1)])
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
441 sql = 'insert into _%s (id,%s) values (%s)' % (klass.classname,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
442 scols, a)
2413
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
443 for row in contents:
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
444 l = []
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
445 for entry in row:
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
446 # mysql will already be a string - psql needs "help"
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
447 if entry is not None and not isinstance(entry, type('')):
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
448 entry = str(entry)
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
449 l.append(entry)
7d0bb6601809 fix some column datatypes in postgresql and mysql
Richard Jones <richard@users.sourceforge.net>
parents: 2379
diff changeset
450 self.cursor.execute(sql, l)
2073
261c2e6ceb1e *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
451
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1838
diff changeset
452 def refresh_database(self):
1951
767ff2a03eee more unit tests to improve coverage (up from 85% to 88% for anydbm! :)
Richard Jones <richard@users.sourceforge.net>
parents: 1926
diff changeset
453 self.post_init()
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1838
diff changeset
454
3544
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
455 def reindex(self, classname=None, show_progress=False):
2650
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
456 if classname:
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
457 classes = [self.getclass(classname)]
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
458 else:
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
459 classes = list(self.classes.values())
2650
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
460 for klass in classes:
3544
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
461 if show_progress:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
462 for nodeid in support.Progress('Reindex %s' % klass.classname,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
463 klass.list()):
3544
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
464 klass.index(nodeid)
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
465 else:
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
466 for nodeid in klass.list():
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
467 klass.index(nodeid)
3295
a615cc230160 added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents: 3239
diff changeset
468 self.indexer.save_index()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
469
5175
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
470 # Used here in the generic backend to determine if the database
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
471 # supports 'DOUBLE PRECISION' for floating point numbers.
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
472 implements_double_precision = True
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
473
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
474 hyperdb_to_sql_datatypes = {
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
475 hyperdb.String : 'TEXT',
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
476 hyperdb.Date : 'TIMESTAMP',
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
477 hyperdb.Link : 'INTEGER',
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
478 hyperdb.Interval : 'VARCHAR(255)',
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
479 hyperdb.Password : 'VARCHAR(255)',
2166
cd42c3c7173a MySQL and Postgresql use BOOL/BOOLEAN for Boolean types
Richard Jones <richard@users.sourceforge.net>
parents: 2102
diff changeset
480 hyperdb.Boolean : 'BOOLEAN',
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
481 hyperdb.Number : 'REAL',
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
482 hyperdb.Integer : 'INTEGER',
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
483 }
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
484
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
485 def hyperdb_to_sql_datatype(self, propclass, prop=None):
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
486
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
487 datatype = self.hyperdb_to_sql_datatypes.get(propclass)
5175
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
488 if self.implements_double_precision and prop and \
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
489 isinstance(prop, Number) and prop.use_double:
5175
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
490 datatype = 'DOUBLE PRECISION'
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
491 if datatype:
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
492 return datatype
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
493
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
494 for k, v in self.hyperdb_to_sql_datatypes.items():
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
495 if issubclass(propclass, k):
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
496 return v
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
497
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
498 raise ValueError('%r is not a hyperdb property class' % propclass)
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
499
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
500 def determine_columns(self, properties):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
501 """ Figure the column names and multilink properties from the spec
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
502
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
503 "properties" is a list of (name, prop) where prop may be an
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
504 instance of a hyperdb "type" _or_ a string repr of that type.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
505 """
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
506 cols = [
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
507 ('_actor', self.hyperdb_to_sql_datatype(hyperdb.Link)),
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
508 ('_activity', self.hyperdb_to_sql_datatype(hyperdb.Date)),
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
509 ('_creator', self.hyperdb_to_sql_datatype(hyperdb.Link)),
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
510 ('_creation', self.hyperdb_to_sql_datatype(hyperdb.Date)),
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
511 ]
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
512 mls = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
513 # add the multilinks separately
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
514 for col, prop in properties:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
515 if isinstance(prop, Multilink):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
516 mls.append(col)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
517 continue
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
518
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
519 if isinstance(prop, type('')):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
520 raise ValueError("string property spec!")
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
521 # and prop.find('Multilink') != -1:
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
522 # mls.append(col)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
523
5175
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
524 datatype = self.hyperdb_to_sql_datatype(prop.__class__, prop)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
525 cols.append(('_'+col, datatype))
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
526
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
527 # Intervals stored as two columns
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
528 if isinstance(prop, Interval):
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
529 cols.append(('__'+col+'_int__', 'BIGINT'))
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
530
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
531 cols.sort()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
532 return cols, mls
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
533
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1838
diff changeset
534 def update_class(self, spec, old_spec, force=0):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
535 """ Determine the differences between the current spec and the
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1838
diff changeset
536 database version of the spec, and update where necessary.
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
537
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1838
diff changeset
538 If 'force' is true, update the database anyway.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
539 """
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
540 new_spec = spec.schema()
1515
a516bbb9896b fixed rdbms table update detection logic [SF#703297]
Richard Jones <richard@users.sourceforge.net>
parents: 1500
diff changeset
541 new_spec[1].sort()
a516bbb9896b fixed rdbms table update detection logic [SF#703297]
Richard Jones <richard@users.sourceforge.net>
parents: 1500
diff changeset
542 old_spec[1].sort()
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1838
diff changeset
543 if not force and new_spec == old_spec:
1479
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
544 # no changes
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
545 return 0
1479
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
546
4474
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
547 if not self.config.RDBMS_ALLOW_ALTER:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
548 raise DatabaseError(_('ALTER operation disallowed: %r -> %r.'% (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
549 old_spec, new_spec)))
4474
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
550
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
551 logger = logging.getLogger('roundup.hyperdb.backend')
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
552 logger.info('update_class %s' % spec.classname)
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
553
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
554 logger.debug('old_spec %r' % (old_spec,))
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
555 logger.debug('new_spec %r' % (new_spec,))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
556
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
557 # detect key prop change for potential index change
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
558 keyprop_changes = {}
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
559 if new_spec[0] != old_spec[0]:
3554
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
560 if old_spec[0]:
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
561 keyprop_changes['remove'] = old_spec[0]
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
562 if new_spec[0]:
5e70726a86dd fixed schema migration problem when Class keys were removed
Richard Jones <richard@users.sourceforge.net>
parents: 3553
diff changeset
563 keyprop_changes['add'] = new_spec[0]
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
564
1479
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
565 # detect multilinks that have been removed, and drop their table
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
566 old_has = {}
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
567 for name, prop in old_spec[1]:
1479
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
568 old_has[name] = 1
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
569 if name in spec.properties:
1479
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
570 continue
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
571
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
572 if prop.find('Multilink to') != -1:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
573 # first drop indexes.
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
574 self.drop_multilink_table_indexes(spec.classname, name)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
575
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
576 # now the multilink table itself
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
577 sql = 'drop table %s_%s' % (spec.classname, name)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
578 else:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
579 # if this is the key prop, drop the index first
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
580 if old_spec[0] == prop:
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
581 self.drop_class_table_key_index(spec.classname, name)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
582 del keyprop_changes['remove']
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
583
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
584 # drop the column
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
585 sql = 'alter table _%s drop column _%s' % (spec.classname,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
586 name)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
587
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
588 self.sql(sql)
1479
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
589
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
590 # if we didn't remove the key prop just then, but the key prop has
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
591 # changed, we still need to remove the old index
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
592 if 'remove' in keyprop_changes:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
593 self.drop_class_table_key_index(spec.classname,
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
594 keyprop_changes['remove'])
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
595
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
596 # add new columns
2196
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
597 for propname, prop in new_spec[1]:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
598 if propname in old_has:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
599 continue
2196
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
600 prop = spec.properties[propname]
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
601 if isinstance(prop, Multilink):
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
602 self.create_multilink_table(spec, propname)
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
603 else:
2727
93e2e5b55a3c new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
604 # add the column
5175
e1e40674a0bc Implement double-precision Number
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5112
diff changeset
605 coltype = self.hyperdb_to_sql_datatype(prop.__class__, prop)
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
606 sql = 'alter table _%s add column _%s %s' % (
2727
93e2e5b55a3c new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
607 spec.classname, propname, coltype)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
608 self.sql(sql)
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
609
2727
93e2e5b55a3c new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
610 # extra Interval column
93e2e5b55a3c new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
611 if isinstance(prop, Interval):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
612 sql = 'alter table _%s add column __%s_int__ BIGINT' % (
2727
93e2e5b55a3c new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
613 spec.classname, propname)
93e2e5b55a3c new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
614 self.sql(sql)
93e2e5b55a3c new Interval props weren't created properly in rdbms
Richard Jones <richard@users.sourceforge.net>
parents: 2721
diff changeset
615
2196
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
616 # if the new column is a key prop, we need an index!
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
617 if new_spec[0] == propname:
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
618 self.create_class_table_key_index(spec.classname, propname)
85954067e496 mysql and postgresql schema mutation now handle added Multilinks; fixed test too
Richard Jones <richard@users.sourceforge.net>
parents: 2185
diff changeset
619 del keyprop_changes['add']
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
620
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
621 # if we didn't add the key prop just then, but the key prop has
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
622 # changed, we still need to add the new index
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
623 if 'add' in keyprop_changes:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
624 self.create_class_table_key_index(spec.classname,
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
625 keyprop_changes['add'])
1479
405e91b5be46 fixed rdbms mutation of properties
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
626
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
627 return 1
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
628
3016
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
629 def determine_all_columns(self, spec):
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
630 """Figure out the columns from the spec and also add internal columns
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
631
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
632 """
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
633 cols, mls = self.determine_columns(list(spec.properties.items()))
3082
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
634
3016
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
635 # add on our special columns
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
636 cols.append(('id', 'INTEGER PRIMARY KEY'))
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
637 cols.append(('__retired__', 'INTEGER DEFAULT 0'))
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
638 return cols, mls
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
639
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
640 def create_class_table(self, spec):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
641 """Create the class table for the given Class "spec". Creates the
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
642 indexes too."""
3016
224c7c0b9708 First checkin of tsearch2 "backend". Miscellaneous notes:
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 2985
diff changeset
643 cols, mls = self.determine_all_columns(spec)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
644
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
645 # create the base table
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
646 scols = ','.join(['%s %s' % x for x in cols])
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
647 sql = 'create table _%s (%s)' % (spec.classname, scols)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
648 self.sql(sql)
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
649
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
650 self.create_class_table_indexes(spec)
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
651
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
652 return cols, mls
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
653
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
654 def create_class_table_indexes(self, spec):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
655 """ create the class table for the given spec
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
656 """
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
657 # create __retired__ index
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
658 index_sql2 = 'create index _%s_retired_idx on _%s(__retired__)' % (
1836
94e430ad4fdb make the RDBMS common backend and the SQLite and MYsql backend create...
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1800
diff changeset
659 spec.classname, spec.classname)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
660 self.sql(index_sql2)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
661
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
662 # create index for key property
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
663 if spec.key:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
664 index_sql3 = 'create index _%s_%s_idx on _%s(_%s)' % (
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
665 spec.classname, spec.key,
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
666 spec.classname, spec.key)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
667 self.sql(index_sql3)
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
668
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
669 # and the unique index for key / retired(id)
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
670 self.add_class_key_required_unique_constraint(spec.classname,
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
671 spec.key)
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
672
2228
1d1362c54c94 Some doc / comment fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 2217
diff changeset
673 # TODO: create indexes on (selected?) Link property columns, as
1d1362c54c94 Some doc / comment fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 2217
diff changeset
674 # they're more likely to be used for lookup
1d1362c54c94 Some doc / comment fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 2217
diff changeset
675
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
676 def add_class_key_required_unique_constraint(self, cn, key):
3969
905faf52a51f fix mysql breakage in 1.4.2
Richard Jones <richard@users.sourceforge.net>
parents: 3967
diff changeset
677 sql = '''create unique index _%s_key_retired_idx
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
678 on _%s(__retired__, _%s)''' % (cn, cn, key)
4774
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
679 try:
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
680 self.sql(sql)
6002
3175bb92ca28 Cleanups for bandit
John Rouillard <rouilj@ieee.org>
parents: 5871
diff changeset
681 except Exception: # nosec
4774
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
682 # XXX catch e.g.:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
683 # _sqlite.DatabaseError: index _status_key_retired_idx
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
684 # already exists
4774
3adff0fb0207 Fixed issue2550595: Allow migrating from roundup 0.x to 1.4
Thomas Arendsen Hein <thomas@intevation.de>
parents: 4652
diff changeset
685 pass
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
686
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
687 def drop_class_table_indexes(self, cn, key):
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
688 # drop the old table indexes first
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
689 l = ['_%s_id_idx' % cn, '_%s_retired_idx' % cn]
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
690 if key:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
691 l.append('_%s_%s_idx' % (cn, key))
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
692
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
693 table_name = '_%s' % cn
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
694 for index_name in l:
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
695 if not self.sql_index_exists(table_name, index_name):
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
696 continue
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
697 index_sql = 'drop index '+index_name
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
698 self.sql(index_sql)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
699
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
700 def create_class_table_key_index(self, cn, key):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
701 """ create the class table for the given spec
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
702 """
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
703 sql = 'create index _%s_%s_idx on _%s(_%s)' % (cn, key, cn, key)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
704 self.sql(sql)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
705
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
706 def drop_class_table_key_index(self, cn, key):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
707 table_name = '_%s' % cn
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
708 index_name = '_%s_%s_idx' % (cn, key)
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
709 if self.sql_index_exists(table_name, index_name):
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
710 sql = 'drop index '+index_name
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
711 self.sql(sql)
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
712
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
713 # and now the retired unique index too
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
714 index_name = '_%s_key_retired_idx' % cn
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
715 if self.sql_index_exists(table_name, index_name):
3967
b1e81ad3fa6a don't recompute the index name
Richard Jones <richard@users.sourceforge.net>
parents: 3963
diff changeset
716 sql = 'drop index '+index_name
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
717 self.sql(sql)
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
718
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
719 def create_journal_table(self, spec):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
720 """ create the journal table for a class given the spec and
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
721 already-determined cols
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
722 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
723 # journal table
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
724 cols = ','.join(['%s varchar' % x
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
725 for x in 'nodeid date tag action params'.split()])
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
726 sql = """create table %s__journal (
3224
1fd11a9803bb use backend datatype for journal timestamps.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3155
diff changeset
727 nodeid integer, date %s, tag varchar(255),
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
728 action varchar(255), params text)""" % (spec.classname,
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
729 self.hyperdb_to_sql_datatype(hyperdb.Date))
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
730 self.sql(sql)
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
731 self.create_journal_table_indexes(spec)
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
732
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
733 def create_journal_table_indexes(self, spec):
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
734 # index on nodeid
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
735 sql = 'create index %s_journ_idx on %s__journal(nodeid)' % (
1836
94e430ad4fdb make the RDBMS common backend and the SQLite and MYsql backend create...
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1800
diff changeset
736 spec.classname, spec.classname)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
737 self.sql(sql)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
738
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
739 def drop_journal_table_indexes(self, classname):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
740 index_name = '%s_journ_idx' % classname
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
741 if not self.sql_index_exists('%s__journal' % classname, index_name):
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
742 return
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
743 index_sql = 'drop index '+index_name
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
744 self.sql(index_sql)
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
745
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
746 def create_multilink_table(self, spec, ml):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
747 """ Create a multilink table for the "ml" property of the class
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
748 given by the spec
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
749 """
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
750 # create the table
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
751 sql = 'create table %s_%s (linkid INTEGER, nodeid INTEGER)' % (
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
752 spec.classname, ml)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
753 self.sql(sql)
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
754 self.create_multilink_table_indexes(spec, ml)
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
755
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
756 def create_multilink_table_indexes(self, spec, ml):
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
757 # create index on linkid
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
758 index_sql = 'create index %s_%s_l_idx on %s_%s(linkid)' % (
2100
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
759 spec.classname, ml, spec.classname, ml)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
760 self.sql(index_sql)
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
761
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
762 # create index on nodeid
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
763 index_sql = 'create index %s_%s_n_idx on %s_%s(nodeid)' % (
2100
62ed6505cbec MySQL migration of old backend database to new, typed database complete.
Richard Jones <richard@users.sourceforge.net>
parents: 2098
diff changeset
764 spec.classname, ml, spec.classname, ml)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
765 self.sql(index_sql)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
766
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
767 def drop_multilink_table_indexes(self, classname, ml):
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
768 l = [
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
769 '%s_%s_l_idx' % (classname, ml),
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
770 '%s_%s_n_idx' % (classname, ml)
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
771 ]
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
772 table_name = '%s_%s' % (classname, ml)
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
773 for index_name in l:
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
774 if not self.sql_index_exists(table_name, index_name):
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
775 continue
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
776 index_sql = 'drop index %s' % index_name
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
777 self.sql(index_sql)
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
778
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
779 def create_class(self, spec):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
780 """ Create a database table according to the given spec.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
781 """
4474
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
782
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
783 if not self.config.RDBMS_ALLOW_CREATE:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
784 raise DatabaseError(_('CREATE operation disallowed: "%s".' %
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
785 spec.classname))
4474
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
786
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
787 cols, mls = self.create_class_table(spec)
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
788 self.create_journal_table(spec)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
789
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
790 # now create the multilink tables
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
791 for ml in mls:
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
792 self.create_multilink_table(spec, ml)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
793
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
794 def drop_class(self, cn, spec):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
795 """ Drop the given table from the database.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
796
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
797 Drop the journal and multilink tables too.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
798 """
4474
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
799
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
800 if not self.config.RDBMS_ALLOW_DROP:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
801 raise DatabaseError(_('DROP operation disallowed: "%s".' % cn))
4474
9b4cf6c96ee2 Add flags to allow to restrict DB modifications.
Stefan Seefeld <stefan@seefeld.name>
parents: 4473
diff changeset
802
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
803 properties = spec[1]
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
804 # figure the multilinks
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
805 mls = []
3924
21d3d7eeea8c assorted pyflakes fixes
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3906
diff changeset
806 for propname, prop in properties:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
807 if isinstance(prop, Multilink):
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
808 mls.append(propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
809
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
810 # drop class table and indexes
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
811 self.drop_class_table_indexes(cn, spec[0])
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
812
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
813 self.drop_class_table(cn)
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
814
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
815 # drop journal table and indexes
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
816 self.drop_journal_table_indexes(cn)
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
817 sql = 'drop table %s__journal' % cn
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
818 self.sql(sql)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
819
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
820 for ml in mls:
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
821 # drop multilink table and indexes
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
822 self.drop_multilink_table_indexes(cn, ml)
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
823 sql = 'drop table %s_%s' % (spec.classname, ml)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
824 self.sql(sql)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
825
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
826 def drop_class_table(self, cn):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
827 sql = 'drop table _%s' % cn
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
828 self.sql(sql)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
829
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
830 #
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
831 # Classes
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
832 #
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
833 def __getattr__(self, classname):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
834 """ A convenient way of calling self.getclass(classname).
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
835 """
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
836 if classname in self.classes:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
837 return self.classes[classname]
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
838 raise AttributeError(classname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
839
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
840 def addclass(self, cl):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
841 """ Add a Class to the hyperdatabase.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
842 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
843 cn = cl.classname
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
844 if cn in self.classes:
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
845 raise ValueError(cn)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
846 self.classes[cn] = cl
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
847
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
848 # add default Edit and View permissions
2649
1df7d4a41da4 Buncha stuff (sorry about the large checkin):
Richard Jones <richard@users.sourceforge.net>
parents: 2640
diff changeset
849 self.security.addPermission(name="Create", klass=cn,
1df7d4a41da4 Buncha stuff (sorry about the large checkin):
Richard Jones <richard@users.sourceforge.net>
parents: 2640
diff changeset
850 description="User is allowed to create "+cn)
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
851 self.security.addPermission(name="Edit", klass=cn,
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
852 description="User is allowed to edit "+cn)
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
853 self.security.addPermission(name="View", klass=cn,
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
854 description="User is allowed to access "+cn)
4517
f8e85cf5f0fe how odd, the Retire permission wasn't being registered;
Richard Jones <richard@users.sourceforge.net>
parents: 4490
diff changeset
855 self.security.addPermission(name="Retire", klass=cn,
f8e85cf5f0fe how odd, the Retire permission wasn't being registered;
Richard Jones <richard@users.sourceforge.net>
parents: 4490
diff changeset
856 description="User is allowed to retire "+cn)
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2075
diff changeset
857
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
858 def getclasses(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
859 """ Return a list of the names of all existing classes.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
860 """
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
861 return sorted(self.classes)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
862
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
863 def getclass(self, classname):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
864 """Get the Class object representing a particular class.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
865
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
866 If 'classname' is not a valid class name, a KeyError is raised.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
867 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
868 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
869 return self.classes[classname]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
870 except KeyError:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
871 raise KeyError('There is no class called "%s"' % classname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
872
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
873 def clear(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
874 """Delete all database contents.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
875
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1988
diff changeset
876 Note: I don't commit here, which is different behaviour to the
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1988
diff changeset
877 "nuke from orbit" behaviour in the dbs.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
878 """
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
879 logging.getLogger('roundup.hyperdb.backend').info('clear')
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
880 for cn in self.classes:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
881 sql = 'delete from _%s' % cn
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
882 self.sql(sql)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
883
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
884 #
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
885 # Nodes
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
886 #
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
887
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
888 hyperdb_to_sql_value = {
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
889 hyperdb.String : str,
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
890 # fractional seconds by default
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
891 hyperdb.Date : lambda x: x.formal(sep=' ', sec='%06.3f'),
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
892 hyperdb.Link : int,
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
893 hyperdb.Interval : str,
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
894 hyperdb.Password : str,
2166
cd42c3c7173a MySQL and Postgresql use BOOL/BOOLEAN for Boolean types
Richard Jones <richard@users.sourceforge.net>
parents: 2102
diff changeset
895 hyperdb.Boolean : lambda x: x and 'TRUE' or 'FALSE',
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
896 hyperdb.Number : lambda x: x,
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
897 hyperdb.Integer : lambda x: x,
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
898 hyperdb.Multilink : lambda x: x, # used in journal marshalling
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
899 }
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
900
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
901 def to_sql_value(self, propklass):
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
902
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
903 fn = self.hyperdb_to_sql_value.get(propklass)
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
904 if fn:
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
905 return fn
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
906
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
907 for k, v in self.hyperdb_to_sql_value.items():
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
908 if issubclass(propklass, k):
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
909 return v
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
910
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
911 raise ValueError('%r is not a hyperdb property class' % propklass)
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
912
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
913 def _cache_del(self, key):
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
914 del self.cache[key]
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
915 self.cache_lru.remove(key)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
916
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
917 def _cache_refresh(self, key):
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
918 self.cache_lru.remove(key)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
919 self.cache_lru.insert(0, key)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
920
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
921 def _cache_save(self, key, node):
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
922 self.cache[key] = node
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
923 # update the LRU
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
924 self.cache_lru.insert(0, key)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
925 if len(self.cache_lru) > self.cache_size:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
926 del self.cache[self.cache_lru.pop()]
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
927
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
928 def addnode(self, classname, nodeid, node):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
929 """ Add the specified node to its class's db.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
930 """
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
931 self.log_debug('addnode %s%s %r' % (classname,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
932 nodeid, node))
1528
96cd422532ef bye bye gadfly - you served your purpose well [SF#701127]
Richard Jones <richard@users.sourceforge.net>
parents: 1523
diff changeset
933
96cd422532ef bye bye gadfly - you served your purpose well [SF#701127]
Richard Jones <richard@users.sourceforge.net>
parents: 1523
diff changeset
934 # determine the column definitions and multilink tables
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
935 cl = self.classes[classname]
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
936 cols, mls = self.determine_columns(list(cl.properties.items()))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
937
1189
23b8d1e87fe3 import fixes
Richard Jones <richard@users.sourceforge.net>
parents: 1187
diff changeset
938 # we'll be supplied these props if we're doing an import
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
939 values = node.copy()
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
940 if 'creator' not in values:
1189
23b8d1e87fe3 import fixes
Richard Jones <richard@users.sourceforge.net>
parents: 1187
diff changeset
941 # add in the "calculated" properties (dupe so we don't affect
23b8d1e87fe3 import fixes
Richard Jones <richard@users.sourceforge.net>
parents: 1187
diff changeset
942 # calling code's node assumptions)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
943 values['creation'] = values['activity'] = date.Date()
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
944 values['actor'] = values['creator'] = self.getuid()
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
945
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
946 cl = self.classes[classname]
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
947 props = cl.getprops(protected=1)
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
948 del props['id']
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
949
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
950 # default the non-multilink columns
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
951 for col, prop in props.items():
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
952 if col not in values:
1496
e6ac4e074acb relaxed CVS importing (feature [SF#693277])
Richard Jones <richard@users.sourceforge.net>
parents: 1492
diff changeset
953 if isinstance(prop, Multilink):
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
954 values[col] = []
1496
e6ac4e074acb relaxed CVS importing (feature [SF#693277])
Richard Jones <richard@users.sourceforge.net>
parents: 1492
diff changeset
955 else:
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
956 values[col] = None
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
957
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
958 # clear this node out of the cache if it's in there
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
959 key = (classname, nodeid)
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
960 if key in self.cache:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
961 self._cache_del(key)
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
962
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
963 # figure the values to insert
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
964 vals = []
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
965 for col, _dt in cols:
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
966 # this is somewhat dodgy....
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
967 if col.endswith('_int__'):
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
968 # XXX eugh, this test suxxors
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
969 value = values[col[2:-6]]
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
970 # this is an Interval special "int" column
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
971 if value is not None:
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
972 vals.append(value.as_seconds())
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
973 else:
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
974 vals.append(value)
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
975 continue
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
976
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
977 prop = props[col[1:]]
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
978 value = values[col[1:]]
2472
f41539b3c486 fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents: 2456
diff changeset
979 if value is not None:
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
980 value = self.to_sql_value(prop.__class__)(value)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
981 vals.append(value)
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
982 vals.append(nodeid)
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
983 vals = tuple(vals)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
984
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
985 # make sure the ordering is correct for column name -> column value
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
986 s = ','.join([self.arg for x in cols]) + ',%s' % self.arg
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
987 cols = ','.join([col for col, dt in cols]) + ',id'
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
988
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
989 # perform the inserts
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
990 sql = 'insert into _%s (%s) values (%s)' % (classname, cols, s)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
991 self.sql(sql, vals)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
992
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
993 # insert the multilink rows
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
994 for col in mls:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
995 t = '%s_%s' % (classname, col)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
996 for entry in node[col]:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
997 sql = 'insert into %s (linkid, nodeid) values (%s,%s)' % (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
998 t, self.arg, self.arg)
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
999 self.sql(sql, (entry, nodeid))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1000
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1001 def setnode(self, classname, nodeid, values, multilink_changes={}):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1002 """ Change the specified node.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1003 """
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1004 self.log_debug('setnode %s%s %r' % (classname, nodeid, values))
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1005
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1006 # clear this node out of the cache if it's in there
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1007 key = (classname, nodeid)
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1008 if key in self.cache:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1009 self._cache_del(key)
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1010
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1011 cl = self.classes[classname]
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1012 props = cl.getprops()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1013
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1014 cols = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1015 mls = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1016 # add the multilinks separately
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1017 for col in values:
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1018 prop = props[col]
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1019 if isinstance(prop, Multilink):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1020 mls.append(col)
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1021 elif isinstance(prop, Interval):
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1022 # Intervals store the seconds value too
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1023 cols.append(col)
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1024 # extra leading '_' added by code below
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1025 cols.append('_' + col + '_int__')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1026 else:
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1027 cols.append(col)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1028 cols.sort()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1029
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1030 # figure the values to insert
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1031 vals = []
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1032 for col in cols:
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1033 if col.endswith('_int__'):
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1034 # XXX eugh, this test suxxors
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1035 # Intervals store the seconds value too
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1036 col = col[1:-6]
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1037 prop = props[col]
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1038 value = values[col]
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1039 if value is None:
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1040 vals.append(None)
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1041 else:
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1042 vals.append(value.as_seconds())
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1043 else:
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1044 prop = props[col]
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1045 value = values[col]
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1046 if value is None:
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1047 e = None
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1048 else:
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1049 e = self.to_sql_value(prop.__class__)(value)
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1050 vals.append(e)
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1051
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1052 vals.append(int(nodeid))
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1053 vals = tuple(vals)
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1054
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1055 # if there's any updates to regular columns, do them
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1056 if cols:
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1057 # make sure the ordering is correct for column name -> column value
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1058 s = ','.join(['_%s=%s' % (x, self.arg) for x in cols])
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1059 cols = ','.join(cols)
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1060
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1061 # perform the update
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1062 sql = 'update _%s set %s where id=%s' % (classname, s, self.arg)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
1063 self.sql(sql, vals)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1064
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1065 # we're probably coming from an import, not a change
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1066 if not multilink_changes:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1067 for name in mls:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1068 prop = props[name]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1069 value = values[name]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1070
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1071 t = '%s_%s' % (classname, name)
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1072
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1073 # clear out previous values for this node
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1074 # XXX numeric ids
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1075 self.sql('delete from %s where nodeid=%s' % (t, self.arg),
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1076 (nodeid,))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1077
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1078 # insert the values for this node
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1079 for entry in values[name]:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1080 sql = 'insert into %s (linkid, nodeid) values (%s,%s)' % (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1081 t, self.arg, self.arg)
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1082 # XXX numeric ids
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1083 self.sql(sql, (entry, nodeid))
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1084
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1085 # we have multilink changes to apply
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1086 for col, (add, remove) in multilink_changes.items():
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1087 tn = '%s_%s' % (classname, col)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1088 if add:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1089 sql = 'insert into %s (nodeid, linkid) values (%s,%s)' % (tn,
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1090 self.arg, self.arg)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1091 for addid in add:
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1092 # XXX numeric ids
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1093 self.sql(sql, (int(nodeid), int(addid)))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1094 if remove:
4039
f44ec45cedb4 Fuse multiple DELETE calls into one for multilinks.
Stefan Seefeld <stefan@seefeld.name>
parents: 4038
diff changeset
1095 s = ','.join([self.arg]*len(remove))
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1096 sql = 'delete from %s where nodeid=%s and linkid in (%s)' % (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1097 tn, self.arg, s)
4039
f44ec45cedb4 Fuse multiple DELETE calls into one for multilinks.
Stefan Seefeld <stefan@seefeld.name>
parents: 4038
diff changeset
1098 # XXX numeric ids
f44ec45cedb4 Fuse multiple DELETE calls into one for multilinks.
Stefan Seefeld <stefan@seefeld.name>
parents: 4038
diff changeset
1099 self.sql(sql, [int(nodeid)] + remove)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1100
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1101 sql_to_hyperdb_value = {
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1102 hyperdb.String : str,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1103 hyperdb.Date : date_to_hyperdb_value,
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1104 # hyperdb.Link : int, # XXX numeric ids
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1105 hyperdb.Link : str,
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1106 hyperdb.Interval : date.Interval,
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1107 hyperdb.Password : lambda x: password.Password(encrypted=x),
2472
f41539b3c486 fixed Boolean values in postgresql (bugs [SF#972546] and [SF#972600])
Richard Jones <richard@users.sourceforge.net>
parents: 2456
diff changeset
1108 hyperdb.Boolean : _bool_cvt,
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1109 hyperdb.Number : _num_cvt,
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1110 hyperdb.Integer : int,
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1111 hyperdb.Multilink : lambda x: x, # used in journal marshalling
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1112 }
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1113
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1114 def to_hyperdb_value(self, propklass):
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1115
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1116 fn = self.sql_to_hyperdb_value.get(propklass)
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1117 if fn:
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1118 return fn
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1119
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1120 for k, v in self.sql_to_hyperdb_value.items():
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1121 if issubclass(propklass, k):
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1122 return v
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1123
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1124 raise ValueError('%r is not a hyperdb property class' % propklass)
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1125
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1126 def _materialize_multilink(self, classname, nodeid, node, propname):
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1127 """ evaluation of single Multilink (lazy eval may have skipped this)
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1128 """
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1129 if propname not in node:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1130 sql = 'select linkid from %s_%s where nodeid=%s' % (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1131 classname, propname, self.arg)
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1132 self.sql(sql, (nodeid,))
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1133 # extract the first column from the result
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1134 # XXX numeric ids
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1135 items = [int(x[0]) for x in self.cursor.fetchall()]
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1136 items.sort()
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1137 node[propname] = [str(x) for x in items]
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1138
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1139 def _materialize_multilinks(self, classname, nodeid, node, props=None):
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1140 """ get all Multilinks of a node (lazy eval may have skipped this)
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1141 """
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1142 cl = self.classes[classname]
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1143 props = props or [pn for (pn, p) in cl.properties.items()
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1144 if isinstance(p, Multilink)]
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1145 for propname in props:
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1146 if propname not in node:
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1147 self._materialize_multilink(classname, nodeid, node, propname)
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1148
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1149 def getnode(self, classname, nodeid, fetch_multilinks=True):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1150 """ Get a node from the database.
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1151 For optimisation optionally we don't fetch multilinks
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1152 (lazy Multilinks).
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1153 But for internal database operations we need them.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1154 """
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1155 # see if we have this node cached
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1156 key = (classname, nodeid)
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1157 if key in self.cache:
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1158 # push us back to the top of the LRU
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1159 self._cache_refresh(key)
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1160 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1161 self.stats['cache_hits'] += 1
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1162 # return the cached information
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1163 if fetch_multilinks:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1164 self._materialize_multilinks(classname, nodeid,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1165 self.cache[key])
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1166 return self.cache[key]
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1167
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1168 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1169 self.stats['cache_misses'] += 1
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1170 start_t = time.time()
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1171
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1172 # figure the columns we're fetching
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1173 cl = self.classes[classname]
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1174 cols, mls = self.determine_columns(list(cl.properties.items()))
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1175 scols = ','.join([col for col, dt in cols])
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1176
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1177 # perform the basic property fetch
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1178 sql = 'select %s from _%s where id=%s' % (scols, classname, self.arg)
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1179 self.sql(sql, (nodeid,))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1180
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1181 values = self.sql_fetchone()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1182 if values is None:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1183 raise IndexError('no such %s node %s' % (classname, nodeid))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1184
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1185 # make up the node
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1186 node = {}
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1187 props = cl.getprops(protected=1)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1188 for col in range(len(cols)):
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1189 name = cols[col][0][1:]
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1190 if name.endswith('_int__'):
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1191 # XXX eugh, this test suxxors
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1192 # ignore the special Interval-as-seconds column
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
1193 continue
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1194 value = values[col]
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1195 if value is not None:
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1196 value = self.to_hyperdb_value(props[name].__class__)(value)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1197 node[name] = value
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1198
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1199 if fetch_multilinks and mls:
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1200 self._materialize_multilinks(classname, nodeid, node, mls)
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1201
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1202 # save off in the cache
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1203 key = (classname, nodeid)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1204 self._cache_save(key, node)
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1205
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1206 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1207 self.stats['get_items'] += (time.time() - start_t)
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
1208
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1209 return node
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1210
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1211 def destroynode(self, classname, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1212 """Remove a node from the database. Called exclusively by the
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1213 destroy() method on Class.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1214 """
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1215 message = 'destroynode %s%s' % (classname, nodeid)
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1216 logging.getLogger('roundup.hyperdb.backend').info(message)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1217
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1218 # make sure the node exists
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1219 if not self.hasnode(classname, nodeid):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1220 raise IndexError('%s has no node %s' % (classname, nodeid))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1221
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1222 # see if we have this node cached
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1223 if (classname, nodeid) in self.cache:
1173
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1224 del self.cache[(classname, nodeid)]
58f1a2c174ed simple LRU cache for SQL databases
Richard Jones <richard@users.sourceforge.net>
parents: 1172
diff changeset
1225
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1226 # see if there's any obvious commit actions that we should get rid of
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1227 for entry in self.transactions[:]:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1228 if entry[1][:2] == (classname, nodeid):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1229 self.transactions.remove(entry)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1230
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1231 # now do the SQL
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1232 sql = 'delete from _%s where id=%s' % (classname, self.arg)
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1233 self.sql(sql, (nodeid,))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1234
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1235 # remove from multilnks
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1236 cl = self.getclass(classname)
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1237 x, mls = self.determine_columns(list(cl.properties.items()))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1238 for col in mls:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1239 # get the link ids
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1240 sql = 'delete from %s_%s where nodeid=%s' % (classname, col,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1241 self.arg)
1906
f255363e6d97 PostgreSQL backend lands.
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
1242 self.sql(sql, (nodeid,))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1243
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1244 # remove journal entries
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1245 sql = 'delete from %s__journal where nodeid=%s' % (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1246 classname, self.arg)
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1247 self.sql(sql, (nodeid,))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1248
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
1249 # cleanup any blob filestorage when we commit
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1250 self.transactions.append((FileStorage.destroy, (self,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1251 classname, nodeid)))
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
1252
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1253 def hasnode(self, classname, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1254 """ Determine if the database has a given node.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1255 """
5086
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1256 # nodeid (aka link) is sql type INTEGER. max positive value
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1257 # for INTEGER is 2^31-1 for Postgres and MySQL. The max
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1258 # positive value for SqLite is 2^63 -1, so arguably this check
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1259 # needs to adapt for the type of the RDBMS. For right now,
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1260 # choose lowest common denominator.
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1261 if int(nodeid) >= 2**31:
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1262 # value out of range return false
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1263 return 0
2b551b6b0830 issue2550549 Postgres error on message templating
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
1264
4031
6e2dec4f9836 Fix issue2550501
Stefan Seefeld <stefan@seefeld.name>
parents: 3993
diff changeset
1265 # If this node is in the cache, then we do not need to go to
6e2dec4f9836 Fix issue2550501
Stefan Seefeld <stefan@seefeld.name>
parents: 3993
diff changeset
1266 # the database. (We don't consider this an LRU hit, though.)
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1267 if (classname, nodeid) in self.cache:
4031
6e2dec4f9836 Fix issue2550501
Stefan Seefeld <stefan@seefeld.name>
parents: 3993
diff changeset
1268 # Return 1, not True, to match the type of the result of
6e2dec4f9836 Fix issue2550501
Stefan Seefeld <stefan@seefeld.name>
parents: 3993
diff changeset
1269 # the SQL operation below.
6e2dec4f9836 Fix issue2550501
Stefan Seefeld <stefan@seefeld.name>
parents: 3993
diff changeset
1270 return 1
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1271 sql = 'select count(*) from _%s where id=%s' % (classname, self.arg)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
1272 self.sql(sql, (nodeid,))
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1273 return int(self.cursor.fetchone()[0])
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1274
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1275 def countnodes(self, classname):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1276 """ Count the number of nodes that exist for a particular Class.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1277 """
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1278 sql = 'select count(*) from _%s' % classname
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
1279 self.sql(sql)
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1280 return self.cursor.fetchone()[0]
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1281
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1282 def addjournal(self, classname, nodeid, action, params, creator=None,
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1283 creation=None):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1284 """ Journal the Action
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1285 'action' may be:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1286
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1287 'set' -- 'params' is a dictionary of property values
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1288 'create' -- 'params' is an empty dictionary as of
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1289 Wed Nov 06 11:38:43 2002 +0000
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1290 'link' or 'unlink' -- 'params' is (classname, nodeid, propname)
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1291 'retired' or 'restored' -- 'params' is None
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1292
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1293 'creator' -- the user performing the action, which defaults to
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1294 the current user.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1295 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1296 # handle supply of the special journalling parameters (usually
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1297 # supplied on importing an existing database)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1298 if creator:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1299 journaltag = creator
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1300 else:
1800
a3b1b1dcf639 Use getuid(), not figure_curuserid()
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 1794
diff changeset
1301 journaltag = self.getuid()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1302 if creation:
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1303 journaldate = creation
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1304 else:
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1305 journaldate = date.Date()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1306
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1307 # create the journal entry
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1308 cols = 'nodeid,date,tag,action,params'
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1309
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1310 self.log_debug('addjournal %s%s %r %s %s %r' % (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1311 classname, nodeid, journaldate, journaltag, action, params))
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1312
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1313 # make the journalled data marshallable
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1314 if isinstance(params, type({})):
2275
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1315 self._journal_marshal(params, classname)
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1316
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
1317 params = repr_export(params)
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1318
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1319 dc = self.to_sql_value(hyperdb.Date)
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1320 journaldate = dc(journaldate)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1321
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1322 self.save_journal(classname, cols, nodeid, journaldate,
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1323 journaltag, action, params)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1324
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1325 def setjournal(self, classname, nodeid, journal):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1326 """Set the journal to the "journal" list."""
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1327 # clear out any existing entries
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1328 self.sql('delete from %s__journal where nodeid=%s' % (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1329 classname, self.arg), (nodeid,))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1330
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1331 # create the journal entry
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1332 cols = 'nodeid,date,tag,action,params'
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1333
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1334 dc = self.to_sql_value(hyperdb.Date)
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1335 for nodeid, journaldate, journaltag, action, params in journal:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1336 self.log_debug('addjournal %s%s %r %s %s %r' % (
4059
ef0b4396888a Enhance and simplify logging.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1337 classname, nodeid, journaldate, journaltag, action,
ef0b4396888a Enhance and simplify logging.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1338 params))
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1339
2275
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1340 # make the journalled data marshallable
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1341 if isinstance(params, type({})):
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1342 self._journal_marshal(params, classname)
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
1343 params = repr_export(params)
2275
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1344
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1345 self.save_journal(classname, cols, nodeid, dc(journaldate),
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1346 journaltag, action, params)
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
1347
2275
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1348 def _journal_marshal(self, params, classname):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1349 """Convert the journal params values into safely repr'able and
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1350 eval'able values."""
2275
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1351 properties = self.getclass(classname).getprops()
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1352 for param, value in params.items():
2275
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1353 if not value:
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1354 continue
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1355 property = properties[param]
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1356 cvt = self.to_sql_value(property.__class__)
2275
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1357 if isinstance(property, Password):
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1358 params[param] = cvt(value)
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1359 elif isinstance(property, Date):
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1360 params[param] = cvt(value)
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1361 elif isinstance(property, Interval):
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1362 params[param] = cvt(value)
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1363 elif isinstance(property, Boolean):
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1364 params[param] = cvt(value)
3197e37346de merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2256
diff changeset
1365
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1366 def getjournal(self, classname, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1367 """ get the journal for id
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1368 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1369 # make sure the node exists
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1370 if not self.hasnode(classname, nodeid):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1371 raise IndexError('%s has no node %s' % (classname, nodeid))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1372
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1373 cols = ','.join('nodeid date tag action params'.split())
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1374 journal = self.load_journal(classname, cols, nodeid)
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1375
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1376 # now unmarshal the data
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1377 dc = self.to_hyperdb_value(hyperdb.Date)
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1378 res = []
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1379 properties = self.getclass(classname).getprops()
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1380 for nodeid, date_stamp, user, action, params in journal:
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
1381 params = eval_import(params)
2248
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1382 if isinstance(params, type({})):
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1383 for param, value in params.items():
2248
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1384 if not value:
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1385 continue
2731
7c472ed6babf handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents: 2727
diff changeset
1386 property = properties.get(param, None)
7c472ed6babf handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents: 2727
diff changeset
1387 if property is None:
7c472ed6babf handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents: 2727
diff changeset
1388 # deleted property
7c472ed6babf handle deleted properties in RDBMS history
Richard Jones <richard@users.sourceforge.net>
parents: 2727
diff changeset
1389 continue
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1390 cvt = self.to_hyperdb_value(property.__class__)
2248
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1391 if isinstance(property, Password):
4483
22bc0426e348 Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4480
diff changeset
1392 params[param] = password.JournalPassword(value)
2248
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1393 elif isinstance(property, Date):
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1394 params[param] = cvt(value)
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1395 elif isinstance(property, Interval):
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1396 params[param] = cvt(value)
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1397 elif isinstance(property, Boolean):
cd7e6d6288c6 fixed rego from email address [SF#947414]
Richard Jones <richard@users.sourceforge.net>
parents: 2244
diff changeset
1398 params[param] = cvt(value)
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1399 # XXX numeric ids
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1400 res.append((str(nodeid), dc(date_stamp), user, action, params))
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1401 return res
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1402
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1403 def save_journal(self, classname, cols, nodeid, journaldate,
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1404 journaltag, action, params):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1405 """ Save the journal entry to the database
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1406 """
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1407 entry = (nodeid, journaldate, journaltag, action, params)
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1408
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1409 # do the insert
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1410 a = self.arg
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1411 sql = 'insert into %s__journal (%s) values (%s,%s,%s,%s,%s)' % (
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1412 classname, cols, a, a, a, a, a)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
1413 self.sql(sql, entry)
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1414
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
1415 def load_journal(self, classname, cols, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1416 """ Load the journal from the database
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1417 """
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1418 # now get the journal entries
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1419 sql = 'select %s from %s__journal where nodeid=%s order by date' % (
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1420 cols, classname, self.arg)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
1421 self.sql(sql, (nodeid,))
2244
ac4f295499a4 fixed journal marshalling in RDBMS backends [SF#943627]
Richard Jones <richard@users.sourceforge.net>
parents: 2240
diff changeset
1422 return self.cursor.fetchall()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1423
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1424 def pack(self, pack_before):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1425 """ Delete all journal entries except "create" before 'pack_before'.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1426 """
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
1427 date_stamp = self.to_sql_value(Date)(pack_before)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1428
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1429 # do the delete
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1430 for classname in self.classes:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1431 sql = "delete from %s__journal where date<%s and "\
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1432 "action<>'create'" % (classname, self.arg)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
1433 self.sql(sql, (date_stamp,))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1434
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1435 def sql_commit(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1436 """ Actually commit to the database.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1437 """
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1438 logging.getLogger('roundup.hyperdb.backend').info('commit')
3687
ff9f4ca42454 Postgres backend allows transaction collisions to be ignored when...
Richard Jones <richard@users.sourceforge.net>
parents: 3685
diff changeset
1439
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1440 self.conn.commit()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1441
2374
31cb1014300c Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents: 2362
diff changeset
1442 # open a new cursor for subsequent work
31cb1014300c Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents: 2362
diff changeset
1443 self.cursor = self.conn.cursor()
31cb1014300c Switch to using sqlite's own locking mechanisms...
Richard Jones <richard@users.sourceforge.net>
parents: 2362
diff changeset
1444
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1445 def commit(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1446 """ Commit the current transactions.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1447
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1448 Save all data changed since the database was opened or since the
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1449 last commit() or rollback().
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1450 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1451 # commit the database
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1452 self.sql_commit()
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1453
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1454 # session and otk are committed with the db but not the other
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1455 # way round
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1456 if self.Session:
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1457 self.Session.commit()
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1458 if self.Otk:
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1459 self.Otk.commit()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1460
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1461 # now, do all the other transaction stuff
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1462 for method, args in self.transactions:
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1463 method(*args)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1464
3295
a615cc230160 added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents: 3239
diff changeset
1465 # save the indexer
a615cc230160 added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents: 3239
diff changeset
1466 self.indexer.save_index()
a615cc230160 added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents: 3239
diff changeset
1467
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1468 # clear out the transactions
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1469 self.transactions = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1470
4448
2784c239e6c8 clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4430
diff changeset
1471 # clear the cache: Don't carry over cached values from one
2784c239e6c8 clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4430
diff changeset
1472 # transaction to the next (there may be other changes from other
2784c239e6c8 clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4430
diff changeset
1473 # transactions)
2784c239e6c8 clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4430
diff changeset
1474 self.clearCache()
2784c239e6c8 clear the cache on commit for rdbms backends:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4430
diff changeset
1475
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1476 def sql_rollback(self):
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1477 self.conn.rollback()
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1478
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1479 def rollback(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1480 """ Reverse all actions from the current transaction.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1481
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1482 Undo all the changes made since the database was opened or the last
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1483 commit() or rollback() was performed.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1484 """
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1485 logging.getLogger('roundup.hyperdb.backend').info('rollback')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1486
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1487 self.sql_rollback()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1488
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1489 # roll back "other" transaction stuff
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1490 for method, args in self.transactions:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1491 # delete temporary files
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1492 if method == self.doStoreFile:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1493 self.rollbackStoreFile(*args)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1494 self.transactions = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1495
1492
2fc7d4a8c9e7 fixed sqlite rollback/caching bug [SF#689383]
Richard Jones <richard@users.sourceforge.net>
parents: 1484
diff changeset
1496 # clear the cache
2fc7d4a8c9e7 fixed sqlite rollback/caching bug [SF#689383]
Richard Jones <richard@users.sourceforge.net>
parents: 1484
diff changeset
1497 self.clearCache()
2fc7d4a8c9e7 fixed sqlite rollback/caching bug [SF#689383]
Richard Jones <richard@users.sourceforge.net>
parents: 1484
diff changeset
1498
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1499 def sql_close(self):
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
1500 logging.getLogger('roundup.hyperdb.backend').info('close')
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1501 self.conn.close()
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1502
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1503 def close(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1504 """ Close off the connection.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1505 """
2093
3f6024ab2c7a That's the last of the RDBMS migration steps done! Yay!
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
1506 self.indexer.close()
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
1507 self.sql_close()
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1508 if self.Session:
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1509 self.Session.close()
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1510 self.Session = None
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1511 if self.Otk:
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1512 self.Otk.close()
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
1513 self.Otk = None
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1514
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1515
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1516 #
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1517 # The base Class class
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1518 #
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1519 class Class(hyperdb.Class):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1520 """ The handle to a particular class of nodes in a hyperdatabase.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1521
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1522 All methods except __repr__ and getnode must be implemented by a
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1523 concrete backend Class.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1524 """
4849
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
1525 # For many databases the LIKE operator ignores case.
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
1526 # Postgres and Oracle have an ILIKE operator to support this.
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
1527 # We define the default here, can be changed in derivative class
e68920390aad Fix SQL wildcards in string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4803
diff changeset
1528 case_insensitive_like = 'LIKE'
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1529
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1530 # For some databases (mysql) the = operator for strings ignores case.
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1531 # We define the default here, can be changed in derivative class
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1532 case_sensitive_equal = '='
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1533
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1534 def schema(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1535 """ A dumpable version of the schema that we can store in the
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1536 database
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1537 """
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1538 return (self.key, [(x, repr(y)) for x, y in self.properties.items()])
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1539
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1540 def enableJournalling(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1541 """Turn journalling on for this class
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1542 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1543 self.do_journal = 1
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1544
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1545 def disableJournalling(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1546 """Turn journalling off for this class
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1547 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1548 self.do_journal = 0
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1549
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1550 # Editing nodes:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1551 def create(self, **propvalues):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1552 """ Create a new node of this class and return its id.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1553
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1554 The keyword arguments in 'propvalues' map property names to values.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1555
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1556 The values of arguments must be acceptable for the types of their
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1557 corresponding properties or a TypeError is raised.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1558
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1559 If this class has a key property, it must be present and its value
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1560 must not collide with other key strings or a ValueError is raised.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1561
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1562 Any other properties on this class that are missing from the
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1563 'propvalues' dictionary are set to None.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1564
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1565 If an id in a link or multilink property does not refer to a valid
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1566 node, an IndexError is raised.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1567 """
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
1568 self.fireAuditors('create', None, propvalues)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
1569 newid = self.create_inner(**propvalues)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
1570 self.fireReactors('create', newid, None)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
1571 return newid
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1572
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
1573 def create_inner(self, **propvalues):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1574 """ Called by create, in-between the audit and react calls.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1575 """
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1576 if 'id' in propvalues:
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1577 raise KeyError('"id" is reserved')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1578
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1579 if self.db.journaltag is None:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1580 raise DatabaseError(_('Database open read-only'))
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1581
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1582 if ('creator' in propvalues or 'actor' in propvalues or
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1583 'creation' in propvalues or 'activity' in propvalues):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1584 raise KeyError('"creator", "actor", "creation" and '
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1585 '"activity" are reserved')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1586
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1587 # new node's id
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1588 newid = self.db.newid(self.classname)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1589
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1590 # validate propvalues
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
1591 num_re = re.compile(r'^\d+$')
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1592 for key, value in propvalues.items():
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1593 if key == self.key:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1594 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1595 self.lookup(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1596 except KeyError:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1597 pass
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1598 else:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1599 raise ValueError('node with key "%s" exists' % value)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1600
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1601 # try to handle this property
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1602 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1603 prop = self.properties[key]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1604 except KeyError:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1605 raise KeyError('"%s" has no property "%s"' % (self.classname,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1606 key))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1607
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1608 if value is not None and isinstance(prop, Link):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1609 if not isinstance(value, type('')):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1610 raise ValueError('link value must be String')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1611 link_class = self.properties[key].classname
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1612 # if it isn't a number, it's a key
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1613 if not num_re.match(value):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1614 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1615 value = self.db.classes[link_class].lookup(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1616 except (TypeError, KeyError):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1617 raise IndexError('new property "%s": %s not a %s' % (
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1618 key, value, link_class))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1619 elif not self.db.getclass(link_class).hasnode(value):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1620 raise IndexError('%s has no node %s' % (link_class,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1621 value))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1622
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1623 # save off the value
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1624 propvalues[key] = value
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1625
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1626 # register the link with the newly linked node
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1627 if self.do_journal and self.properties[key].do_journal:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1628 self.db.addjournal(link_class, value, 'link',
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1629 (self.classname, newid, key))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1630
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1631 elif isinstance(prop, Multilink):
3872
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
1632 if value is None:
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
1633 value = []
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1634 if not hasattr(value, '__iter__') or \
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1635 isinstance(value, type('')):
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1636 raise TypeError('new property "%s" not an iterable of ids'
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1637 % key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1638 # clean up and validate the list of links
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1639 link_class = self.properties[key].classname
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1640 l = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1641 for entry in value:
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1642 if not isinstance(entry, type('')):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1643 raise ValueError('"%s" multilink value (%r) '
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1644 'must contain Strings' % (
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1645 key, value))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1646 # if it isn't a number, it's a key
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1647 if not num_re.match(entry):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1648 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1649 entry = self.db.classes[link_class].lookup(entry)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1650 except (TypeError, KeyError):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1651 raise IndexError('new property "%s": %s not a %s' % (
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1652 key, entry, self.properties[key].classname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1653 l.append(entry)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1654 value = l
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1655 propvalues[key] = value
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1656
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1657 # handle additions
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1658 for nodeid in value:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1659 if not self.db.getclass(link_class).hasnode(nodeid):
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1660 raise IndexError('%s has no node %s' % (link_class,
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1661 nodeid))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1662 # register the link with the newly linked node
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1663 if self.do_journal and self.properties[key].do_journal:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1664 self.db.addjournal(link_class, nodeid, 'link',
6065
1ec4aa670b0c Flake8 whitespace; remove imports; type()!=type() -> isinstance()
John Rouillard <rouilj@ieee.org>
parents: 6002
diff changeset
1665 (self.classname, newid, key))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1666
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1667 elif isinstance(prop, String):
1383
f19dde90e473 applied unicode patch
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1365
diff changeset
1668 if type(value) != type('') and type(value) != type(u''):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1669 raise TypeError('new property "%s" not a string'%key)
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2887
diff changeset
1670 if prop.indexme:
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2887
diff changeset
1671 self.db.indexer.add_text((self.classname, newid, key),
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2887
diff changeset
1672 value)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1673
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1674 elif isinstance(prop, Password):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1675 if not isinstance(value, password.Password):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1676 raise TypeError('new property "%s" not a Password'%key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1677
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1678 elif isinstance(prop, Date):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1679 if value is not None and not isinstance(value, date.Date):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1680 raise TypeError('new property "%s" not a Date'%key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1681
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1682 elif isinstance(prop, Interval):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1683 if value is not None and not isinstance(value, date.Interval):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1684 raise TypeError('new property "%s" not an Interval'%key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1685
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1686 elif value is not None and isinstance(prop, Number):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1687 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1688 float(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1689 except ValueError:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1690 raise TypeError('new property "%s" not numeric'%key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1691
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1692 elif value is not None and isinstance(prop, Integer):
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1693 try:
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1694 int(value)
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1695 except ValueError:
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1696 raise TypeError('new property "%s" not integer'%key)
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1697
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1698 elif value is not None and isinstance(prop, Boolean):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1699 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1700 int(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1701 except ValueError:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1702 raise TypeError('new property "%s" not boolean'%key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1703
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1704 # make sure there's data where there needs to be
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
1705 for key, prop in self.properties.items():
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1706 if key in propvalues:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1707 continue
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1708 if key == self.key:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1709 raise ValueError('key property "%s" is required'%key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1710 if isinstance(prop, Multilink):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1711 propvalues[key] = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1712 else:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1713 propvalues[key] = None
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1714
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1715 # done
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1716 self.db.addnode(self.classname, newid, propvalues)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1717 if self.do_journal:
2546
1b07c9740bba mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2539
diff changeset
1718 self.db.addjournal(self.classname, newid, ''"create", {})
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1719
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1720 # XXX numeric ids
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
1721 return str(newid)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1722
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1723 def get(self, nodeid, propname, default=_marker, cache=1):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1724 """Get the value of a property on an existing node of this class.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1725
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1726 'nodeid' must be the id of an existing node of this class or an
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1727 IndexError is raised. 'propname' must be the name of a property
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1728 of this class or a KeyError is raised.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1729
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
1730 'cache' exists for backwards compatibility, and is not used.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1731 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1732 if propname == 'id':
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1733 return nodeid
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1734
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1735 # get the node's dict
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1736 d = self.db.getnode(self.classname, nodeid, fetch_multilinks=False)
4476
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1737 # handle common case -- that property is in dict -- first
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1738 # if None and one of creator/creation actor/activity return None
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1739 if propname in d:
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1740 r = d [propname]
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1741 # return copy of our list
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1742 if isinstance (r, list):
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1743 return r[:]
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1744 if r is not None:
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1745 return r
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1746 elif propname in ('creation', 'activity', 'creator', 'actor'):
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1747 return r
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1748
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1749 # propname not in d:
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1750 if propname == 'creation' or propname == 'activity':
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1751 return date.Date()
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1752 if propname == 'creator' or propname == 'actor':
143f52d48e60 - small performance optimisation for 'get': do common case first
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4474
diff changeset
1753 return self.db.getuid()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1754
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1755 # get the property (raises KeyError if invalid)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1756 prop = self.properties[propname]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1757
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1758 # lazy evaluation of Multilink
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1759 if propname not in d and isinstance(prop, Multilink):
4490
559d9a2a0191 Fixed bug in filter_iter refactoring (lazy multilinks)...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4483
diff changeset
1760 self.db._materialize_multilink(self.classname, nodeid, d, propname)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1761
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1762 # handle there being no value in the table for the property
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1763 if propname not in d or d[propname] is None:
2692
f1c9873496f0 fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2691
diff changeset
1764 if default is _marker:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1765 if isinstance(prop, Multilink):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1766 return []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1767 else:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1768 return None
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1769 else:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1770 return default
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1771
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1772 # don't pass our list to other code
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1773 if isinstance(prop, Multilink):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1774 return d[propname][:]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1775
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1776 return d[propname]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1777
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1778 def set(self, nodeid, **propvalues):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1779 """Modify a property on an existing node of this class.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1780
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1781 'nodeid' must be the id of an existing node of this class or an
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1782 IndexError is raised.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1783
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1784 Each key in 'propvalues' must be the name of a property of this
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1785 class or a KeyError is raised.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1786
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1787 All values in 'propvalues' must be acceptable types for their
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1788 corresponding properties or a TypeError is raised.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1789
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1790 If the value of the key property is set, it must not collide with
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1791 other key strings or a ValueError is raised.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1792
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1793 If the value of a Link or Multilink property contains an invalid
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1794 node id, a ValueError is raised.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1795 """
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1796 self.fireAuditors('set', nodeid, propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1797 oldvalues = copy.deepcopy(self.db.getnode(self.classname, nodeid))
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1798 propvalues = self.set_inner(nodeid, **propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1799 self.fireReactors('set', nodeid, oldvalues)
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
1800 return propvalues
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1801
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1802 def set_inner(self, nodeid, **propvalues):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1803 """ Called by set, in-between the audit and react calls.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
1804 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1805 if not propvalues:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1806 return propvalues
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1807
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1808 if ('creator' in propvalues or 'actor' in propvalues or
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1809 'creation' in propvalues or 'activity' in propvalues):
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1810 raise KeyError('"creator", "actor", "creation" and '
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1811 '"activity" are reserved')
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1812
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1813 if 'id' in propvalues:
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1814 raise KeyError('"id" is reserved')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1815
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1816 if self.db.journaltag is None:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1817 raise DatabaseError(_('Database open read-only'))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1818
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1819 node = self.db.getnode(self.classname, nodeid)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1820 if self.is_retired(nodeid):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1821 raise IndexError('Requested item is retired')
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
1822 num_re = re.compile(r'^\d+$')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1823
3082
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
1824 # make a copy of the values dictionary - we'll modify the contents
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
1825 propvalues = propvalues.copy()
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
1826
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1827 # if the journal value is to be different, store it in here
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1828 journalvalues = {}
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1829
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1830 # remember the add/remove stuff for multilinks, making it easier
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1831 # for the Database layer to do its stuff
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1832 multilink_changes = {}
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1833
5112
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
1834 # omit quiet properties from history/changelog
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
1835 quiet_props = []
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
1836
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1837 for propname, value in list(propvalues.items()):
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1838 # check to make sure we're not duplicating an existing key
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1839 if propname == self.key and node[propname] != value:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1840 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1841 self.lookup(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1842 except KeyError:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1843 pass
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1844 else:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1845 raise ValueError('node with key "%s" exists'%value)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1846
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1847 # this will raise the KeyError if the property isn't valid
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1848 # ... we don't use getprops() here because we only care about
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1849 # the writeable properties.
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1850 try:
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1851 prop = self.properties[propname]
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
1852 except KeyError:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1853 raise KeyError('"%s" has no property named "%s"'%(
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1854 self.classname, propname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1855
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1856 # if the value's the same as the existing value, no sense in
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1857 # doing anything
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1295
diff changeset
1858 current = node.get(propname, None)
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1295
diff changeset
1859 if value == current:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1860 del propvalues[propname]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1861 continue
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1295
diff changeset
1862 journalvalues[propname] = current
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1863
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1864 # do stuff based on the prop type
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1865 if isinstance(prop, Link):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1866 link_class = prop.classname
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1867 # if it isn't a number, it's a key
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1868 if value is not None and not isinstance(value, type('')):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1869 raise ValueError('property "%s" link value be a string'%(
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1870 propname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1871 if isinstance(value, type('')) and not num_re.match(value):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1872 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1873 value = self.db.classes[link_class].lookup(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1874 except (TypeError, KeyError):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1875 raise IndexError('new property "%s": %s not a %s'%(
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1876 propname, value, prop.classname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1877
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1878 if (value is not None and
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1879 not self.db.getclass(link_class).hasnode(value)):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1880 raise IndexError('%s has no node %s'%(link_class,
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1881 value))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1882
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1883 if self.do_journal and prop.do_journal:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1884 # register the unlink with the old linked node
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1885 if node[propname] is not None:
2546
1b07c9740bba mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2539
diff changeset
1886 self.db.addjournal(link_class, node[propname],
1b07c9740bba mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2539
diff changeset
1887 ''"unlink", (self.classname, nodeid, propname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1888
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1889 # register the link with the newly linked node
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1890 if value is not None:
2546
1b07c9740bba mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2539
diff changeset
1891 self.db.addjournal(link_class, value, ''"link",
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1892 (self.classname, nodeid, propname))
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1893
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1894 elif isinstance(prop, Multilink):
3872
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
1895 if value is None:
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
1896 value = []
5467
630a00b20394 strings have an __iter__ method on Python 3, but we don't want to consider them iterable
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5416
diff changeset
1897 if not hasattr(value, '__iter__') or type(value) == type(''):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1898 raise TypeError('new property "%s" not an iterable of'
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1899 ' ids'%propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1900 link_class = self.properties[propname].classname
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1901 l = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1902 for entry in value:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1903 # if it isn't a number, it's a key
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1904 if type(entry) != type(''):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1905 raise ValueError('new property "%s" link value '
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1906 'must be a string'%propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1907 if not num_re.match(entry):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1908 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1909 entry = self.db.classes[link_class].lookup(entry)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1910 except (TypeError, KeyError):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1911 raise IndexError('new property "%s": %s not a %s'%(
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1912 propname, entry,
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1913 self.properties[propname].classname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1914 l.append(entry)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1915 value = l
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1916 propvalues[propname] = value
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1917
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1918 # figure the journal entry for this property
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1919 add = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1920 remove = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1921
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1922 # handle removals
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1923 if propname in node:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1924 l = node[propname]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1925 else:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1926 l = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1927 for id in l[:]:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1928 if id in value:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1929 continue
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1930 # register the unlink with the old linked node
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1931 if self.do_journal and self.properties[propname].do_journal:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1932 self.db.addjournal(link_class, id, 'unlink',
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1933 (self.classname, nodeid, propname))
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1934 l.remove(id)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1935 remove.append(id)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1936
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1937 # handle additions
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1938 for id in value:
4095
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
1939 if id in l:
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
1940 continue
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
1941 # We can safely check this condition after
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
1942 # checking that this is an addition to the
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
1943 # multilink since the condition was checked for
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
1944 # existing entries at the point they were added to
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
1945 # the multilink. Since the hasnode call will
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
1946 # result in a SQL query, it is more efficient to
01eb89b07c13 Don't check for node's existence if we know it exists.
Stefan Seefeld <stefan@seefeld.name>
parents: 4085
diff changeset
1947 # avoid the check if possible.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1948 if not self.db.getclass(link_class).hasnode(id):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1949 raise IndexError('%s has no node %s'%(link_class,
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1950 id))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1951 # register the link with the newly linked node
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1952 if self.do_journal and self.properties[propname].do_journal:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1953 self.db.addjournal(link_class, id, 'link',
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1954 (self.classname, nodeid, propname))
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1955 l.append(id)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1956 add.append(id)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1957
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1958 # figure the journal entry
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1959 l = []
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1960 if add:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1961 l.append(('+', add))
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1962 if remove:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1963 l.append(('-', remove))
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1964 multilink_changes[propname] = (add, remove)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1965 if l:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1966 journalvalues[propname] = tuple(l)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1967
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1968 elif isinstance(prop, String):
1383
f19dde90e473 applied unicode patch
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1365
diff changeset
1969 if value is not None and type(value) != type('') and type(value) != type(u''):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1970 raise TypeError('new property "%s" not a string'%propname)
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2887
diff changeset
1971 if prop.indexme:
3399
3a0d4e4a0f34 merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3383
diff changeset
1972 if value is None: value = ''
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2887
diff changeset
1973 self.db.indexer.add_text((self.classname, nodeid, propname),
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2887
diff changeset
1974 value)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1975
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1976 elif isinstance(prop, Password):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1977 if not isinstance(value, password.Password):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1978 raise TypeError('new property "%s" not a Password'%propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1979 propvalues[propname] = value
4483
22bc0426e348 Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4480
diff changeset
1980 journalvalues[propname] = \
22bc0426e348 Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4480
diff changeset
1981 current and password.JournalPassword(current)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1982
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1983 elif value is not None and isinstance(prop, Date):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1984 if not isinstance(value, date.Date):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1985 raise TypeError('new property "%s" not a Date'% propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1986 propvalues[propname] = value
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1987
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1988 elif value is not None and isinstance(prop, Interval):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1989 if not isinstance(value, date.Interval):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1990 raise TypeError('new property "%s" not an '
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1991 'Interval'%propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1992 propvalues[propname] = value
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1993
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1994 elif value is not None and isinstance(prop, Number):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1995 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1996 float(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1997 except ValueError:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
1998 raise TypeError('new property "%s" not numeric'%propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1999
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
2000 elif value is not None and isinstance(prop, Integer):
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
2001 try:
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
2002 int(value)
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
2003 except ValueError:
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
2004 raise TypeError('new property "%s" not integer'%propname)
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
2005
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2006 elif value is not None and isinstance(prop, Boolean):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2007 try:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2008 int(value)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2009 except ValueError:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2010 raise TypeError('new property "%s" not boolean'%propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2011
5112
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2012 # record quiet properties to omit from history/changelog
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2013 if prop.quiet:
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2014 quiet_props.append(propname)
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2015
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2016 # nothing to do?
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2017 if not propvalues:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2018 return propvalues
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2019
3082
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
2020 # update the activity time
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
2021 propvalues['activity'] = date.Date()
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
2022 propvalues['actor'] = self.db.getuid()
29bd9f06160e actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3057
diff changeset
2023
3328
475c8560ef9b merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3310
diff changeset
2024 # do the set
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
2025 self.db.setnode(self.classname, nodeid, propvalues, multilink_changes)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2026
3328
475c8560ef9b merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3310
diff changeset
2027 # remove the activity props now they're handled
475c8560ef9b merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3310
diff changeset
2028 del propvalues['activity']
475c8560ef9b merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3310
diff changeset
2029 del propvalues['actor']
475c8560ef9b merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3310
diff changeset
2030
475c8560ef9b merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3310
diff changeset
2031 # journal the set
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2032 if self.do_journal:
2546
1b07c9740bba mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2539
diff changeset
2033 self.db.addjournal(self.classname, nodeid, ''"set", journalvalues)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2034
5112
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2035 # remove quiet properties from output
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2036 for propname in quiet_props:
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2037 if propname in propvalues:
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2038 del propvalues[propname]
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
2039
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
2040 return propvalues
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2041
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2042 def retire(self, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2043 """Retire a node.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
2044
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2045 The properties on the node remain available from the get() method,
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2046 and the node's id is never reused.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
2047
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2048 Retired nodes are not returned by the find(), list(), or lookup()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2049 methods, and other nodes may reuse the values of their key properties.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2050 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2051 if self.db.journaltag is None:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2052 raise DatabaseError(_('Database open read-only'))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2053
1345
618aa9c37d65 fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents: 1333
diff changeset
2054 self.fireAuditors('retire', nodeid, None)
618aa9c37d65 fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents: 1333
diff changeset
2055
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2056 # use the arg for __retired__ to cope with any odd database type
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2057 # conversion (hello, sqlite)
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2058 sql = 'update _%s set __retired__=%s where id=%s'%(self.classname,
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2059 self.db.arg, self.db.arg)
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2060 self.db.sql(sql, (nodeid, nodeid))
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2061 if self.do_journal:
2546
1b07c9740bba mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2539
diff changeset
2062 self.db.addjournal(self.classname, nodeid, ''"retired", None)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2063
1345
618aa9c37d65 fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents: 1333
diff changeset
2064 self.fireReactors('retire', nodeid, None)
618aa9c37d65 fire auditors and reactors in rdbms retire (thanks Sheila King)
Richard Jones <richard@users.sourceforge.net>
parents: 1333
diff changeset
2065
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2066 def restore(self, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2067 """Restore a retired node.
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2068
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2069 Make node available for all operations like it was before retirement.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2070 """
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2071 if self.db.journaltag is None:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2072 raise DatabaseError(_('Database open read-only'))
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2073
1523
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2074 node = self.db.getnode(self.classname, nodeid)
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2075 # check if key property was overrided
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2076 key = self.getkey()
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2077 try:
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2078 id = self.lookup(node[key])
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2079 except KeyError:
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2080 pass
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2081 else:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2082 raise KeyError("Key property (%s) of retired node clashes "
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2083 "with existing one (%s)" % (key, node[key]))
1523
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
2084
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2085 self.fireAuditors('restore', nodeid, None)
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2086 # use the arg for __retired__ to cope with any odd database type
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2087 # conversion (hello, sqlite)
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2088 sql = 'update _%s set __retired__=%s where id=%s'%(self.classname,
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2089 self.db.arg, self.db.arg)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
2090 self.db.sql(sql, (0, nodeid))
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2091 if self.do_journal:
2546
1b07c9740bba mark names of journal actions for translation.
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2539
diff changeset
2092 self.db.addjournal(self.classname, nodeid, ''"restored", None)
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2093
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1515
diff changeset
2094 self.fireReactors('restore', nodeid, None)
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
2095
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2096 def is_retired(self, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2097 """Return true if the node is rerired
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2098 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2099 sql = 'select __retired__ from _%s where id=%s'%(self.classname,
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2100 self.db.arg)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
2101 self.db.sql(sql, (nodeid,))
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2102 return int(self.db.sql_fetchone()[0]) > 0
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2103
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2104 def destroy(self, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2105 """Destroy a node.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
2106
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2107 WARNING: this method should never be used except in extremely rare
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2108 situations where there could never be links to the node being
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2109 deleted
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1988
diff changeset
2110
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2111 WARNING: use retire() instead
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1988
diff changeset
2112
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2113 WARNING: the properties of this node will not be available ever again
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1988
diff changeset
2114
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2115 WARNING: really, use retire() instead
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2116
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2117 Well, I think that's enough warnings. This method exists mostly to
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2118 support the session storage of the cgi interface.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2119
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2120 The node is completely removed from the hyperdb, including all journal
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2121 entries. It will no longer be available, and will generally break code
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2122 if there are any references to the node.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2123 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2124 if self.db.journaltag is None:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2125 raise DatabaseError(_('Database open read-only'))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2126 self.db.destroynode(self.classname, nodeid)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2127
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2128 # Locating nodes:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2129 def hasnode(self, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2130 """Determine if the given nodeid actually exists
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2131 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2132 return self.db.hasnode(self.classname, nodeid)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2133
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2134 def setkey(self, propname):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2135 """Select a String property of this class to be the key property.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2136
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2137 'propname' must be the name of a String property of this class or
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2138 None, or a TypeError is raised. The values of the key property on
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2139 all existing nodes must be unique or a ValueError is raised.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2140 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2141 prop = self.getprops()[propname]
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2142 if not isinstance(prop, String):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2143 raise TypeError('key properties must be String')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2144 self.key = propname
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2145
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2146 def getkey(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2147 """Return the name of the key property for this class or None."""
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2148 return self.key
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2149
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2150 def lookup(self, keyvalue):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2151 """Locate a particular node by its key property and return its id.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2152
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2153 If this class has no key property, a TypeError is raised. If the
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2154 'keyvalue' matches one of the values for the key property among
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2155 the nodes in this class, the matching node's id is returned;
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2156 otherwise a KeyError is raised.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2157 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2158 if not self.key:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2159 raise TypeError('No key property set for class %s'%self.classname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2160
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2161 # use the arg to handle any odd database type conversion (hello,
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2162 # sqlite)
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2163 sql = "select id from _%s where _%s=%s and __retired__=%s"%(
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2164 self.classname, self.key, self.db.arg, self.db.arg)
3977
732a37da3a10 Fix for postgres 8.3 compatibility (and bug) (patch [SF#2030479])
Richard Jones <richard@users.sourceforge.net>
parents: 3969
diff changeset
2165 self.db.sql(sql, (str(keyvalue), 0))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2166
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1173
diff changeset
2167 # see if there was a result that's not retired
1203
735adcbfc665 fix to SQL lookup() and retirement
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
2168 row = self.db.sql_fetchone()
735adcbfc665 fix to SQL lookup() and retirement
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
2169 if not row:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2170 raise KeyError('No key (%s) value "%s" for "%s"'%(self.key,
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2171 keyvalue, self.classname))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2172
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2173 # return the id
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2174 # XXX numeric ids
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2175 return str(row[0])
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2176
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2177 def find(self, **propspec):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2178 """Get the ids of nodes in this class which link to the given nodes.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2179
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2180 'propspec' consists of keyword args propname=nodeid or
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2181 propname={nodeid:1, }
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2182 'propname' must be the name of a property in this class, or a
1912
2b0ab61db194 fixes for [SF#818339]
Richard Jones <richard@users.sourceforge.net>
parents: 1911
diff changeset
2183 KeyError is raised. That property must be a Link or
2b0ab61db194 fixes for [SF#818339]
Richard Jones <richard@users.sourceforge.net>
parents: 1911
diff changeset
2184 Multilink property, or a TypeError is raised.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2185
3239
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3224
diff changeset
2186 Any node in this class whose 'propname' property links to any of
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3224
diff changeset
2187 the nodeids will be returned. Examples::
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3224
diff changeset
2188
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3224
diff changeset
2189 db.issue.find(messages='1')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2190 db.issue.find(messages={'1':1,'3':1}, files={'7':1})
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2191 """
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2192 # shortcut
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2193 if not propspec:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2194 return []
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2195
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2196 # validate the args
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2197 props = self.getprops()
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
2198 for propname, nodeids in propspec.items():
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2199 # check the prop is OK
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2200 prop = props[propname]
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2201 if not isinstance(prop, Link) and not isinstance(prop, Multilink):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2202 raise TypeError("'%s' not a Link/Multilink property"%propname)
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2203
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2204 # first, links
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2205 a = self.db.arg
2733
ef396596a24e more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents: 2731
diff changeset
2206 allvalues = ()
ef396596a24e more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents: 2731
diff changeset
2207 sql = []
1988
5660b89f8903 more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents: 1986
diff changeset
2208 where = []
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
2209 for prop, values in propspec.items():
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2210 if not isinstance(props[prop], hyperdb.Link):
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2211 continue
1912
2b0ab61db194 fixes for [SF#818339]
Richard Jones <richard@users.sourceforge.net>
parents: 1911
diff changeset
2212 if type(values) is type({}) and len(values) == 1:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2213 values = list(values)[0]
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2214 if type(values) is type(''):
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2215 allvalues += (values,)
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2216 where.append('_%s = %s'%(prop, a))
1563
e2a8ce4d2317 Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents: 1557
diff changeset
2217 elif values is None:
e2a8ce4d2317 Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents: 1557
diff changeset
2218 where.append('_%s is NULL'%prop)
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2219 else:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2220 values = list(values)
2494
ea7fb2f416db fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents: 2472
diff changeset
2221 s = ''
ea7fb2f416db fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents: 2472
diff changeset
2222 if None in values:
ea7fb2f416db fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents: 2472
diff changeset
2223 values.remove(None)
ea7fb2f416db fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents: 2472
diff changeset
2224 s = '_%s is NULL or '%prop
ea7fb2f416db fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents: 2472
diff changeset
2225 allvalues += tuple(values)
ea7fb2f416db fixed RDBMS Class.find() to handle None value in multiple find...
Richard Jones <richard@users.sourceforge.net>
parents: 2472
diff changeset
2226 s += '_%s in (%s)'%(prop, ','.join([a]*len(values)))
2530
a182c536b72d fix filtering by Link property:
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2516
diff changeset
2227 where.append('(' + s +')')
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2228 if where:
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2229 allvalues = (0, ) + allvalues
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2230 sql.append("""select id from _%s where __retired__=%s
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2231 and %s"""%(self.classname, a, ' and '.join(where)))
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2232
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2233 # now multilinks
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
2234 for prop, values in propspec.items():
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2235 if not isinstance(props[prop], hyperdb.Multilink):
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2236 continue
1988
5660b89f8903 more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents: 1986
diff changeset
2237 if not values:
5660b89f8903 more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents: 1986
diff changeset
2238 continue
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2239 allvalues += (0, )
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2240 if type(values) is type(''):
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2241 allvalues += (values,)
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2242 s = a
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2243 else:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2244 allvalues += tuple(values)
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1212
diff changeset
2245 s = ','.join([a]*len(values))
1988
5660b89f8903 more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents: 1986
diff changeset
2246 tn = '%s_%s'%(self.classname, prop)
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2247 sql.append("""select id from _%s, %s where __retired__=%s
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2248 and id = %s.nodeid and %s.linkid in (%s)"""%(self.classname,
2733
ef396596a24e more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents: 2731
diff changeset
2249 tn, a, tn, tn, s))
ef396596a24e more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents: 2731
diff changeset
2250
ef396596a24e more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents: 2731
diff changeset
2251 if not sql:
1988
5660b89f8903 more compliance testing, this time for find()
Richard Jones <richard@users.sourceforge.net>
parents: 1986
diff changeset
2252 return []
2733
ef396596a24e more efficient find() in RDBMS [SF#1012781]
Richard Jones <richard@users.sourceforge.net>
parents: 2731
diff changeset
2253 sql = ' union '.join(sql)
1183
08a13a84ed43 Some speedups - both of the SQL backends can handle using only one cursor.
Richard Jones <richard@users.sourceforge.net>
parents: 1181
diff changeset
2254 self.db.sql(sql, allvalues)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2255 # XXX numeric ids
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2256 l = [str(x[0]) for x in self.db.sql_fetchall()]
1195
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2257 return l
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2258
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2259 def stringFind(self, **requirements):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2260 """Locate a particular node by matching a set of its String
1195
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2261 properties in a caseless search.
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2262
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2263 If the property is not a String property, a TypeError is raised.
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
2264
1195
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2265 The return is a list of the id of all nodes that match.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2266 """
1195
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2267 where = []
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2268 args = []
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2269 for propname in requirements:
1195
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2270 prop = self.properties[propname]
1951
767ff2a03eee more unit tests to improve coverage (up from 85% to 88% for anydbm! :)
Richard Jones <richard@users.sourceforge.net>
parents: 1926
diff changeset
2271 if not isinstance(prop, String):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2272 raise TypeError("'%s' not a String property"%propname)
1195
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2273 where.append(propname)
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2274 args.append(requirements[propname].lower())
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2275
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2276 # generate the where clause
1549
a53a7e197360 fixed rdbms email address lookup (case insensitivity)
Richard Jones <richard@users.sourceforge.net>
parents: 1539
diff changeset
2277 s = ' and '.join(['lower(_%s)=%s'%(col, self.db.arg) for col in where])
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2278 sql = 'select id from _%s where %s and __retired__=%s'%(
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
2279 self.classname, s, self.db.arg)
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2280 args.append(0)
1195
e0032f4ab334 added missing stringFind to sql backends
Richard Jones <richard@users.sourceforge.net>
parents: 1189
diff changeset
2281 self.db.sql(sql, tuple(args))
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2282 # XXX numeric ids
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2283 l = [str(x[0]) for x in self.db.sql_fetchall()]
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2284 return l
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2285
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2286 def list(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2287 """ Return a list of the ids of the active nodes in this class.
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2288 """
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1479
diff changeset
2289 return self.getnodeids(retired=0)
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1479
diff changeset
2290
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1479
diff changeset
2291 def getnodeids(self, retired=None):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2292 """ Retrieve all the ids of the nodes for a particular Class.
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1479
diff changeset
2293
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
2294 Set retired=None to get all nodes. Otherwise it'll get all the
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1479
diff changeset
2295 retired or non-retired nodes, depending on the flag.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2296 """
1707
3d627e34f18e sqlite backend now passes all tests under 2.3.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1599
diff changeset
2297 # flip the sense of the 'retired' flag if we don't want all of them
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1479
diff changeset
2298 if retired is not None:
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2299 args = (0, )
1719
eeb167fb8faf *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 1707
diff changeset
2300 if retired:
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2301 compare = '>'
1719
eeb167fb8faf *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 1707
diff changeset
2302 else:
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2303 compare = '='
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2304 sql = 'select id from _%s where __retired__%s%s'%(self.classname,
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2305 compare, self.db.arg)
1539
800b5896e14a fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents: 1528
diff changeset
2306 else:
800b5896e14a fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents: 1528
diff changeset
2307 args = ()
800b5896e14a fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents: 1528
diff changeset
2308 sql = 'select id from _%s'%self.classname
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
2309 self.db.sql(sql, args)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2310 # XXX numeric ids
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2311 ids = [str(x[0]) for x in self.db.cursor.fetchall()]
1539
800b5896e14a fixed rdbms export - getnodeids in particular with NULL values
Richard Jones <richard@users.sourceforge.net>
parents: 1528
diff changeset
2312 return ids
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2313
3635
53987aa153d2 Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2314 def _subselect(self, classname, multilink_table):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2315 """Create a subselect. This is factored out because some
3633
a292054b4393 The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3612
diff changeset
2316 databases (hmm only one, so far) doesn't support subselects
a292054b4393 The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3612
diff changeset
2317 look for "I can't believe it's not a toy RDBMS" in the mysql
a292054b4393 The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3612
diff changeset
2318 backend.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2319 """
3635
53987aa153d2 Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2320 return '_%s.id not in (select nodeid from %s)'%(classname,
53987aa153d2 Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2321 multilink_table)
3633
a292054b4393 The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3612
diff changeset
2322
3685
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2323 # Some DBs order NULL values last. Set this variable in the backend
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2324 # for prepending an order by clause for each attribute that causes
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2325 # correct sort order for NULLs. Examples:
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2326 # order_by_null_values = '(%s is not NULL)'
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2327 # order_by_null_values = 'notnull(%s)'
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2328 # The format parameter is replaced with the attribute.
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2329 order_by_null_values = None
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2330
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2331 def supports_subselects(self):
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2332 '''Assuming DBs can do subselects, overwrite if they cannot.
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2333 '''
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2334 return True
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2335
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2336 def _filter_multilink_expression_fallback(
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2337 self, classname, multilink_table, expr):
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2338 '''This is a fallback for database that do not support
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2339 subselects.'''
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2340
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2341 is_valid = expr.evaluate
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2342
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2343 last_id, kws = None, []
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2344
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2345 ids = IdListOptimizer()
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2346 append = ids.append
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2347
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2348 # This join and the evaluation in program space
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2349 # can be expensive for larger databases!
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2350 # TODO: Find a faster way to collect the data needed
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2351 # to evalute the expression.
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2352 # Moving the expression evaluation into the database
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2353 # would be nice but this tricky: Think about the cases
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2354 # where the multilink table does not have join values
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2355 # needed in evaluation.
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2356
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2357 stmnt = "SELECT c.id, m.linkid FROM _%s c " \
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2358 "LEFT OUTER JOIN %s m " \
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2359 "ON c.id = m.nodeid ORDER BY c.id" % (
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2360 classname, multilink_table)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2361 self.db.sql(stmnt)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2362
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2363 # collect all multilink items for a class item
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2364 for nid, kw in self.db.sql_fetchiter():
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2365 if nid != last_id:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2366 if last_id is None:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2367 last_id = nid
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2368 else:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2369 # we have all multilink items -> evaluate!
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2370 if is_valid(kws): append(last_id)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2371 last_id, kws = nid, []
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2372 if kw is not None:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2373 kws.append(kw)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2374
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2375 if last_id is not None and is_valid(kws):
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2376 append(last_id)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2377
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2378 # we have ids of the classname table
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2379 return ids.where("_%s.id" % classname, self.db.arg)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2380
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2381 def _filter_multilink_expression(self, classname, multilink_table, v):
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2382 """ Filters out elements of the classname table that do not
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2383 match the given expression.
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2384 Returns tuple of 'WHERE' introns for the overall filter.
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2385 """
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2386 try:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2387 opcodes = [int(x) for x in v]
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2388 if min(opcodes) >= -1: raise ValueError()
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2389
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2390 expr = compile_expression(opcodes)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2391
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2392 if not self.supports_subselects():
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2393 # We heavily rely on subselects. If there is
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2394 # no decent support fall back to slower variant.
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2395 return self._filter_multilink_expression_fallback(
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2396 classname, multilink_table, expr)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2397
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2398 atom = \
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2399 "%s IN(SELECT linkid FROM %s WHERE nodeid=a.id)" % (
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2400 self.db.arg,
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2401 multilink_table)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2402
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2403 intron = \
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2404 "_%(classname)s.id in (SELECT id " \
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2405 "FROM _%(classname)s AS a WHERE %(condition)s) " % {
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2406 'classname' : classname,
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2407 'condition' : expr.generate(lambda n: atom) }
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2408
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2409 values = []
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2410 def collect_values(n): values.append(n.x)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2411 expr.visit(collect_values)
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2412
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2413 return intron, values
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2414 except:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2415 # original behavior
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2416 where = "%s.linkid in (%s)" % (
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2417 multilink_table, ','.join([self.db.arg] * len(v)))
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2418 return where, v, True # True to indicate original
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2419
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
2420 def _filter_sql (self, search_matches, filterspec, srt=[], grp=[], retr=0,
5869
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2421 retired=False, exact_match_spec={}, limit=None,
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2422 offset=None):
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2423 """ Compute the proptree and the SQL/ARGS for a filter.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2424 For argument description see filter below.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2425 We return a 3-tuple, the proptree, the sql and the sql-args
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2426 or None if no SQL is necessary.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2427 The flag retr serves to retrieve *all* non-Multilink properties
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2428 (for filling the cache during a filter_iter)
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2429 """
2578
7f25486ff85e fixed RDBMS filter() for no matches from full-text search [SF#990778]
Richard Jones <richard@users.sourceforge.net>
parents: 2546
diff changeset
2430 # we can't match anything if search_matches is empty
4044
27a9906cd8d1 Fix issue2550505.
Stefan Seefeld <stefan@seefeld.name>
parents: 4039
diff changeset
2431 if not search_matches and search_matches is not None:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2432 return None
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
2433
3634
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2434 icn = self.classname
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2435
2379
a2025bdd1491 fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
2436 # vars to hold the components of the SQL statement
2586
3c67f4bfa225 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 2578
diff changeset
2437 frum = [] # FROM clauses
2379
a2025bdd1491 fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
2438 loj = [] # LEFT OUTER JOIN clauses
a2025bdd1491 fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
2439 where = [] # WHERE clauses
a2025bdd1491 fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
2440 args = [] # *any* positional arguments
a2025bdd1491 fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
2441 a = self.db.arg
a2025bdd1491 fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
2442
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2443 # figure the WHERE clause from the filterspec
2256
0b198ed096af fixes for py2.1 (booleans, sigh)
Richard Jones <richard@users.sourceforge.net>
parents: 2248
diff changeset
2444 mlfilt = 0 # are we joining with Multilink tables?
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2445 sortattr = self._sortattr (group = grp, sort = srt)
5871
acc4a128ab9b Backwards-compatible signature of _proptree method
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5869
diff changeset
2446 proptree = self._proptree(filterspec, exact_match_spec, sortattr, retr)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2447 mlseen = 0
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2448 for pt in reversed(proptree.sortattr):
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2449 p = pt
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2450 while p.parent:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2451 if isinstance (p.propclass, Multilink):
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2452 mlseen = True
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2453 if mlseen:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2454 p.sort_ids_needed = True
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2455 p.tree_sort_done = False
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2456 p = p.parent
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2457 if not mlseen:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2458 pt.attr_sort_done = pt.tree_sort_done = True
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2459 proptree.compute_sort_done()
3693
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
2460
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2461 cols = ['_%s.id'%icn]
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2462 mlsort = []
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2463 rhsnum = 0
3635
53987aa153d2 Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2464 for p in proptree:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2465 rc = ac = oc = None
3634
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2466 cn = p.classname
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2467 ln = p.uniqname
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2468 pln = p.parent.uniqname
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2469 pcn = p.parent.classname
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2470 k = p.name
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2471 v = p.val
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2472 propclass = p.propclass
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2473 if p.parent == proptree and p.name == 'id' \
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2474 and 'retrieve' in p.need_for:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2475 p.sql_idx = 0
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2476 if 'sort' in p.need_for or 'retrieve' in p.need_for:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2477 rc = oc = ac = '_%s._%s'%(pln, k)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2478 if isinstance(propclass, Multilink):
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2479 if 'search' in p.need_for:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2480 mlfilt = 1
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2481 tn = '%s_%s'%(pcn, k)
4061
52431f2ad76b Correct handling of a Multilink filterspec with an empty list of linked-to items.
Stefan Seefeld <stefan@seefeld.name>
parents: 4060
diff changeset
2482 if v in ('-1', ['-1'], []):
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2483 # only match rows that have count(linkid)=0 in the
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2484 # corresponding multilink table)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2485 where.append(self._subselect(pcn, tn))
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2486 else:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2487 frum.append(tn)
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2488 gen_join = True
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2489
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2490 if p.has_values and isinstance(v, type([])):
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2491 result = self._filter_multilink_expression(pln,
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2492 tn, v)
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2493 # XXX: We dont need an id join if we used the filter
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2494 gen_join = len(result) == 3
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2495
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2496 if gen_join:
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2497 where.append('_%s.id=%s.nodeid'%(pln,tn))
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2498
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2499 if p.children:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2500 frum.append('_%s as _%s' % (cn, ln))
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2501 where.append('%s.linkid=_%s.id'%(tn, ln))
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2502
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2503 if p.has_values:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2504 if isinstance(v, type([])):
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2505 where.append(result[0])
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4462
diff changeset
2506 args += result[1]
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2507 else:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2508 where.append('%s.linkid=%s'%(tn, a))
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2509 args.append(v)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2510 if 'sort' in p.need_for:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2511 assert not p.attr_sort_done and not p.sort_ids_needed
1365
4884fb0860f9 fixed rdbms searching by ID [SF#666615]
Richard Jones <richard@users.sourceforge.net>
parents: 1351
diff changeset
2512 elif k == 'id':
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2513 if 'search' in p.need_for:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2514 if isinstance(v, type([])):
4462
0f4a8fcb9a1d fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents: 4448
diff changeset
2515 # If there are no permitted values, then the
0f4a8fcb9a1d fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents: 4448
diff changeset
2516 # where clause will always be false, and we
0f4a8fcb9a1d fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents: 4448
diff changeset
2517 # can optimize the query away.
0f4a8fcb9a1d fix SQL generation for empty WHERE clause.
Stefan Seefeld <stefan@seefeld.name>
parents: 4448
diff changeset
2518 if not v:
4803
5e679e364f9a Bug-fix: need to return None for empty query
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4787
diff changeset
2519 return None
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2520 s = ','.join([a for x in v])
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2521 where.append('_%s.%s in (%s)'%(pln, k, s))
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2522 args = args + v
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2523 else:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2524 where.append('_%s.%s=%s'%(pln, k, a))
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2525 args.append(v)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2526 if 'sort' in p.need_for or 'retrieve' in p.need_for:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2527 rc = oc = ac = '_%s.id'%pln
3683
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3682
diff changeset
2528 elif isinstance(propclass, String):
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2529 if 'search' in p.need_for:
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2530 exact = []
3683
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3682
diff changeset
2531 if not isinstance(v, type([])):
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3682
diff changeset
2532 v = [v]
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2533 new_v = []
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2534 for x in v:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2535 if isinstance(x, hyperdb.Exact_Match):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2536 exact.append(True)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2537 new_v.append(x.value)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2538 else:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2539 exact.append(False)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2540 # Quote special search characters '%' and '_' for
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2541 # correct matching with LIKE/ILIKE
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2542 # Note that we now pass the elements of v as query
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2543 # arguments and don't interpolate the quoted string
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2544 # into the sql statement. Should be safer.
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2545 new_v.append(self.db.search_stringquote(x))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2546 v = new_v
3683
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3682
diff changeset
2547
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3682
diff changeset
2548 # now add to the where clause
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2549 w = []
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2550 for vv, ex in zip(v, exact):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2551 if ex:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2552 w.append("_%s._%s %s %s"%(
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2553 pln, k, self.case_sensitive_equal, a))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2554 args.append(vv)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2555 else:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2556 w.append("_%s._%s %s %s ESCAPE %s"%(
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2557 pln, k, self.case_insensitive_like, a, a))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2558 args.extend((vv, '\\'))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2559 where.append ('(' + ' and '.join(w) + ')')
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2560 if 'sort' in p.need_for:
3683
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3682
diff changeset
2561 oc = ac = 'lower(_%s._%s)'%(pln, k)
1170
af104fa52746 Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents: 1168
diff changeset
2562 elif isinstance(propclass, Link):
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2563 if 'search' in p.need_for:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2564 if p.children:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2565 if 'sort' not in p.need_for:
3694
595041d78104 Fix a peculiarity in sql generation.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3693
diff changeset
2566 frum.append('_%s as _%s' % (cn, ln))
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2567 where.append('_%s._%s=_%s.id'%(pln, k, ln))
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2568 if p.has_values:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2569 if isinstance(v, type([])):
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2570 d = {}
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2571 for entry in v:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2572 if entry == '-1':
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2573 entry = None
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2574 d[entry] = entry
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2575 l = []
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2576 if None in d or not d:
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2577 if None in d: del d[None]
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2578 l.append('_%s._%s is NULL'%(pln, k))
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2579 if d:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2580 v = list(d)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2581 s = ','.join([a for x in v])
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2582 l.append('(_%s._%s in (%s))'%(pln, k, s))
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2583 args = args + v
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2584 if l:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2585 where.append('(' + ' or '.join(l) +')')
3636
fa7becc62534 Of course for Links there is the same bug as I fixed before for multilinks.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3635
diff changeset
2586 else:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2587 if v in ('-1', None):
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2588 v = None
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2589 where.append('_%s._%s is NULL'%(pln, k))
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2590 else:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2591 where.append('_%s._%s=%s'%(pln, k, a))
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2592 args.append(v)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2593 if 'sort' in p.need_for:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2594 lp = p.cls.labelprop()
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2595 oc = ac = '_%s._%s'%(pln, k)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2596 if lp != 'id':
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2597 if p.tree_sort_done:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2598 loj.append(
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2599 'LEFT OUTER JOIN _%s as _%s on _%s._%s=_%s.id'%(
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2600 cn, ln, pln, k, ln))
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2601 oc = '_%s._%s'%(ln, lp)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2602 if 'retrieve' in p.need_for:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2603 rc = '_%s._%s'%(pln, k)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2604 elif isinstance(propclass, Date) and 'search' in p.need_for:
4060
2a68d7494bbc Robustify SQL<->HyperDB data type conversion.
Stefan Seefeld <stefan@seefeld.name>
parents: 4059
diff changeset
2605 dc = self.db.to_sql_value(hyperdb.Date)
1351
d1bfb479e527 fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents: 1345
diff changeset
2606 if isinstance(v, type([])):
d1bfb479e527 fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents: 1345
diff changeset
2607 s = ','.join([a for x in v])
3634
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2608 where.append('_%s._%s in (%s)'%(pln, k, s))
3633
a292054b4393 The whole filter method was replicated in back_mysql.py from rdbms_common.py
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3612
diff changeset
2609 args = args + [dc(date.Date(x)) for x in v]
1351
d1bfb479e527 fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents: 1345
diff changeset
2610 else:
1499
8ee69708da0c added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1496
diff changeset
2611 try:
6118
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2612 wh = []
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2613 ar = []
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2614 for d in v.split(','):
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2615 w1 = []
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2616 if d == '-':
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2617 wh.append('_%s._%s is NULL'%(pln, k))
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2618 continue
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2619 # Try to filter on range of dates
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2620 date_rng = propclass.range_from_raw(d, self.db)
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2621 if date_rng.from_value:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2622 w1.append('_%s._%s >= %s'%(pln, k, a))
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2623 ar.append(dc(date_rng.from_value))
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2624 if date_rng.to_value:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2625 w1.append('_%s._%s <= %s'%(pln, k, a))
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2626 ar.append(dc(date_rng.to_value))
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2627 wh.append (' and '.join (w1))
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2628 where.append ('(' + ' or '.join (wh) + ')')
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6065
diff changeset
2629 args.extend (ar)
1499
8ee69708da0c added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1496
diff changeset
2630 except ValueError:
8ee69708da0c added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1496
diff changeset
2631 # If range creation fails - ignore that search parameter
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
2632 pass
1351
d1bfb479e527 fixed searching on date / interval fields [SF#658157]
Richard Jones <richard@users.sourceforge.net>
parents: 1345
diff changeset
2633 elif isinstance(propclass, Interval):
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2634 # filter/sort using the __<prop>_int__ column
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2635 if 'search' in p.need_for:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2636 if isinstance(v, type([])):
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2637 s = ','.join([a for x in v])
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2638 where.append('_%s.__%s_int__ in (%s)'%(pln, k, s))
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2639 args = args + [date.Interval(x).as_seconds() for x in v]
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2640 else:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2641 try:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2642 # Try to filter on range of intervals
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2643 date_rng = Range(v, date.Interval)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2644 if date_rng.from_value:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2645 where.append('_%s.__%s_int__ >= %s'%(pln, k, a))
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2646 args.append(date_rng.from_value.as_seconds())
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2647 if date_rng.to_value:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2648 where.append('_%s.__%s_int__ <= %s'%(pln, k, a))
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2649 args.append(date_rng.to_value.as_seconds())
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2650 except ValueError:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2651 # If range creation fails - ignore search parameter
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2652 pass
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2653 if 'sort' in p.need_for:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2654 oc = ac = '_%s.__%s_int__'%(pln,k)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2655 if 'retrieve' in p.need_for:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2656 rc = '_%s._%s'%(pln,k)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2657 elif isinstance(propclass, Boolean) and 'search' in p.need_for:
4365
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2658 if type(v) == type(""):
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2659 v = v.split(',')
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2660 if type(v) != type([]):
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2661 v = [v]
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2662 bv = []
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2663 for val in v:
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2664 if type(val) is type(''):
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2665 bv.append(propclass.from_raw (val))
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2666 else:
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2667 bv.append(bool(val))
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2668 if len(bv) == 1:
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2669 where.append('_%s._%s=%s'%(pln, k, a))
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2670 args = args + bv
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2671 else:
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2672 s = ','.join([a for x in v])
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2673 where.append('_%s._%s in (%s)'%(pln, k, s))
667c818f6a22 - unify bool searching (filter method) across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4359
diff changeset
2674 args = args + bv
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2675 elif 'search' in p.need_for:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2676 if isinstance(v, type([])):
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
2677 s = ','.join([a for x in v])
3634
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2678 where.append('_%s._%s in (%s)'%(pln, k, s))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2679 args = args + v
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2680 else:
3634
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2681 where.append('_%s._%s=%s'%(pln, k, a))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2682 args.append(v)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2683 if oc:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2684 if p.sort_ids_needed:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2685 if rc == ac:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2686 p.sql_idx = len(cols)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2687 p.auxcol = len(cols)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2688 cols.append(ac)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2689 if p.tree_sort_done and p.sort_direction:
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2690 # Don't select top-level id or multilink twice
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2691 if (not p.sort_ids_needed or ac != oc) and (p.name != 'id'
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2692 or p.parent != proptree):
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2693 if rc == oc:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2694 p.sql_idx = len(cols)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2695 cols.append(oc)
3685
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2696 desc = ['', ' desc'][p.sort_direction == '-']
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2697 # Some SQL dbs sort NULL values last -- we want them first.
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2698 if (self.order_by_null_values and p.name != 'id'):
4d9adb8bc3b1 Null-value sorting fixes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3683
diff changeset
2699 nv = self.order_by_null_values % oc
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2700 cols.append(nv)
3693
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
2701 p.orderby.append(nv + desc)
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
2702 p.orderby.append(oc + desc)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2703 if 'retrieve' in p.need_for and p.sql_idx is None:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2704 assert(rc)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2705 p.sql_idx = len(cols)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2706 cols.append (rc)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2707
3634
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2708 props = self.getprops()
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2709
1740
5ca448ff8052 don't have RDBMS backends list retired nodes [SF#767319]
Richard Jones <richard@users.sourceforge.net>
parents: 1719
diff changeset
2710 # don't match retired nodes
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
2711 if retired is not None:
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
2712 op = '='
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
2713 if retired:
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
2714 op = '!='
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
2715 where.append('_%s.__retired__%s0'%(icn, op))
1740
5ca448ff8052 don't have RDBMS backends list retired nodes [SF#767319]
Richard Jones <richard@users.sourceforge.net>
parents: 1719
diff changeset
2716
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2717 # add results of full text search
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2718 if search_matches is not None:
4044
27a9906cd8d1 Fix issue2550505.
Stefan Seefeld <stefan@seefeld.name>
parents: 4039
diff changeset
2719 s = ','.join([a for x in search_matches])
3634
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2720 where.append('_%s.id in (%s)'%(icn, s))
4113
f4641359b892 Fix issue2550505
Stefan Seefeld <stefan@seefeld.name>
parents: 4095
diff changeset
2721 args = args + [x for x in search_matches]
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2722
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2723 # construct the SQL
3634
57c66056ffe4 Implemented what I'll call for now "transitive searching"...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3633
diff changeset
2724 frum.append('_'+icn)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2725 frum = ','.join(frum)
1170
af104fa52746 Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents: 1168
diff changeset
2726 if where:
af104fa52746 Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents: 1168
diff changeset
2727 where = ' where ' + (' and '.join(where))
af104fa52746 Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents: 1168
diff changeset
2728 else:
af104fa52746 Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents: 1168
diff changeset
2729 where = ''
2240
ad4b717e12b9 Small optimisation to only use "select distinct(id) ..."...
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
2730 if mlfilt:
ad4b717e12b9 Small optimisation to only use "select distinct(id) ..."...
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
2731 # we're joining tables on the id, so we will get dupes if we
ad4b717e12b9 Small optimisation to only use "select distinct(id) ..."...
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
2732 # don't distinct()
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2733 cols[0] = 'distinct(_%s.id)'%icn
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2734
3693
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
2735 order = []
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
2736 # keep correct sequence of order attributes.
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
2737 for sa in proptree.sortattr:
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
2738 if not sa.attr_sort_done:
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
2739 continue
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
2740 order.extend(sa.orderby)
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
2741 if order:
a775afeeee8b Database interface fixes.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3688
diff changeset
2742 order = ' order by %s'%(','.join(order))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2743 else:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2744 order = ''
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2745
5869
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2746 if limit is not None:
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2747 limit = ' LIMIT %s' % limit
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2748 else:
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2749 limit = ''
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2750 if offset is not None:
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2751 offset = ' OFFSET %s' % offset
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2752 else:
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2753 offset = ''
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2754 cols = ','.join(cols)
2379
a2025bdd1491 fix grouping by NULL linked values
Richard Jones <richard@users.sourceforge.net>
parents: 2374
diff changeset
2755 loj = ' '.join(loj)
5869
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2756 sql = 'select %s from %s %s %s%s%s%s'%(
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2757 cols, frum, loj, where, order, limit, offset)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2758 args = tuple(args)
2319
7cf7e3bd1b31 more column uniqueness fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2318
diff changeset
2759 __traceback_info__ = (sql, args)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2760 return proptree, sql, args
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2761
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
2762 def filter(self, search_matches, filterspec, sort=[], group=[],
5869
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2763 retired=False, exact_match_spec={}, limit=None, offset=None):
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2764 """Return a list of the ids of the active nodes in this class that
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2765 match the 'filter' spec, sorted by the group spec and then the
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2766 sort spec
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2767
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2768 "filterspec" is {propname: value(s)}
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2769
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2770 "sort" and "group" are [(dir, prop), ...] where dir is '+', '-'
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2771 or None and prop is a prop name or None. Note that for
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2772 backward-compatibility reasons a single (dir, prop) tuple is
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2773 also allowed.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2774
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2775 "search_matches" is a container type or None
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2776
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2777 The filter must match all properties specificed. If the property
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2778 value to match is a list:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2779
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2780 1. String properties must match all elements in the list, and
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2781 2. Other properties must match any of the elements in the list.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2782 """
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2783 if __debug__:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2784 start_t = time.time()
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2785
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
2786 sq = self._filter_sql (search_matches, filterspec, sort, group,
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2787 retired=retired,
5869
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2788 exact_match_spec=exact_match_spec,
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2789 limit=limit, offset=offset)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2790 # nothing to match?
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2791 if sq is None:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2792 return []
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2793 proptree, sql, args = sq
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2794
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
2795 self.db.sql(sql, args)
1911
f5c804379c85 fixed ZRoundup - mostly changes to classic template
Richard Jones <richard@users.sourceforge.net>
parents: 1906
diff changeset
2796 l = self.db.sql_fetchall()
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2797
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2798 # Compute values needed for sorting in proptree.sort
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2799 for p in proptree:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2800 if hasattr(p, 'auxcol'):
4886
a116de39e38c Fix sort-representation for RDBMS backends
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4849
diff changeset
2801 p.sort_ids = [row[p.auxcol] for row in l]
a116de39e38c Fix sort-representation for RDBMS backends
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4849
diff changeset
2802 p.sort_result = p._sort_repr (p.propclass.sort_repr, p.sort_ids)
1170
af104fa52746 Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents: 1168
diff changeset
2803 # return the IDs (the first column)
2098
18addf2a8596 Implemented proper datatypes in mysql and postgresql backends...
Richard Jones <richard@users.sourceforge.net>
parents: 2093
diff changeset
2804 # XXX numeric ids
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
2805 l = [str(row[0]) for row in l]
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2806 l = proptree.sort (l)
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
2807
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
2808 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2234
diff changeset
2809 self.db.stats['filtering'] += (time.time() - start_t)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3636
diff changeset
2810 return l
1168
94620e088e3a fixes to the rdbms backends
Richard Jones <richard@users.sourceforge.net>
parents: 1165
diff changeset
2811
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
2812 def filter_iter(self, search_matches, filterspec, sort=[], group=[],
5869
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2813 retired=False, exact_match_spec={}, limit=None,
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2814 offset=None):
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2815 """Iterator similar to filter above with same args.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2816 Limitation: We don't sort on multilinks.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2817 This uses an optimisation: We put all nodes that are in the
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2818 current row into the node cache. Then we return the node id.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2819 That way a fetch of a node won't create another sql-fetch (with
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2820 a join) from the database because the nodes are already in the
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2821 cache. We're using our own temporary cursor.
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2822 """
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
2823 sq = self._filter_sql(search_matches, filterspec, sort, group, retr=1,
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
2824 retired=retired,
5869
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2825 exact_match_spec=exact_match_spec,
16e1255b16cf Implement limit and offset for filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5867
diff changeset
2826 limit=limit, offset=offset)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2827 # nothing to match?
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2828 if sq is None:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2829 return
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2830 proptree, sql, args = sq
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2831 cursor = self.db.conn.cursor()
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2832 self.db.sql(sql, args, cursor)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2833 classes = {}
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2834 for p in proptree:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2835 if 'retrieve' in p.need_for:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2836 cn = p.parent.classname
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2837 ptid = p.parent.id # not the nodeid!
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2838 key = (cn, ptid)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2839 if key not in classes:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2840 classes[key] = {}
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2841 name = p.name
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2842 assert (name)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2843 classes[key][name] = p
4473
fccf7e09af0c - optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4472
diff changeset
2844 p.to_hyperdb = self.db.to_hyperdb_value(p.propclass.__class__)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2845 while True:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2846 row = cursor.fetchone()
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2847 if not row: break
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2848 # populate cache with current items
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
2849 for (classname, ptid), pt in classes.items():
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2850 nodeid = str(row[pt['id'].sql_idx])
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2851 key = (classname, nodeid)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2852 if key in self.db.cache:
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2853 self.db._cache_refresh(key)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2854 continue
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2855 node = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
2856 for propname, p in pt.items():
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2857 value = row[p.sql_idx]
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2858 if value is not None:
4473
fccf7e09af0c - optimisation for date:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4472
diff changeset
2859 value = p.to_hyperdb(value)
4472
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2860 node[propname] = value
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2861 self.db._cache_save(key, node)
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2862 yield str(row[0])
34dce76bb202 Multilink fixes and optimizations:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
2863
3688
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2864 def filter_sql(self, sql):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2865 """Return a list of the ids of the items in this class that match
3688
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2866 the SQL provided. The SQL is a complete "select" statement.
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2867
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2868 The SQL select must include the item id as the first column.
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2869
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2870 This function DOES NOT filter out retired items, add on a where
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
2871 clause "__retired__=0" if you don't want retired nodes.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2872 """
3688
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2873 if __debug__:
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2874 start_t = time.time()
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2875
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2876 self.db.sql(sql)
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2877 l = self.db.sql_fetchall()
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2878
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2879 if __debug__:
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2880 self.db.stats['filtering'] += (time.time() - start_t)
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2881 return l
722ab52d47fc added filter_sql to SQL backends which takes an arbitrary SQL statement...
Richard Jones <richard@users.sourceforge.net>
parents: 3687
diff changeset
2882
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2883 def count(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2884 """Get the number of nodes in this class.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2885
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2886 If the returned integer is 'numnodes', the ids of all the nodes
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2887 in this class run from 1 to numnodes, and numnodes+1 will be the
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2888 id of the next node to be created in this class.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2889 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2890 return self.db.countnodes(self.classname)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2891
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2892 # Manipulating properties:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2893 def getprops(self, protected=1):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2894 """Return a dictionary mapping property names to property objects.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2895 If the "protected" flag is true, we include protected properties -
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2896 those which may not be modified.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2897 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2898 d = self.properties.copy()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2899 if protected:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2900 d['id'] = String()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2901 d['creation'] = hyperdb.Date()
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2902 d['activity'] = hyperdb.Date()
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1175
diff changeset
2903 d['creator'] = hyperdb.Link('user')
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
2904 d['actor'] = hyperdb.Link('user')
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2905 return d
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2906
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2907 def addprop(self, **properties):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2908 """Add properties to this class.
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2909
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2910 The keyword arguments in 'properties' must map names to property
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2911 objects, or a TypeError is raised. None of the keys in 'properties'
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2912 may collide with the names of existing properties, or a ValueError
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2913 is raised before any properties have been added.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2914 """
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2915 for key in properties:
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2916 if key in self.properties:
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2917 raise ValueError(key)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2918 self.properties.update(properties)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2919
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2920 def index(self, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2921 """Add (or refresh) the node to search indexes
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2922 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2923 # find all the String properties that have indexme
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
2924 for prop, propclass in self.getprops().items():
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2925 if isinstance(propclass, String) and propclass.indexme:
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2926 self.db.indexer.add_text((self.classname, nodeid, prop),
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2927 str(self.get(nodeid, prop)))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2928
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2929 #
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2930 # import / export support
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2931 #
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2932 def export_list(self, propnames, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2933 """ Export a node - generate a list of CSV-able data in the order
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2934 specified by propnames for the given node.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2935 """
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2936 properties = self.getprops()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2937 l = []
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2938 for prop in propnames:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2939 proptype = properties[prop]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2940 value = self.get(nodeid, prop)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2941 # "marshal" data where needed
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2942 if value is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2943 pass
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2944 elif isinstance(proptype, hyperdb.Date):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2945 value = value.get_tuple()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2946 elif isinstance(proptype, hyperdb.Interval):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2947 value = value.get_tuple()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2948 elif isinstance(proptype, hyperdb.Password):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2949 value = str(value)
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2950 l.append(repr_export(value))
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2951 l.append(repr_export(self.is_retired(nodeid)))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2952 return l
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2953
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2954 def import_list(self, propnames, proplist):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2955 """ Import a node - all information including "id" is present and
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2956 should not be sanity checked. Triggers are not triggered. The
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2957 journal should be initialised using the "creator" and "created"
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2958 information.
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2959
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2960 Return the nodeid of the node imported.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
2961 """
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2962 if self.db.journaltag is None:
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
2963 raise DatabaseError(_('Database open read-only'))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2964 properties = self.getprops()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2965
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2966 # make the new node's property map
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2967 d = {}
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2968 retire = 0
2505
bdd112cf61ba rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
2969 if not "id" in propnames:
bdd112cf61ba rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
2970 newid = self.db.newid(self.classname)
bdd112cf61ba rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
2971 else:
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2972 newid = eval_import(proplist[propnames.index("id")])
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2973 for i in range(len(propnames)):
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2974 # Use eval_import to reverse the repr_export() used to
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2975 # output the CSV
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2976 value = eval_import(proplist[i])
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2977
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2978 # Figure the property for this column
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2979 propname = propnames[i]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2980
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2981 # "unmarshal" where necessary
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2982 if propname == 'id':
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2983 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2984 elif propname == 'is retired':
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2985 # is the item retired?
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2986 if int(value):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2987 retire = 1
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2988 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2989 elif value is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2990 d[propname] = None
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2991 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2992
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2993 prop = properties[propname]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2994 if value is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2995 # don't set Nones
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2996 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2997 elif isinstance(prop, hyperdb.Date):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2998 value = date.Date(value)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
2999 elif isinstance(prop, hyperdb.Interval):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3000 value = date.Interval(value)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3001 elif isinstance(prop, hyperdb.Password):
4480
1613754d2646 Fix first part of Password handling security issue2550688
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4476
diff changeset
3002 value = password.Password(encrypted=value)
3544
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
3003 elif isinstance(prop, String):
5416
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5395
diff changeset
3004 value = us2s(value)
3544
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
3005 if not isinstance(value, str):
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3006 raise TypeError('new property "%(propname)s" not a '
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3007 'string: %(value)r'%locals())
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2887
diff changeset
3008 if prop.indexme:
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2887
diff changeset
3009 self.db.indexer.add_text((self.classname, newid, propname),
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2887
diff changeset
3010 value)
3544
5cd1c83dea50 Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 3525
diff changeset
3011 d[propname] = value
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3012
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3013 # get a new id if necessary
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
3014 if newid is None:
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3015 newid = self.db.newid(self.classname)
2217
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
3016
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
3017 # insert new node or update existing?
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
3018 if not self.hasnode(newid):
98d3bf8ffb19 store Intervals as two columns (and other fixes
Richard Jones <richard@users.sourceforge.net>
parents: 2196
diff changeset
3019 self.db.addnode(self.classname, newid, d) # insert
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3020 else:
2516
125311efe783 fix invalid sql produced for multilink condition with empty value list;
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2514
diff changeset
3021 self.db.setnode(self.classname, newid, d) # update
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3022
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3023 # retire?
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3024 if retire:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3025 # use the arg for __retired__ to cope with any odd database type
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3026 # conversion (hello, sqlite)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3027 sql = 'update _%s set __retired__=%s where id=%s'%(self.classname,
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3028 self.db.arg, self.db.arg)
3963
3230f9c88086 Fix race condition for key properties in rdbms backends [SF#1876683]
Richard Jones <richard@users.sourceforge.net>
parents: 3933
diff changeset
3029 self.db.sql(sql, (newid, newid))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3030 return newid
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3031
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3032 def export_journals(self):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3033 """Export a class's journal - generate a list of lists of
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3034 CSV-able data:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3035
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3036 nodeid, date, user, action, params
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3037
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3038 No heading here - the columns are fixed.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3039 """
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3040 properties = self.getprops()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3041 r = []
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3042 for nodeid in self.getnodeids():
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
3043 for nodeid, date, user, action, params in self.history(nodeid,
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5175
diff changeset
3044 enforceperm=False, skipquiet=False):
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3045 date = date.get_tuple()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3046 if action == 'set':
3365
e2d65f6c8d83 handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents: 3328
diff changeset
3047 export_data = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
3048 for propname, value in params.items():
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3049 if propname not in properties:
3365
e2d65f6c8d83 handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents: 3328
diff changeset
3050 # property no longer in the schema
e2d65f6c8d83 handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents: 3328
diff changeset
3051 continue
e2d65f6c8d83 handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents: 3328
diff changeset
3052
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3053 prop = properties[propname]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3054 # make sure the params are eval()'able
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3055 if value is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3056 pass
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3057 elif isinstance(prop, Date):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3058 value = value.get_tuple()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3059 elif isinstance(prop, Interval):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3060 value = value.get_tuple()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3061 elif isinstance(prop, Password):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3062 value = str(value)
3365
e2d65f6c8d83 handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents: 3328
diff changeset
3063 export_data[propname] = value
e2d65f6c8d83 handle dropped properies in rdbms/metakit journal export [SF#1203569]
Richard Jones <richard@users.sourceforge.net>
parents: 3328
diff changeset
3064 params = export_data
3933
9095a4da67f9 Handle export and import of old trackers that have data attached to...
Richard Jones <richard@users.sourceforge.net>
parents: 3925
diff changeset
3065 elif action == 'create' and params:
9095a4da67f9 Handle export and import of old trackers that have data attached to...
Richard Jones <richard@users.sourceforge.net>
parents: 3925
diff changeset
3066 # old tracker with data stored in the create!
9095a4da67f9 Handle export and import of old trackers that have data attached to...
Richard Jones <richard@users.sourceforge.net>
parents: 3925
diff changeset
3067 params = {}
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3068 l = [nodeid, date, user, action, params]
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
3069 r.append(list(map(repr_export, l)))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3070 return r
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2166
diff changeset
3071
2597
c86b2179085b fix journal export of files to remove content from CSV files
Richard Jones <richard@users.sourceforge.net>
parents: 2586
diff changeset
3072 class FileClass(hyperdb.FileClass, Class):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3073 """This class defines a large chunk of data. To support this, it has a
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3074 mandatory String property "content" which is typically saved off
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3075 externally to the hyperdb.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3076
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3077 The default MIME type of this data is defined by the
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3078 "default_mime_type" class attribute, which may be overridden by each
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3079 node if the class defines a "type" String property.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3080 """
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3081 def __init__(self, db, classname, **properties):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3082 """The newly-created class automatically includes the "content"
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3083 and "type" properties.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3084 """
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3085 if 'content' not in properties:
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3086 properties['content'] = hyperdb.String(indexme='yes')
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3087 if 'type' not in properties:
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3088 properties['type'] = hyperdb.String()
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3089 Class.__init__(self, db, classname, **properties)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3090
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3091 def create(self, **propvalues):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3092 """ snaffle the file propvalue and store in a file
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3093 """
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3094 # we need to fire the auditors now, or the content property won't
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3095 # be in propvalues for the auditors to play with
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3096 self.fireAuditors('create', None, propvalues)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3097
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3098 # now remove the content property so it's not stored in the db
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3099 content = propvalues['content']
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3100 del propvalues['content']
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3101
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3102 # do the database create
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3103 newid = self.create_inner(**propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3104
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3105 # figure the mime type
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3106 mime_type = propvalues.get('type', self.default_mime_type)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3107
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3108 # and index!
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3109 if self.properties['content'].indexme:
5544
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3110 index_content = content
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3111 if bytes != str and isinstance(content, bytes):
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3112 index_content = content.decode('utf-8', errors='ignore')
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3113 self.db.indexer.add_text((self.classname, newid, 'content'),
5544
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3114 index_content, mime_type)
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3115
3979
954971d612ee Fire reactors after file storage is all done (patch [SF#2001243])
Richard Jones <richard@users.sourceforge.net>
parents: 3977
diff changeset
3116 # store off the content as a file
5492
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
3117 self.db.storefile(self.classname, newid, None, bs2b(content))
3979
954971d612ee Fire reactors after file storage is all done (patch [SF#2001243])
Richard Jones <richard@users.sourceforge.net>
parents: 3977
diff changeset
3118
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3119 # fire reactors
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3120 self.fireReactors('create', newid, None)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1417
diff changeset
3121
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3122 return newid
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3123
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3124 def get(self, nodeid, propname, default=_marker, cache=1):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3125 """ Trap the content propname and get it from the file
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
3126
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
3127 'cache' exists for backwards compatibility, and is not used.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3128 """
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3129 poss_msg = 'Possibly a access right configuration problem.'
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3130 if propname == 'content':
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3131 try:
5492
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
3132 return b2s(self.db.getfile(self.classname, nodeid, None))
5248
198b6e810c67 Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents: 5236
diff changeset
3133 except IOError as strerror:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3134 # BUG: by catching this we donot see an error in the log.
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3135 return 'ERROR reading file: %s%s\n%s\n%s'%(
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3136 self.classname, nodeid, poss_msg, strerror)
5725
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
3137 except UnicodeDecodeError as e:
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
3138 # if content is not text (e.g. jpeg file) we get
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
3139 # unicode error trying to convert to string in python 3.
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
3140 # trap it and supply an error message. Include md5sum
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
3141 # of content as this string is included in the etag
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
3142 # calculation of the object.
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
3143 return ('%s%s is not text, retrieve using '
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
3144 'binary_content property. mdsum: %s')%(self.classname,
6002
3175bb92ca28 Cleanups for bandit
John Rouillard <rouilj@ieee.org>
parents: 5871
diff changeset
3145 nodeid, md5(self.db.getfile(self.classname, nodeid, None)).hexdigest()) # nosec - bandit md5 use ok
5492
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
3146 elif propname == 'binary_content':
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
3147 return self.db.getfile(self.classname, nodeid, None)
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
3148
2692
f1c9873496f0 fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2691
diff changeset
3149 if default is not _marker:
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
3150 return Class.get(self, nodeid, propname, default)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3151 else:
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
3152 return Class.get(self, nodeid, propname)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3153
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3154 def set(self, itemid, **propvalues):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3155 """ Snarf the "content" propvalue and update it in a file
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3156 """
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3157 self.fireAuditors('set', itemid, propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3158 oldvalues = copy.deepcopy(self.db.getnode(self.classname, itemid))
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3159
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3160 # now remove the content property so it's not stored in the db
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3161 content = None
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3162 if 'content' in propvalues:
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3163 content = propvalues['content']
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3164 del propvalues['content']
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3165
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3166 # do the database create
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3167 propvalues = self.set_inner(itemid, **propvalues)
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3168
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3169 # do content?
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3170 if content:
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3171 # store and possibly index
5492
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
3172 self.db.storefile(self.classname, itemid, None, bs2b(content))
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3173 if self.properties['content'].indexme:
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3174 mime_type = self.get(itemid, 'type', self.default_mime_type)
5544
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3175 index_content = content
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3176 if bytes != str and isinstance(content, bytes):
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3177 index_content = content.decode('utf-8', errors='ignore')
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3178 self.db.indexer.add_text((self.classname, itemid, 'content'),
5544
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3179 index_content, mime_type)
2640
b01eca163779 The "type" parameter is supposed to be optional
Richard Jones <richard@users.sourceforge.net>
parents: 2634
diff changeset
3180 propvalues['content'] = content
b01eca163779 The "type" parameter is supposed to be optional
Richard Jones <richard@users.sourceforge.net>
parents: 2634
diff changeset
3181
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3182 # fire reactors
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3183 self.fireReactors('set', itemid, oldvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
3184 return propvalues
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3185
2505
bdd112cf61ba rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
3186 def index(self, nodeid):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3187 """ Add (or refresh) the node to search indexes.
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3188
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3189 Use the content-type property for the content property.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3190 """
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3191 # find all the String properties that have indexme
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5387
diff changeset
3192 for prop, propclass in self.getprops().items():
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3193 if prop == 'content' and propclass.indexme:
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3194 mime_type = self.get(nodeid, 'type', self.default_mime_type)
5544
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3195 index_content = self.get(nodeid, 'binary_content')
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3196 if bytes != str and isinstance(index_content, bytes):
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3197 index_content = index_content.decode('utf-8',
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3198 errors='ignore')
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3199 self.db.indexer.add_text((self.classname, nodeid, 'content'),
5544
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
3200 index_content, mime_type)
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3201 elif isinstance(propclass, hyperdb.String) and propclass.indexme:
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3202 # index them under (classname, nodeid, property)
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3203 try:
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3204 value = str(self.get(nodeid, prop))
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3205 except IndexError:
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3206 # node has been destroyed
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3207 continue
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
3208 self.db.indexer.add_text((self.classname, nodeid, prop), value)
2505
bdd112cf61ba rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
3209
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3210 # XXX deviation from spec - was called ItemClass
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3211 class IssueClass(Class, roundupdb.IssueClass):
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3212 # Overridden methods:
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3213 def __init__(self, db, classname, **properties):
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3214 """The newly-created class automatically includes the "messages",
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3215 "files", "nosy", and "superseder" properties. If the 'properties'
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3216 dictionary attempts to specify any of these properties or a
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
3217 "creation", "creator", "activity" or "actor" property, a ValueError
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
3218 is raised.
3834
8068eb6c704e Use """ instead of ''' for multi-line strings...
Erik Forsberg <forsberg@users.sourceforge.net>
parents: 3821
diff changeset
3219 """
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3220 if 'title' not in properties:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3221 properties['title'] = hyperdb.String(indexme='yes')
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3222 if 'messages' not in properties:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3223 properties['messages'] = hyperdb.Multilink("msg")
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3224 if 'files' not in properties:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3225 properties['files'] = hyperdb.Multilink("file")
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3226 if 'nosy' not in properties:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3227 # note: journalling is turned off as it really just wastes
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3228 # space. this behaviour may be overridden in an instance
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3229 properties['nosy'] = hyperdb.Multilink("user", do_journal="no")
4359
b9abbdd15259 another module modernised
Richard Jones <richard@users.sourceforge.net>
parents: 4113
diff changeset
3230 if 'superseder' not in properties:
1165
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3231 properties['superseder'] = hyperdb.Multilink(classname)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3232 Class.__init__(self, db, classname, **properties)
14467c765167 sqlite backend!
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3233
2692
f1c9873496f0 fix Class.get(): it was relying on self._marker...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2691
diff changeset
3234 # vim: set et sts=4 sw=4 :
Roundup Issue Tracker: http://roundup-tracker.org/