annotate roundup/backends/back_anydbm.py @ 8534:1f8492d68aca

bug: using 'null' value for attributes causes error. In rest.py, filter out any attributes that are set to 'None'. GET on an endpoint can return 'null' values when the attribute is unset. E.G. for a user: { "address": "baddy@example.com", "alternate_addresses": null, "last_login": "2026-03-18.05:57:09", "organisation": null, "password": null, "phone": null, "queries": [], "realname": "Fred Jones", "roles": "User", "timezone": null, "username": "badeggs" } But this json can not be submitted to a PUT or POST endpoint. The validators for passwords, strings, integers etc. don't expect a None value. This change handles attributes with "null" (None) values in json objects by filtering them from the python object before processing. The null value can't be used to unset an attribute via PUT or POST. The 'remove' action using the PATCH verb can unset the value. Also there appears to be some missing checks in the back_anydbm and rdbms_common files for the password type. All the other types have a check: value is not None and not isinstance(.....) but passwords only have the 'not isinstance(....)' part. Not sure why this was the case. Looking at commit history didn't make me think it was intentional.
author John Rouillard <rouilj@ieee.org>
date Wed, 18 Mar 2026 17:24:14 -0400
parents be227ab4c2e1
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
213
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
1 #
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/)
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
3 # This module is free software, and you may redistribute it and/or modify
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
4 # under the same terms as Python, so long as this copyright message and
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
5 # disclaimer are retained in their original form.
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
6 #
214
18134bffab37 stupid typo
Richard Jones <richard@users.sourceforge.net>
parents: 213
diff changeset
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR
213
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
10 # POSSIBILITY OF SUCH DAMAGE.
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
11 #
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS"
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
17 #
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
18 """This module defines a backend that saves the hyperdatabase in a
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
19 database chosen by anydbm. It is guaranteed to always be available in python
440
de5bf4191f11 Enabled transaction support in the bsddb backend.
Richard Jones <richard@users.sourceforge.net>
parents: 431
diff changeset
20 versions >2.1.1 (the dumbdbm fallback in 2.1.1 and earlier has several
de5bf4191f11 Enabled transaction support in the bsddb backend.
Richard Jones <richard@users.sourceforge.net>
parents: 431
diff changeset
21 serious bugs, and is not available)
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
22 """
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
23 __docformat__ = 'restructuredtext'
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
24
6998
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
25 import copy
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
26 import logging
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
27 import marshal
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
28 import os
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
29 import re
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
30 import shutil
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
31 import time
1809
bd127cafe3a8 Simplify backend importing, by moving the imports into the backend modules.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 1800
diff changeset
32
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
33 from roundup.anypy.dbm_ import anydbm, whichdb
8304
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
34 from roundup.anypy.strings import b2s, repr_export, eval_import, is_us
2908
95813789cf70 translate UI message
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2906
diff changeset
35
3589
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
36 from roundup import hyperdb, date, password, roundupdb, security, support
8241
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
37 from roundup.mlink_expr import Expression, ExpressionError
2908
95813789cf70 translate UI message
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2906
diff changeset
38 from roundup.backends import locking
95813789cf70 translate UI message
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2906
diff changeset
39 from roundup.i18n import _
95813789cf70 translate UI message
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2906
diff changeset
40
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
41 from roundup.backends.blobfiles import FileStorage
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
42 from roundup.backends import sessions_dbm
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
43
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
44 try:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
45 from roundup.backends import sessions_redis
6820
feb175271f3e redis import failure on python2 causes crash
John Rouillard <rouilj@ieee.org>
parents: 6814
diff changeset
46 except ImportError:
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
47 sessions_redis = None
3295
a615cc230160 added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents: 3239
diff changeset
48
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
49 from roundup.backends.indexer_common import get_indexer
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
50
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
51
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
52 def db_exists(config):
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
53 # check for the user db
7884
88a2a6079a89 fix: support dumbdbm on windows.
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
54 for db in 'nodes.user nodes.user.db nodes.user.dat'.split():
2736
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
55 if os.path.exists(os.path.join(config.DATABASE, db)):
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
56 return 1
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
57 return 0
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
58
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
59
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
60 def db_nuke(config):
2736
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
61 shutil.rmtree(config.DATABASE)
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
62
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
63
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
64 # python 3 doesn't have a unicode type
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
65 try:
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
66 unicode # noqa: F821
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
67 except NameError:
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
68 unicode = str
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
69
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
70
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
71 # marker used for an unspecified keyword argument
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
72 _marker = []
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
73
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
74
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
75 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
76 # Now the database
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
77 #
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
78
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
79
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
80 class Database(FileStorage, hyperdb.Database, roundupdb.Database):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
81 """A database for storing records containing flexible data types.
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
82
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
83 Transaction stuff TODO:
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
84
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
85 - check the timestamp of the class file and nuke the cache if it's
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
86 modified. Do some sort of conflict checking on the dirty stuff.
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
87 - perhaps detect write collisions (related to above)?
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
88
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
89 attributes:
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
90 dbtype:
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
91 holds the value for the type of db. It is used by indexer to
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
92 identify the database type so it can import the correct indexer
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
93 module when using native text search mode.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
94 """
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
95
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
96 dbtype = "anydbm"
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
97
5100
5775959d81a3 issue1926124: fix crash in roundup_admin migrate option wih anydbm
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
98 # used by migrate roundup_admin command. Is a no-op for anydbm.
5775959d81a3 issue1926124: fix crash in roundup_admin migrate option wih anydbm
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
99 # but needed to stop traceback in admin.
5775959d81a3 issue1926124: fix crash in roundup_admin migrate option wih anydbm
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
100 db_version_updated = False
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
101
524
dce4c75bef5a changed all config accesses...
Richard Jones <richard@users.sourceforge.net>
parents: 475
diff changeset
102 def __init__(self, config, journaltag=None):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
103 """Open a hyperdatabase given a specifier to some storage.
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
104
524
dce4c75bef5a changed all config accesses...
Richard Jones <richard@users.sourceforge.net>
parents: 475
diff changeset
105 The 'storagelocator' is obtained from config.DATABASE.
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
106 The meaning of 'storagelocator' depends on the particular
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
107 implementation of the hyperdatabase. It could be a file name,
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
108 a directory path, a socket descriptor for a connection to a
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
109 database over the network, etc.
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
110
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
111 The 'journaltag' is a token that will be attached to the journal
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
112 entries for any edits done on the database. If 'journaltag' is
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
113 None, the database is opened in read-only mode: the Class.create(),
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
114 Class.set(), Class.retire(), and Class.restore() methods are
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
115 disabled.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
116 """
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3872
diff changeset
117 FileStorage.__init__(self, config.UMASK)
6658
408fd477761f Add i18n object to roundupdb.Database
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6427
diff changeset
118 roundupdb.Database.__init__(self)
524
dce4c75bef5a changed all config accesses...
Richard Jones <richard@users.sourceforge.net>
parents: 475
diff changeset
119 self.config, self.journaltag = config, journaltag
dce4c75bef5a changed all config accesses...
Richard Jones <richard@users.sourceforge.net>
parents: 475
diff changeset
120 self.dir = config.DATABASE
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
121 self.classes = {}
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
122 self.cache = {} # cache of nodes loaded or created
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
123 self.stats = {'cache_hits': 0, 'cache_misses': 0, 'get_items': 0,
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
124 'filtering': 0}
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
125 self.dirtynodes = {} # keep track of the dirty nodes by class
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
126 self.newnodes = {} # keep track of the new nodes by class
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
127 self.destroyednodes = {} # keep track of the destroyed nodes by class
252
76c6994aa4e8 CGI interfaces now spit up a top-level index of all instances they can serve.
Richard Jones <richard@users.sourceforge.net>
parents: 224
diff changeset
128 self.transactions = []
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
129 self.indexer = get_indexer(config, self)
902
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
130 self.security = security.Security(self)
3609
f2fda3e6fc8b umask is now configurable (with the same 0002 default)
Richard Jones <richard@users.sourceforge.net>
parents: 3601
diff changeset
131 os.umask(config.UMASK)
440
de5bf4191f11 Enabled transaction support in the bsddb backend.
Richard Jones <richard@users.sourceforge.net>
parents: 431
diff changeset
132
5041
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 5011
diff changeset
133 # make sure the database directory exists
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 5011
diff changeset
134 if not os.path.isdir(self.config.DATABASE):
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 5011
diff changeset
135 os.makedirs(self.config.DATABASE)
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 5011
diff changeset
136
1333
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
137 # lock it
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
138 lockfilenm = os.path.join(self.dir, 'lock')
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
139 self.lockfile = locking.acquire_lock(lockfilenm)
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
140 self.lockfile.write(str(os.getpid()))
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
141 self.lockfile.flush()
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
142
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
143 self.Session = None
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
144 self.Otk = None
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
145
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
146 def post_init(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
147 """Called once the schema initialisation has finished.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
148 """
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
149 super(Database, self).post_init()
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
150 # reindex the db if necessary
826
6d7a45c8464a Added reindex command to roundup-admin.
Richard Jones <richard@users.sourceforge.net>
parents: 825
diff changeset
151 if self.indexer.should_reindex():
6d7a45c8464a Added reindex command to roundup-admin.
Richard Jones <richard@users.sourceforge.net>
parents: 825
diff changeset
152 self.reindex()
6d7a45c8464a Added reindex command to roundup-admin.
Richard Jones <richard@users.sourceforge.net>
parents: 825
diff changeset
153
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1809
diff changeset
154 def refresh_database(self):
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
155 """Rebuild the database
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
156 """
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1809
diff changeset
157 self.reindex()
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1809
diff changeset
158
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
159 def getSessionManager(self):
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
160 if not self.Session:
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
161 if self.config.SESSIONDB_BACKEND == "redis":
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
162 if sessions_redis is None:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
163 self.Session = sessions_dbm.Sessions(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
164 raise ValueError("[redis] session is set, but "
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
165 "redis is not found")
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
166 self.Session = sessions_redis.Sessions(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
167 else:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
168 self.Session = sessions_dbm.Sessions(self)
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
169 return self.Session
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
170
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
171 def getOTKManager(self):
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
172 if not self.Otk:
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
173 if self.config.SESSIONDB_BACKEND == "redis":
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
174 if sessions_redis is None:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
175 self.Session = sessions_dbm.OneTimeKeys(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
176 raise ValueError("[redis] session is set, but "
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
177 "redis is not found")
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
178 self.Otk = sessions_redis.OneTimeKeys(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
179 else:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
180 self.Otk = sessions_dbm.OneTimeKeys(self)
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
181 return self.Otk
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
182
3589
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
183 def reindex(self, classname=None, show_progress=False):
2650
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
184 if classname:
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
185 classes = [self.getclass(classname)]
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
186 else:
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
187 classes = self.classes.values()
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
188 for klass in classes:
3589
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
189 if show_progress:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
190 for nodeid in support.Progress('Reindex %s' %
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
191 klass.classname, klass.list()):
3589
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
192 klass.index(nodeid)
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
193 else:
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
194 for nodeid in klass.list():
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
195 klass.index(nodeid)
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
196 self.indexer.save_index()
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
197
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
198 def __repr__(self):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
199 return '<back_anydbm instance at %x>' % id(self)
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
200
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
201 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
202 # Classes
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
203 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
204 def __getattr__(self, classname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
205 """A convenient way of calling self.getclass(classname)."""
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
206 if classname in self.classes:
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
207 return self.classes[classname]
5378
35ea9b1efc14 Python 3 preparation: "raise" syntax.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5319
diff changeset
208 raise AttributeError(classname)
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
209
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
210 def addclass(self, cl):
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
211 cn = cl.classname
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
212 if cn in self.classes:
7750
216662fbaaee fix(i18n): fix incorrect lookup of some translations
John Rouillard <rouilj@ieee.org>
parents: 7547
diff changeset
213 raise ValueError(_('Class "%s" already defined.') % cn)
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
214 self.classes[cn] = cl
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
215
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
216 # add default Edit and View permissions
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
217 self.security.addPermission(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
218 name="Create", klass=cn,
2649
1df7d4a41da4 Buncha stuff (sorry about the large checkin):
Richard Jones <richard@users.sourceforge.net>
parents: 2640
diff changeset
219 description="User is allowed to create "+cn)
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
220 self.security.addPermission(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
221 name="Edit", klass=cn,
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
222 description="User is allowed to edit "+cn)
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
223 self.security.addPermission(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
224 name="View", klass=cn,
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
225 description="User is allowed to access "+cn)
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
226 self.security.addPermission(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
227 name="Retire", klass=cn,
4517
f8e85cf5f0fe how odd, the Retire permission wasn't being registered;
Richard Jones <richard@users.sourceforge.net>
parents: 4483
diff changeset
228 description="User is allowed to retire "+cn)
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
229
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
230 def getclasses(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
231 """Return a list of the names of all existing classes."""
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
232 return sorted(self.classes.keys())
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
233
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
234 def getclass(self, classname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
235 """Get the Class object representing a particular class.
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
236
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
237 If 'classname' is not a valid class name, a KeyError is raised.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
238 """
1145
81941abedb0a nicer error message for invalid class lookup
Richard Jones <richard@users.sourceforge.net>
parents: 1143
diff changeset
239 try:
81941abedb0a nicer error message for invalid class lookup
Richard Jones <richard@users.sourceforge.net>
parents: 1143
diff changeset
240 return self.classes[classname]
81941abedb0a nicer error message for invalid class lookup
Richard Jones <richard@users.sourceforge.net>
parents: 1143
diff changeset
241 except KeyError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
242 raise KeyError('There is no class called "%s"' % classname)
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
243
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
244 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
245 # Class DBs
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
246 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
247 def clear(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
248 """Delete all database contents
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
249 """
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
250 logging.getLogger('roundup.hyperdb.backend').info('clear')
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
251 for cn in self.classes:
7008
f92c56e74f82 in clear() clear journals and node files.
John Rouillard <rouilj@ieee.org>
parents: 7006
diff changeset
252 for data_type in 'nodes', 'journals':
f92c56e74f82 in clear() clear journals and node files.
John Rouillard <rouilj@ieee.org>
parents: 7006
diff changeset
253 path = os.path.join(self.dir, '%s.%s' % (data_type, cn))
443
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
254 if os.path.exists(path):
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
255 os.remove(path)
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
256 elif os.path.exists(path+'.db'): # dbm appends .db
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
257 os.remove(path+'.db')
2736
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
258 # reset id sequences
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
259 path = os.path.join(os.getcwd(), self.dir, '_ids')
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
260 if os.path.exists(path):
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
261 os.remove(path)
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
262 elif os.path.exists(path+'.db'): # dbm appends .db
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
263 os.remove(path+'.db')
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
264
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
265 def getclassdb(self, classname, mode='r'):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
266 """ grab a connection to the class db that will be used for
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
267 multiple actions
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
268 """
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
269 return self.opendb('nodes.%s' % classname, mode)
444
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
270
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
271 def determine_db_type(self, path):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
272 """ determine which DB wrote the class file
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
273 """
444
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
274 db_type = ''
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
275 if os.path.exists(path):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
276 db_type = whichdb(path)
444
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
277 if not db_type:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
278 raise hyperdb.DatabaseError(_("Couldn't identify database type"))
444
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
279 elif os.path.exists(path+'.db'):
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
280 # if the path ends in '.db', it's a dbm database, whether
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
281 # anydbm says it's dbhash or not!
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
282 db_type = 'dbm'
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
283 return db_type
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
284
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
285 def opendb(self, name, mode):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
286 """Low-level database opener that gets around anydbm/dbm
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
287 eccentricities.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
288 """
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
289 # figure the class db type
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
290 path = os.path.join(os.getcwd(), self.dir, name)
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
291 db_type = self.determine_db_type(path)
443
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
292
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
293 # new database? let anydbm pick the best dbm
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
294 # in Python 3+ the "dbm" ("anydbm" to us) module already uses the
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
295 # whichdb() function to do this
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
296 if not db_type or hasattr(anydbm, 'whichdb'):
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
297 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
298 logging.getLogger('roundup.hyperdb.backend').debug(
8403
be227ab4c2e1 perf: lazy evaluate debug logger command.
John Rouillard <rouilj@ieee.org>
parents: 8305
diff changeset
299 "opendb anydbm.open(%(path)r, 'c')",
be227ab4c2e1 perf: lazy evaluate debug logger command.
John Rouillard <rouilj@ieee.org>
parents: 8305
diff changeset
300 {"path": path,})
1028
16498e77e3ff allow overiding of the index args roundup/cgi/templating.py
Richard Jones <richard@users.sourceforge.net>
parents: 1022
diff changeset
301 return anydbm.open(path, 'c')
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
302
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
303 # in Python <3 it anydbm was a little dumb so manually open the
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
304 # database with the correct module
443
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
305 try:
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
306 dbm = __import__(db_type)
444
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
307 except ImportError:
6209
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
308 if db_type == 'gdbm':
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
309 try:
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
310 dbm = __import__('dbm.gnu')
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
311 except ImportError:
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
312 raise hyperdb.DatabaseError(_(
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
313 "Couldn't open database - the required module '%s' "
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
314 "(as dbm.gnu) is not available") % db_type)
6209
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
315 else:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
316 raise hyperdb.DatabaseError(_(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
317 "Couldn't open database - the "
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
318 "required module '%s' is not available") % db_type)
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
319 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
320 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
321 "opendb %r.open(%r, %r)" % (db_type, path, mode))
443
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
322 return dbm.open(path, mode)
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
323
690
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
324 #
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
325 # Node IDs
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
326 #
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
327 def newid(self, classname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
328 """ Generate a new id for the given class
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
329 """
690
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
330 # open the ids DB - create if if doesn't exist
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
331 db = self.opendb('_ids', 'c')
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
332 if classname in db:
690
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
333 newid = db[classname] = str(int(db[classname]) + 1)
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
334 else:
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
335 # the count() bit is transitional - older dbs won't start at 1
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
336 newid = str(self.getclass(classname).count()+1)
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
337 db[classname] = newid
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
338 db.close()
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
339 return newid
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
340
952
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
341 def setid(self, classname, setid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
342 """ Set the id counter: used during import of database
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
343 """
952
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
344 # open the ids DB - create if if doesn't exist
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
345 db = self.opendb('_ids', 'c')
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
346 db[classname] = str(setid)
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
347 db.close()
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
348
48
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
349 #
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
350 # Nodes
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
351 #
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
352 def addnode(self, classname, nodeid, node):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
353 """ add the specified node to its class's db
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
354 """
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
355 # we'll be supplied these props if we're doing an import
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
356 if 'creator' not in node:
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
357 # add in the "calculated" properties (dupe so we don't affect
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
358 # calling code's node assumptions)
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
359 node = node.copy()
1800
a3b1b1dcf639 Use getuid(), not figure_curuserid()
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 1794
diff changeset
360 node['creator'] = self.getuid()
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
361 node['actor'] = self.getuid()
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
362 node['creation'] = node['activity'] = date.Date()
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
363
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
364 self.newnodes.setdefault(classname, {})[nodeid] = 1
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
365 self.cache.setdefault(classname, {})[nodeid] = node
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
366 self.savenode(classname, nodeid, node)
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
367
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
368 def setnode(self, classname, nodeid, node):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
369 """ change the specified node
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
370 """
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
371 self.dirtynodes.setdefault(classname, {})[nodeid] = 1
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
372
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
373 # can't set without having already loaded the node
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
374 self.cache[classname][nodeid] = node
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
375 self.savenode(classname, nodeid, node)
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
376
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
377 def savenode(self, classname, nodeid, node):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
378 """ perform the saving of data specified by the set/addnode
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
379 """
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
380 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
381 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
382 'save %s%s %r' % (classname, nodeid, node))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
383 self.transactions.append((self.doSaveNode, (classname, nodeid, node)))
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
384
8305
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
385 def getnode(self, classname, nodeid, db=None, cache=1, allow_abort=True):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
386 """ get a node from the database
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
387
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
388 Note the "cache" parameter is not used, and exists purely for
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
389 backward compatibility!
8305
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
390
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
391 'allow_abort' is used only in sql backends.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
392 """
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
393 # try the cache
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
394 cache_dict = self.cache.setdefault(classname, {})
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
395 if nodeid in cache_dict:
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
396 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
397 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
398 'get %s%s cached' % (classname, nodeid))
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
399 self.stats['cache_hits'] += 1
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
400 return cache_dict[nodeid]
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
401
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
402 if __debug__:
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
403 self.stats['cache_misses'] += 1
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
404 start_t = time.time()
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
405 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
406 'get %s%s' % (classname, nodeid))
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
407
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
408 # get from the database and save in the cache
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
409 if db is None:
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
410 db = self.getclassdb(classname)
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
411 if nodeid not in db:
8130
9056b0c8fcd6 Fix file descriptor leak in anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents: 7936
diff changeset
412 db.close()
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
413 raise IndexError("no such %s %s" % (classname, nodeid))
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
414
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
415 # check the uncommitted, destroyed nodes
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
416 if (classname in self.destroyednodes and
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
417 nodeid in self.destroyednodes[classname]):
8130
9056b0c8fcd6 Fix file descriptor leak in anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents: 7936
diff changeset
418 db.close()
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
419 raise IndexError("no such %s %s" % (classname, nodeid))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
420
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
421 # decode
48
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
422 res = marshal.loads(db[nodeid])
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
423
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
424 # reverse the serialisation
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
425 res = self.unserialise(classname, res)
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
426
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
427 # store off in the cache dict
475
a1a44636bace Fix breakage caused by transaction changes.
Richard Jones <richard@users.sourceforge.net>
parents: 464
diff changeset
428 if cache:
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
429 cache_dict[nodeid] = res
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
430
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
431 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
432 self.stats['get_items'] += (time.time() - start_t)
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
433
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
434 return res
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
435
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
436 def destroynode(self, classname, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
437 """Remove a node from the database. Called exclusively by the
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
438 destroy() method on Class.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
439 """
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
440 logging.getLogger('roundup.hyperdb.backend').info(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
441 'destroy %s%s' % (classname, nodeid))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
442
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
443 # remove from cache and newnodes if it's there
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
444 if (classname in self.cache and nodeid in self.cache[classname]):
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
445 del self.cache[classname][nodeid]
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
446 if (classname in self.newnodes and nodeid in self.newnodes[classname]):
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
447 del self.newnodes[classname][nodeid]
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
448
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
449 # see if there's any obvious commit actions that we should get rid of
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
450 for entry in self.transactions[:]:
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
451 if entry[1][:2] == (classname, nodeid):
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
452 self.transactions.remove(entry)
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
453
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
454 # add to the destroyednodes map
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
455 self.destroyednodes.setdefault(classname, {})[nodeid] = 1
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
456
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
457 # add the destroy commit action
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
458 self.transactions.append((self.doDestroyNode, (classname, nodeid)))
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
459 self.transactions.append((FileStorage.destroy, (self, classname, nodeid)))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
460
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
461 def serialise(self, classname, node):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
462 """Copy the node contents, converting non-marshallable data into
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
463 marshallable data.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
464 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
465 properties = self.getclass(classname).getprops()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
466 d = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
467 for k, v in node.items():
2964
b71f6d0a463d merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2908
diff changeset
468 if k == self.RETIRED_FLAG:
b71f6d0a463d merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2908
diff changeset
469 d[k] = v
b71f6d0a463d merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2908
diff changeset
470 continue
b71f6d0a463d merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2908
diff changeset
471
b71f6d0a463d merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2908
diff changeset
472 # if the property doesn't exist then we really don't care
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
473 if k not in properties:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
474 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
475
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
476 # get the property spec
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
477 prop = properties[k]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
478
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
479 if isinstance(prop, hyperdb.Password) and v is not None:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
480 d[k] = str(v)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
481 elif isinstance(prop, hyperdb.Date) and v is not None:
964
832d1209aaa2 Preparing to turn back on link/unlink journal events.
Richard Jones <richard@users.sourceforge.net>
parents: 952
diff changeset
482 d[k] = v.serialise()
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
483 elif isinstance(prop, hyperdb.Interval) and v is not None:
964
832d1209aaa2 Preparing to turn back on link/unlink journal events.
Richard Jones <richard@users.sourceforge.net>
parents: 952
diff changeset
484 d[k] = v.serialise()
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
485 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
486 d[k] = v
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
487 return d
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
488
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
489 def unserialise(self, classname, node):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
490 """Decode the marshalled node data
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
491 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
492 properties = self.getclass(classname).getprops()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
493 d = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
494 for k, v in node.items():
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
495 # if the property doesn't exist, or is the "retired" flag then
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
496 # it won't be in the properties dict
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
497 if k not in properties:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
498 d[k] = v
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
499 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
500
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
501 # get the property spec
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
502 prop = properties[k]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
503
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
504 if isinstance(prop, hyperdb.Date) and v is not None:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
505 d[k] = date.Date(v)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
506 elif isinstance(prop, hyperdb.Interval) and v is not None:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
507 d[k] = date.Interval(v)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
508 elif isinstance(prop, hyperdb.Password) and v is not None:
7211
506c86823abb Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents: 7038
diff changeset
509 d[k] = password.Password(encrypted=v, config=self.config)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
510 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
511 d[k] = v
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
512 return d
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
513
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
514 def hasnode(self, classname, nodeid, db=None):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
515 """ determine if the database has a given node
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
516 """
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
517 # try the cache
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
518 cache = self.cache.setdefault(classname, {})
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
519 if nodeid in cache:
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
520 return 1
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
521
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
522 # not in the cache - check the database
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
523 if db is None:
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
524 db = self.getclassdb(classname)
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
525 return nodeid in db
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
526
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
527 def countnodes(self, classname, db=None):
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
528 count = 0
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
529
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
530 # include the uncommitted nodes
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
531 if classname in self.newnodes:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
532 count += len(self.newnodes[classname])
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
533 if classname in self.destroyednodes:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
534 count -= len(self.destroyednodes[classname])
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
535
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
536 # and count those in the DB
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
537 if db is None:
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
538 db = self.getclassdb(classname)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
539 return count + len(db)
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
540
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
541 #
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
542 # Files - special node properties
646
07abfe8f0c01 use blobfiles in back_anydbm which is used in back_bsddb.
Engelbert Gruber <grubert@users.sourceforge.net>
parents: 624
diff changeset
543 # inherited from FileStorage
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
544
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
545 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
546 # Journal
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
547 #
1143
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
548 def addjournal(self, classname, nodeid, action, params, creator=None,
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
549 creation=None):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
550 """ Journal the Action
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
551 'action' may be:
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
552
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
553 'set' -- 'params' is a dictionary of property values
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
554 'create' -- 'params' is an empty dictionary as of
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
555 Wed Nov 06 11:38:43 2002 +0000
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
556 'link' or 'unlink' -- 'params' is (classname, nodeid, propname)
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
557 'retired' or 'restored' -- 'params' is None
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
558
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
559 'creator' -- the user performing the action, which defaults to
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
560 the current user.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
561 """
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
562 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
563 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
564 'addjournal %s%s %s %r %s %r' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
565 classname, nodeid, action, params, creator, creation))
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
566 if creator is None:
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
567 creator = self.getuid()
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
568 self.transactions.append((self.doSaveJournal, (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
569 classname, nodeid, action, params, creator, creation)))
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
570
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
571 def setjournal(self, classname, nodeid, journal):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
572 """Set the journal to the "journal" list."""
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
573 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
574 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
575 'setjournal %s%s %r' % (classname, nodeid, journal))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
576 self.transactions.append((self.doSetJournal, (classname, nodeid,
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
577 journal)))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
578
4534
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
579 def fix_journal(self, classname, journal):
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
580 """ fix password entries to correct type """
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
581 pwprops = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
582 for pn, prop in self.getclass(classname).properties.items():
4534
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
583 if isinstance(prop, hyperdb.Password):
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
584 pwprops[pn] = 1
4534
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
585 if not pwprops:
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
586 return journal
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
587 for j in journal:
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
588 if j[3] == 'set':
7038
f524ddc27af8 replace for a,b in x.items() with for a in x.keys()
John Rouillard <rouilj@ieee.org>
parents: 7012
diff changeset
589 for k in j[4].keys():
4538
fd972e18b21a - fix import/export regression test for anydbm for latest journal fix
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4534
diff changeset
590 if k in pwprops and j[4][k]:
4534
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
591 j[4][k] = password.JournalPassword(j[4][k])
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
592 return journal
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
593
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
594 def getjournal(self, classname, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
595 """ get the journal for id
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
596
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
597 Raise IndexError if the node doesn't exist (as per history()'s
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
598 API)
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
599 """
1567
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
600 # our journal result
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
601 res = []
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
602
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
603 # add any journal entries for transactions not committed to the
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
604 # database
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
605 for method, args in self.transactions:
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
606 if method != self.doSaveJournal:
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
607 continue
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
608 (cache_classname, cache_nodeid, cache_action, cache_params,
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
609 cache_creator, cache_creation) = args
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
610 if cache_classname == classname and cache_nodeid == nodeid:
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
611 if not cache_creator:
1800
a3b1b1dcf639 Use getuid(), not figure_curuserid()
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 1794
diff changeset
612 cache_creator = self.getuid()
1567
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
613 if not cache_creation:
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
614 cache_creation = date.Date()
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
615 res.append((cache_nodeid, cache_creation, cache_creator,
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
616 cache_action, cache_params))
1567
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
617
48
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
618 # attempt to open the journal - in some rare cases, the journal may
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
619 # not exist
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
620 try:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
621 db = self.opendb('journals.%s' % classname, 'r')
5248
198b6e810c67 Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents: 5232
diff changeset
622 except anydbm.error as error:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
623 if str(error) == "need 'c' or 'n' flag to open new db":
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
624 raise IndexError('no such %s %s' % (classname, nodeid))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
625 elif error.args[0] != 2:
1920
f9316d2cd5ba Fixed retirement of items in rdbms imports [SF#841355]
Richard Jones <richard@users.sourceforge.net>
parents: 1904
diff changeset
626 # this isn't a "not found" error, be alarmed!
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
627 raise
1920
f9316d2cd5ba Fixed retirement of items in rdbms imports [SF#841355]
Richard Jones <richard@users.sourceforge.net>
parents: 1904
diff changeset
628 if res:
f9316d2cd5ba Fixed retirement of items in rdbms imports [SF#841355]
Richard Jones <richard@users.sourceforge.net>
parents: 1904
diff changeset
629 # we have unsaved journal entries, return them
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
630 return self.fix_journal(classname, res)
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
631 raise IndexError('no such %s %s' % (classname, nodeid))
787
b6b0a92e0738 More informative error message
Richard Jones <richard@users.sourceforge.net>
parents: 778
diff changeset
632 try:
b6b0a92e0738 More informative error message
Richard Jones <richard@users.sourceforge.net>
parents: 778
diff changeset
633 journal = marshal.loads(db[nodeid])
b6b0a92e0738 More informative error message
Richard Jones <richard@users.sourceforge.net>
parents: 778
diff changeset
634 except KeyError:
843
01f9d02faea1 ...except of course it's nice to use valid Python syntax
Richard Jones <richard@users.sourceforge.net>
parents: 842
diff changeset
635 db.close()
1567
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
636 if res:
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
637 # we have some unsaved journal entries, be happy!
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
638 return self.fix_journal(classname, res)
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
639 raise IndexError('no such %s %s' % (classname, nodeid))
843
01f9d02faea1 ...except of course it's nice to use valid Python syntax
Richard Jones <richard@users.sourceforge.net>
parents: 842
diff changeset
640 db.close()
1567
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
641
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
642 # add all the saved journal entries for this node
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
643 for nodeid, date_stamp, user, action, params in journal:
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
644 res.append((nodeid, date.Date(date_stamp), user, action, params))
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
645 return self.fix_journal(classname, res)
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
646
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
647 def pack(self, pack_before):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
648 """ Delete all journal entries except "create" before 'pack_before'.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
649 """
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
650 pack_before = pack_before.serialise()
990
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
651 for classname in self.getclasses():
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
652 packed = 0
990
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
653 # get the journal db
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
654 db_name = 'journals.%s' % classname
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
655 path = os.path.join(os.getcwd(), self.dir, classname)
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
656 db_type = self.determine_db_type(path)
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
657 db = self.opendb(db_name, 'w')
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
658
5444
167f0d25ea8e Python 3 preparation: convert dbm keys back from bytes to strings.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5395
diff changeset
659 for key in map(b2s, db.keys()):
990
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
660 # get the journal for this db entry
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
661 journal = marshal.loads(db[key])
7010
1bb7aa8414b8 change variable name l -> kept_entries
John Rouillard <rouilj@ieee.org>
parents: 7009
diff changeset
662 kept_entries = []
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
663 for entry in journal:
990
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
664 # unpack the entry
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
665 (nodeid, date_stamp, self.journaltag, action,
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
666 params) = entry
990
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
667 # if the entry is after the pack date, _or_ the initial
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
668 # create entry, then it stays
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
669 if date_stamp > pack_before or action == 'create':
7010
1bb7aa8414b8 change variable name l -> kept_entries
John Rouillard <rouilj@ieee.org>
parents: 7009
diff changeset
670 kept_entries.append(entry)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
671 else:
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
672 packed += 1
7010
1bb7aa8414b8 change variable name l -> kept_entries
John Rouillard <rouilj@ieee.org>
parents: 7009
diff changeset
673 db[key] = marshal.dumps(kept_entries)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
674
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
675 logging.getLogger('roundup.hyperdb.backend').info(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
676 'packed %d %s items' % (packed, classname))
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
677
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
678 if db_type == 'gdbm':
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
679 db.reorganize()
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
680 db.close()
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
681
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
682 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
683 # Basic transaction support
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
684 #
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
685 def commit(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
686 """ Commit the current transactions.
3687
ff9f4ca42454 Postgres backend allows transaction collisions to be ignored when...
Richard Jones <richard@users.sourceforge.net>
parents: 3682
diff changeset
687
ff9f4ca42454 Postgres backend allows transaction collisions to be ignored when...
Richard Jones <richard@users.sourceforge.net>
parents: 3682
diff changeset
688 Save all data changed since the database was opened or since the
ff9f4ca42454 Postgres backend allows transaction collisions to be ignored when...
Richard Jones <richard@users.sourceforge.net>
parents: 3682
diff changeset
689 last commit() or rollback().
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
690 """
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
691 logging.getLogger('roundup.hyperdb.backend').info(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
692 'commit %s transactions' % (len(self.transactions)))
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
693
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
694 # keep a handle to all the database files opened
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
695 self.databases = {}
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
696
1904
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
697 try:
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
698 # now, do all the transactions
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
699 reindex = {}
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
700 for method, args in self.transactions:
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
701 reindex[method(*args)] = 1
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
702 finally:
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
703 # make sure we close all the database files
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
704 for db in self.databases.values():
1904
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
705 db.close()
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
706 del self.databases
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
707
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3955
diff changeset
708 # clear the transactions list now so the blobfile implementation
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3955
diff changeset
709 # doesn't think there's still pending file commits when it tries
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3955
diff changeset
710 # to access the file data
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3955
diff changeset
711 self.transactions = []
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3955
diff changeset
712
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
713 # reindex the nodes that request it
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
714 for classname, nodeid in [k for k in reindex if k]:
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
715 self.getclass(classname).index(nodeid)
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
716
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
717 # save the indexer state
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
718 self.indexer.save_index()
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
719
1181
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
720 self.clearCache()
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
721
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
722 def clearCache(self):
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
723 # all transactions committed, back to normal
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
724 self.cache = {}
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
725 self.dirtynodes = {}
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
726 self.newnodes = {}
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
727 self.destroyednodes = {}
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
728 self.transactions = []
4652
dfbc0cfa9811 Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4538
diff changeset
729 # upcall is necessary!
dfbc0cfa9811 Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4538
diff changeset
730 roundupdb.Database.clearCache(self)
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
731
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
732 def getCachedClassDB(self, classname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
733 """ get the class db, looking in our cache of databases for commit
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
734 """
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
735 # get the database handle
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
736 db_name = 'nodes.%s' % classname
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
737 if db_name not in self.databases:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
738 self.databases[db_name] = self.getclassdb(classname, 'c')
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
739 return self.databases[db_name]
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
740
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
741 def doSaveNode(self, classname, nodeid, node):
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
742 db = self.getCachedClassDB(classname)
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
743
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
744 # now save the marshalled data
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
745 db[nodeid] = marshal.dumps(self.serialise(classname, node))
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
746
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
747 # return the classname, nodeid so we reindex this content
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
748 return (classname, nodeid)
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
749
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
750 def getCachedJournalDB(self, classname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
751 """ get the journal db, looking in our cache of databases for commit
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
752 """
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
753 # get the database handle
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
754 db_name = 'journals.%s' % classname
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
755 if db_name not in self.databases:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
756 self.databases[db_name] = self.opendb(db_name, 'c')
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
757 return self.databases[db_name]
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
758
1143
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
759 def doSaveJournal(self, classname, nodeid, action, params, creator,
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
760 creation):
1143
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
761 # serialise the parameters now if necessary
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
762 if isinstance(params, type({})):
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
763 if action in ('set', 'create'):
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
764 params = self.serialise(classname, params)
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
765
952
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
766 # handle supply of the special journalling parameters (usually
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
767 # supplied on importing an existing database)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
768 journaltag = creator
1143
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
769 if creation:
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
770 journaldate = creation.serialise()
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
771 else:
964
832d1209aaa2 Preparing to turn back on link/unlink journal events.
Richard Jones <richard@users.sourceforge.net>
parents: 952
diff changeset
772 journaldate = date.Date().serialise()
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
773
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
774 # create the journal entry
952
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
775 entry = (nodeid, journaldate, journaltag, action, params)
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
776
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
777 db = self.getCachedJournalDB(classname)
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
778
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
779 # now insert the journal entry
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
780 if nodeid in db:
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
781 # append to existing
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
782 s = db[nodeid]
650
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 646
diff changeset
783 l = marshal.loads(s)
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
784 l.append(entry)
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
785 else:
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
786 l = [entry]
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
787
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
788 db[nodeid] = marshal.dumps(l)
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
789
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
790 def doSetJournal(self, classname, nodeid, journal):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
791 l = []
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
792 for nodeid, journaldate, journaltag, action, params in journal:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
793 # serialise the parameters now if necessary
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
794 if isinstance(params, type({})):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
795 if action in ('set', 'create'):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
796 params = self.serialise(classname, params)
2512
f5542d92307a fix anydbm journal import [SF#983166]
Richard Jones <richard@users.sourceforge.net>
parents: 2505
diff changeset
797 journaldate = journaldate.serialise()
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
798 l.append((nodeid, journaldate, journaltag, action, params))
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
799 db = self.getCachedJournalDB(classname)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
800 db[nodeid] = marshal.dumps(l)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
801
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
802 def doDestroyNode(self, classname, nodeid):
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
803 # delete from the class database
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
804 db = self.getCachedClassDB(classname)
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
805 if nodeid in db:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
806 del db[nodeid]
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
807
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
808 # delete from the database
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
809 db = self.getCachedJournalDB(classname)
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
810 if nodeid in db:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
811 del db[nodeid]
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
812
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
813 def rollback(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
814 """ Reverse all actions from the current transaction.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
815 """
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
816 logging.getLogger('roundup.hyperdb.backend').info(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
817 'rollback %s transactions' % (len(self.transactions)))
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
818
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
819 for method, args in self.transactions:
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
820 # delete temporary files
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
821 if method == self.doStoreFile:
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
822 self.rollbackStoreFile(*args)
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
823 self.cache = {}
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
824 self.dirtynodes = {}
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
825 self.newnodes = {}
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
826 self.destroyednodes = {}
252
76c6994aa4e8 CGI interfaces now spit up a top-level index of all instances they can serve.
Richard Jones <richard@users.sourceforge.net>
parents: 224
diff changeset
827 self.transactions = []
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
828
1131
92e92ae58494 add close() methods where they are missing!
Richard Jones <richard@users.sourceforge.net>
parents: 1127
diff changeset
829 def close(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
830 """ Nothing to do
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
831 """
1333
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
832 if self.lockfile is not None:
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
833 locking.release_lock(self.lockfile)
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
834 self.lockfile.close()
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
835 self.lockfile = None
1131
92e92ae58494 add close() methods where they are missing!
Richard Jones <richard@users.sourceforge.net>
parents: 1127
diff changeset
836
7002
1c56bd744be4 More whiespace fixes
John Rouillard <rouilj@ieee.org>
parents: 7001
diff changeset
837
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
838 class Class(hyperdb.Class):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
839 """The handle to a particular class of nodes in a hyperdatabase."""
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
840
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
841 def enableJournalling(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
842 """Turn journalling on for this class
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
843 """
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
844 self.do_journal = 1
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
845
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
846 def disableJournalling(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
847 """Turn journalling off for this class
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
848 """
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
849 self.do_journal = 0
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
850
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
851 # Editing nodes:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
852
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
853 def create(self, **propvalues):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
854 """Create a new node of this class and return its id.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
855
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
856 The keyword arguments in 'propvalues' map property names to values.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
857
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
858 The values of arguments must be acceptable for the types of their
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
859 corresponding properties or a TypeError is raised.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
860
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
861 If this class has a key property, it must be present and its value
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
862 must not collide with other key strings or a ValueError is raised.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
863
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
864 Any other properties on this class that are missing from the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
865 'propvalues' dictionary are set to None.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
866
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
867 If an id in a link or multilink property does not refer to a valid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
868 node, an IndexError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
869
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
870 These operations trigger detectors and can be vetoed. Attempts
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
871 to modify the "creation" or "activity" properties cause a KeyError.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
872 """
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4342
diff changeset
873 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
874 raise hyperdb.DatabaseError(_('Database open read-only'))
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
875 self.fireAuditors('create', None, propvalues)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
876 newid = self.create_inner(**propvalues)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
877 self.fireReactors('create', newid, None)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
878 return newid
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
879
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
880 def create_inner(self, **propvalues):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
881 """ Called by create, in-between the audit and react calls.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
882 """
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
883 if 'id' in propvalues:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
884 raise KeyError('"id" is reserved')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
885
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
886 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
887 raise hyperdb.DatabaseError(_('Database open read-only'))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
888
7002
1c56bd744be4 More whiespace fixes
John Rouillard <rouilj@ieee.org>
parents: 7001
diff changeset
889 if ('creator' in propvalues or 'actor' in propvalues or
1c56bd744be4 More whiespace fixes
John Rouillard <rouilj@ieee.org>
parents: 7001
diff changeset
890 'creation' in propvalues or 'activity' in propvalues):
5704
aa13a517cc63 Anydbm back end didn't list creator or actor as protected properties.
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
891 raise KeyError('"creator", "actor", "creation" and '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
892 '"activity" are reserved')
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
893
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
894 for p in propvalues:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
895 prop = self.properties[p]
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
896 if prop.computed:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
897 raise KeyError('"%s" is a computed property' % p)
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
898
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
899 # new node's id
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
900 newid = self.db.newid(self.classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
901
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
902 # validate propvalues
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
903 num_re = re.compile(r'^\d+$')
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
904 for key, value in propvalues.items():
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
905 if key == self.key:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
906 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
907 self.lookup(value)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
908 except KeyError:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
909 pass
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
910 else:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
911 raise ValueError('node with key "%s" exists' % value)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
912
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
913 # try to handle this property
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
914 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
915 prop = self.properties[key]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
916 except KeyError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
917 raise KeyError('"%s" has no property "%s"' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
918 self.classname, key))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
919
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
920 if value is not None and isinstance(prop, hyperdb.Link):
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
921 if not isinstance(value, str):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
922 raise ValueError('link value must be String')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
923 link_class = self.properties[key].classname
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
924 # if it isn't a number, it's a key
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
925 if not num_re.match(value):
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
926 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
927 value = self.db.classes[link_class].lookup(value)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
928 except (TypeError, KeyError):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
929 raise IndexError('new property "%s": %s not a %s' % (
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
930 key, value, link_class))
902
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
931 elif not self.db.getclass(link_class).hasnode(value):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
932 raise IndexError('%s has no node %s' % (link_class, value))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
933
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
934 # save off the value
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
935 propvalues[key] = value
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
936
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
937 # register the link with the newly linked node
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
938 if self.do_journal and self.properties[key].do_journal:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
939 self.db.addjournal(link_class, value, 'link',
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
940 (self.classname, newid, key))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
941
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
942 elif isinstance(prop, hyperdb.Multilink):
3872
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
943 if value is None:
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
944 value = []
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
945 if not hasattr(value, '__iter__') or isinstance(value, str):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
946 raise TypeError(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
947 'new property "%s" not an iterable of ids' % key)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
948
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
949 # clean up and validate the list of links
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
950 link_class = self.properties[key].classname
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
951 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
952 for entry in value:
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
953 if not isinstance(entry, str):
7004
d3d099432ba2 Remove uneeded \
John Rouillard <rouilj@ieee.org>
parents: 7003
diff changeset
954 raise ValueError('"%s" multilink value (%r) '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
955 'must contain Strings' % (key, value))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
956 # if it isn't a number, it's a key
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
957 if not num_re.match(entry):
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
958 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
959 entry = self.db.classes[link_class].lookup(entry)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
960 except (TypeError, KeyError):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
961 raise IndexError(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
962 'new property "%s": %s not a %s' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
963 key, entry,
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
964 self.properties[key].classname))
6999
e5a28b8ded72 backout checkin 60b37f632601
John Rouillard <rouilj@ieee.org>
parents: 6998
diff changeset
965 l.append(entry)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
966 value = l
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
967 propvalues[key] = value
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
968
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
969 # handle additions
902
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
970 for nodeid in value:
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
971 if not self.db.getclass(link_class).hasnode(nodeid):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
972 raise IndexError('%s has no node %s' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
973 link_class, nodeid))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
974 # register the link with the newly linked node
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
975 if self.do_journal and self.properties[key].do_journal:
902
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
976 self.db.addjournal(link_class, nodeid, 'link',
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
977 (self.classname, newid, key))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
978
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
979 elif isinstance(prop, hyperdb.String):
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
980 if not isinstance(value, (str, unicode)):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
981 raise TypeError('new property "%s" not a string' % key)
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2736
diff changeset
982 if prop.indexme:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
983 self.db.indexer.add_text(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
984 (self.classname, newid, key), value)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
985
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
986 elif isinstance(prop, hyperdb.Password):
8534
1f8492d68aca bug: using 'null' value for attributes causes error.
John Rouillard <rouilj@ieee.org>
parents: 8403
diff changeset
987 if value is not None and not isinstance(value, password.Password):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
988 raise TypeError('new property "%s" not a Password' % key)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
989
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
990 elif isinstance(prop, hyperdb.Date):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
991 if value is not None and not isinstance(value, date.Date):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
992 raise TypeError('new property "%s" not a Date' % key)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
993
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
994 elif isinstance(prop, hyperdb.Interval):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
995 if value is not None and not isinstance(value, date.Interval):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
996 raise TypeError('new property "%s" not an Interval' % key)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
997
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
998 elif value is not None and isinstance(prop, hyperdb.Number):
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
999 try:
890
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1000 float(value)
887
e7169d6e6e45 added tests for number type too
Richard Jones <richard@users.sourceforge.net>
parents: 886
diff changeset
1001 except ValueError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1002 raise TypeError('new property "%s" not numeric' % key)
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
1003
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1004 elif value is not None and isinstance(prop, hyperdb.Integer):
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1005 try:
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1006 int(value)
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1007 except ValueError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1008 raise TypeError('new property "%s" not an integer' % key)
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1009
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1010 elif value is not None and isinstance(prop, hyperdb.Boolean):
890
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1011 try:
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1012 int(value)
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1013 except ValueError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1014 raise TypeError('new property "%s" not boolean' % key)
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
1015
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1016 # make sure there's data where there needs to be
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
1017 for key, prop in self.properties.items():
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1018 if key in propvalues:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1019 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1020 if key == self.key:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1021 raise ValueError('key property "%s" is required' % key)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1022 if isinstance(prop, hyperdb.Multilink):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1023 propvalues[key] = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1024
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1025 # done
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1026 self.db.addnode(self.classname, newid, propvalues)
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1027 if self.do_journal:
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1303
diff changeset
1028 self.db.addjournal(self.classname, newid, 'create', {})
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1029
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1030 return newid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1031
8305
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
1032 def get(self, nodeid, propname, default=_marker, cache=1, allow_abort=True):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1033 """Get the value of a property on an existing node of this class.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1034
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1035 'nodeid' must be the id of an existing node of this class or an
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1036 IndexError is raised. 'propname' must be the name of a property
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1037 of this class or a KeyError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1038
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
1039 'cache' exists for backward compatibility, and is not used.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1040
8305
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
1041 'allow_abort' is used only in sql backends.
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
1042
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1043 Attempts to get the "creation" or "activity" properties should
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1044 do the right thing.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1045 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1046 if propname == 'id':
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1047 return nodeid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1048
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1049 # get the node's dict
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
1050 d = self.db.getnode(self.classname, nodeid)
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1051
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1052 # check for one of the special props
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1053 if propname == 'creation':
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1054 if 'creation' in d:
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1055 return d['creation']
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1056 if not self.do_journal:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1057 raise ValueError('Journalling is disabled for this class')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1058 journal = self.db.getjournal(self.classname, nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1059 if journal:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents: 4075
diff changeset
1060 return journal[0][1]
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1061 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1062 # on the strange chance that there's no journal
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1063 return date.Date()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1064 if propname == 'activity':
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1065 if 'activity' in d:
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1066 return d['activity']
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1067 if not self.do_journal:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1068 raise ValueError('Journalling is disabled for this class')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1069 journal = self.db.getjournal(self.classname, nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1070 if journal:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1071 return self.db.getjournal(self.classname, nodeid)[-1][1]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1072 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1073 # on the strange chance that there's no journal
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1074 return date.Date()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1075 if propname == 'creator':
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1076 if 'creator' in d:
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1077 return d['creator']
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1078 if not self.do_journal:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1079 raise ValueError('Journalling is disabled for this class')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1080 journal = self.db.getjournal(self.classname, nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1081 if journal:
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
1082 num_re = re.compile(r'^\d+$')
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1083 value = journal[0][2]
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1084 if num_re.match(value):
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1085 return value
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1086 else:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1087 # old-style "username" journal tag
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1088 try:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1089 return self.db.user.lookup(value)
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1090 except KeyError:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1091 # user's been retired, return admin
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1092 return '1'
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1093 else:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1094 return self.db.getuid()
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1095 if propname == 'actor':
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1096 if 'actor' in d:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1097 return d['actor']
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1098 if not self.do_journal:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1099 raise ValueError('Journalling is disabled for this class')
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1100 journal = self.db.getjournal(self.classname, nodeid)
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1101 if journal:
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
1102 num_re = re.compile(r'^\d+$')
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1103 value = journal[-1][2]
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1104 if num_re.match(value):
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1105 return value
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1106 else:
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1107 # old-style "username" journal tag
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1108 try:
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1109 return self.db.user.lookup(value)
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1110 except KeyError:
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1111 # user's been retired, return admin
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1112 return '1'
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1113 else:
1800
a3b1b1dcf639 Use getuid(), not figure_curuserid()
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 1794
diff changeset
1114 return self.db.getuid()
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1115
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1116 # get the property (raises KeyErorr if invalid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1117 prop = self.properties[propname]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1118
6179
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
1119 if isinstance(prop, hyperdb.Multilink) and prop.computed:
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
1120 cls = self.db.getclass(prop.rev_classname)
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
1121 ids = cls.find(**{prop.rev_propname: nodeid})
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
1122 return ids
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
1123
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1124 if propname not in d:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1125 if default is _marker:
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1126 if isinstance(prop, hyperdb.Multilink):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1127 return []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1128 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1129 return None
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1130 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1131 return default
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1132
1014
8816534e6a1a Fixed nasty bug that was preventing changes to multilinks going through.
Richard Jones <richard@users.sourceforge.net>
parents: 1002
diff changeset
1133 # return a dupe of the list so code doesn't get confused
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1134 if isinstance(prop, hyperdb.Multilink):
6351
0db59cc2cd37 Enable testMultilinkOrdering check. Fix back_anydbm to pass.
John Rouillard <rouilj@ieee.org>
parents: 6238
diff changeset
1135 ids = d[propname][:]
0db59cc2cd37 Enable testMultilinkOrdering check. Fix back_anydbm to pass.
John Rouillard <rouilj@ieee.org>
parents: 6238
diff changeset
1136 ids.sort(key=lambda x: int(x))
0db59cc2cd37 Enable testMultilinkOrdering check. Fix back_anydbm to pass.
John Rouillard <rouilj@ieee.org>
parents: 6238
diff changeset
1137 return ids
1014
8816534e6a1a Fixed nasty bug that was preventing changes to multilinks going through.
Richard Jones <richard@users.sourceforge.net>
parents: 1002
diff changeset
1138
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1139 return d[propname]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1140
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1141 def set(self, nodeid, **propvalues):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1142 """Modify a property on an existing node of this class.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1143
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1144 'nodeid' must be the id of an existing node of this class or an
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1145 IndexError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1146
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1147 Each key in 'propvalues' must be the name of a property of this
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1148 class or a KeyError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1149
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1150 All values in 'propvalues' must be acceptable types for their
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1151 corresponding properties or a TypeError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1152
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1153 If the value of the key property is set, it must not collide with
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1154 other key strings or a ValueError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1155
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1156 If the value of a Link or Multilink property contains an invalid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1157 node id, a ValueError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1158
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1159 These operations trigger detectors and can be vetoed. Attempts
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1160 to modify the "creation" or "activity" properties cause a KeyError.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1161 """
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4342
diff changeset
1162 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1163 raise hyperdb.DatabaseError(_('Database open read-only'))
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4342
diff changeset
1164
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1165 self.fireAuditors('set', nodeid, propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1166 oldvalues = copy.deepcopy(self.db.getnode(self.classname, nodeid))
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
1167 for name, prop in self.getprops(protected=0).items():
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1168 if name in oldvalues:
2608
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
1169 continue
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1170 if isinstance(prop, hyperdb.Multilink):
2608
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
1171 oldvalues[name] = []
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
1172 else:
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
1173 oldvalues[name] = None
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1174 propvalues = self.set_inner(nodeid, **propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1175 self.fireReactors('set', nodeid, oldvalues)
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1176 return propvalues
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1177
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1178 def set_inner(self, nodeid, **propvalues):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1179 """ Called by set, in-between the audit and react calls.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1180 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1181 if not propvalues:
930
3c344e942055 Use same regex to split search terms as used to index text.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 927
diff changeset
1182 return propvalues
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1183
7002
1c56bd744be4 More whiespace fixes
John Rouillard <rouilj@ieee.org>
parents: 7001
diff changeset
1184 if ('creator' in propvalues or 'actor' in propvalues or
1c56bd744be4 More whiespace fixes
John Rouillard <rouilj@ieee.org>
parents: 7001
diff changeset
1185 'creation' in propvalues or 'activity' in propvalues):
5704
aa13a517cc63 Anydbm back end didn't list creator or actor as protected properties.
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
1186 raise KeyError('"creator", "actor", "creation" and '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1187 '"activity" are reserved')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1188
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1189 if 'id' in propvalues:
5378
35ea9b1efc14 Python 3 preparation: "raise" syntax.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5319
diff changeset
1190 raise KeyError('"id" is reserved')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1191
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1192 for p in propvalues:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1193 prop = self.properties[p]
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1194 if prop.computed:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1195 raise KeyError('"%s" is a computed property' % p)
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1196
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1197 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1198 raise hyperdb.DatabaseError(_('Database open read-only'))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1199
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1200 node = self.db.getnode(self.classname, nodeid)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1201 if self.db.RETIRED_FLAG in node:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1202 raise IndexError
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
1203 num_re = re.compile(r'^\d+$')
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1204
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1205 # if the journal value is to be different, store it in here
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1206 journalvalues = {}
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1207
5112
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1208 # omit quiet properties from history/changelog
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1209 quiet_props = []
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1210
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1211 # list() propvalues 'cos it might be modified by the loop
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1212 for propname, value in list(propvalues.items()):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1213 # check to make sure we're not duplicating an existing key
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1214 if propname == self.key and node[propname] != value:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1215 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1216 self.lookup(value)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1217 except KeyError:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1218 pass
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1219 else:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1220 raise ValueError('node with key "%s" exists' % value)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1221
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1222 # this will raise the KeyError if the property isn't valid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1223 # ... we don't use getprops() here because we only care about
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1224 # the writeable properties.
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1225 try:
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1226 prop = self.properties[propname]
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1227 except KeyError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1228 raise KeyError('"%s" has no property named "%s"' % (
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1229 self.classname, propname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1230
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1231 # if the value's the same as the existing value, no sense in
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1232 # doing anything
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1303
diff changeset
1233 current = node.get(propname, None)
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1303
diff changeset
1234 if value == current:
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1235 del propvalues[propname]
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1236 continue
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1303
diff changeset
1237 journalvalues[propname] = current
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1238
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1239 # do stuff based on the prop type
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1240 if isinstance(prop, hyperdb.Link):
927
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1241 link_class = prop.classname
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1242 # if it isn't a number, it's a key
927
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1243 if value is not None and not isinstance(value, type('')):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1244 raise ValueError('property "%s" link value be a string' % (
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1245 propname))
927
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1246 if isinstance(value, type('')) and not num_re.match(value):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1247 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1248 value = self.db.classes[link_class].lookup(value)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1249 except (TypeError, KeyError):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1250 raise IndexError('new property "%s": %s not a %s' % (
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1251 propname, value, prop.classname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1252
927
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1253 if (value is not None and
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1254 not self.db.getclass(link_class).hasnode(value)):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1255 raise IndexError('%s has no node %s' % (link_class,
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1256 value))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1257
927
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1258 if self.do_journal and prop.do_journal:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1259 # register the unlink with the old linked node
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1260 if propname in node and node[propname] is not None:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1261 self.db.addjournal(link_class, node[propname],
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1262 'unlink',
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1263 (self.classname, nodeid, propname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1264
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1265 # register the link with the newly linked node
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1266 if value is not None:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1267 self.db.addjournal(link_class, value, 'link',
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1268 (self.classname, nodeid, propname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1269
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1270 elif isinstance(prop, hyperdb.Multilink):
3872
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
1271 if value is None:
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
1272 value = []
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
1273 if not hasattr(value, '__iter__') or isinstance(value, str):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1274 raise TypeError('new property "%s" not an iterable of'
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1275 ' ids' % propname)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1276 link_class = self.properties[propname].classname
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1277 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1278 for entry in value:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1279 # if it isn't a number, it's a key
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
1280 if not isinstance(entry, str):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1281 raise ValueError('new property "%s" link value '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1282 'must be a string' % propname)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1283 if not num_re.match(entry):
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1284 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1285 entry = self.db.classes[link_class].lookup(entry)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1286 except (TypeError, KeyError):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1287 raise IndexError(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1288 'new property "%s": %s not a %s' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1289 propname, entry,
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1290 self.properties[propname].classname))
6999
e5a28b8ded72 backout checkin 60b37f632601
John Rouillard <rouilj@ieee.org>
parents: 6998
diff changeset
1291 l.append(entry)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1292 value = l
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1293 propvalues[propname] = value
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1294
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1295 # figure the journal entry for this property
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1296 add = []
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1297 remove = []
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1298
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1299 # handle removals
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1300 if propname in node:
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1301 l = node[propname]
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1302 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1303 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1304 for id in l[:]:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1305 if id in value:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1306 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1307 # register the unlink with the old linked node
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1308 if self.do_journal and self.properties[propname].do_journal:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1309 self.db.addjournal(link_class, id, 'unlink',
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1310 (self.classname, nodeid, propname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1311 l.remove(id)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1312 remove.append(id)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1313
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1314 # handle additions
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1315 for id in value:
902
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
1316 if not self.db.getclass(link_class).hasnode(id):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1317 raise IndexError('%s has no node %s' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1318 link_class, id))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1319 if id in l:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1320 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1321 # register the link with the newly linked node
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1322 if self.do_journal and self.properties[propname].do_journal:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1323 self.db.addjournal(link_class, id, 'link',
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1324 (self.classname, nodeid, propname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1325 l.append(id)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1326 add.append(id)
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1327
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1328 # figure the journal entry
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1329 l = []
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1330 if add:
967
dd35bab19dd9 use more robust date stamp comparisons in pack(), make journal smaller too
Richard Jones <richard@users.sourceforge.net>
parents: 964
diff changeset
1331 l.append(('+', add))
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1332 if remove:
967
dd35bab19dd9 use more robust date stamp comparisons in pack(), make journal smaller too
Richard Jones <richard@users.sourceforge.net>
parents: 964
diff changeset
1333 l.append(('-', remove))
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1334 if l:
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1335 journalvalues[propname] = tuple(l)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1336
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1337 elif isinstance(prop, hyperdb.String):
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1338 if value is not None and not isinstance(value, (str, unicode)):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1339 raise TypeError('new property "%s" not a '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1340 'string' % propname)
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2736
diff changeset
1341 if prop.indexme:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1342 self.db.indexer.add_text(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1343 (self.classname, nodeid, propname),
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2736
diff changeset
1344 value)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1345
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1346 elif isinstance(prop, hyperdb.Password):
8534
1f8492d68aca bug: using 'null' value for attributes causes error.
John Rouillard <rouilj@ieee.org>
parents: 8403
diff changeset
1347 if value is not None and not isinstance(value, password.Password):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1348 raise TypeError('new property "%s" not a '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1349 'Password' % propname)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1350 propvalues[propname] = value
4483
22bc0426e348 Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4480
diff changeset
1351 journalvalues[propname] = \
22bc0426e348 Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4480
diff changeset
1352 current and password.JournalPassword(current)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1353
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1354 elif value is not None and isinstance(prop, hyperdb.Date):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1355 if not isinstance(value, date.Date):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1356 raise TypeError('new property "%s" not a '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1357 'Date' % propname)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1358 propvalues[propname] = value
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1359
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1360 elif value is not None and isinstance(prop, hyperdb.Interval):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1361 if not isinstance(value, date.Interval):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1362 raise TypeError('new property "%s" not an '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1363 'Interval' % propname)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1364 propvalues[propname] = value
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1365
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1366 elif value is not None and isinstance(prop, hyperdb.Number):
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
1367 try:
890
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1368 float(value)
887
e7169d6e6e45 added tests for number type too
Richard Jones <richard@users.sourceforge.net>
parents: 886
diff changeset
1369 except ValueError:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1370 raise TypeError('new property "%s" not '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1371 'numeric' % propname)
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
1372
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1373 elif value is not None and isinstance(prop, hyperdb.Integer):
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1374 try:
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1375 int(value)
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1376 except ValueError:
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1377 raise TypeError('new property "%s" not '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1378 'numeric' % propname)
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1379
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1380 elif value is not None and isinstance(prop, hyperdb.Boolean):
890
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1381 try:
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1382 int(value)
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1383 except ValueError:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1384 raise TypeError('new property "%s" not '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1385 'boolean' % propname)
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
1386
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1387 node[propname] = value
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1388
5112
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1389 # record quiet properties to omit from history/changelog
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1390 if prop.quiet:
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1391 quiet_props.append(propname)
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1392
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1393 # nothing to do?
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1394 if not propvalues:
930
3c344e942055 Use same regex to split search terms as used to index text.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 927
diff changeset
1395 return propvalues
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1396
3083
b9a819ef7554 actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3052
diff changeset
1397 # update the activity time
b9a819ef7554 actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3052
diff changeset
1398 node['activity'] = date.Date()
b9a819ef7554 actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3052
diff changeset
1399 node['actor'] = self.db.getuid()
b9a819ef7554 actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3052
diff changeset
1400
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1401 # do the set, and journal it
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1402 self.db.setnode(self.classname, nodeid, node)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1403
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1404 if self.do_journal:
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1303
diff changeset
1405 self.db.addjournal(self.classname, nodeid, 'set', journalvalues)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1406
5112
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1407 # remove quiet properties from output
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1408 for propname in quiet_props:
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1409 if propname in propvalues:
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1410 del propvalues[propname]
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1411
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1412 return propvalues
930
3c344e942055 Use same regex to split search terms as used to index text.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 927
diff changeset
1413
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1414 def retire(self, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1415 """Retire a node.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1416
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1417 The properties on the node remain available from the get() method,
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1418 and the node's id is never reused.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1419
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1420 Retired nodes are not returned by the find(), list(), or lookup()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1421 methods, and other nodes may reuse the values of their key properties.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1422
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1423 These operations trigger detectors and can be vetoed. Attempts
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1424 to modify the "creation" or "activity" properties cause a KeyError.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1425 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1426 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1427 raise hyperdb.DatabaseError(_('Database open read-only'))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1428
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1429 self.fireAuditors('retire', nodeid, None)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1430
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1431 node = self.db.getnode(self.classname, nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1432 node[self.db.RETIRED_FLAG] = 1
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1433 self.db.setnode(self.classname, nodeid, node)
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1434 if self.do_journal:
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1435 self.db.addjournal(self.classname, nodeid, 'retired', None)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1436
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1437 self.fireReactors('retire', nodeid, None)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1438
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1439 def restore(self, nodeid):
5107
33ac07a7ef96 spelling error fixed in comment
John Rouillard <rouilj@ieee.org>
parents: 5100
diff changeset
1440 """Restore a retired node.
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1441
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1442 Make node available for all operations like it was before retirement.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1443 """
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1444 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1445 raise hyperdb.DatabaseError(_('Database open read-only'))
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1446
1523
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1447 node = self.db.getnode(self.classname, nodeid)
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1448 # check if key property was overrided
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1449 key = self.getkey()
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1450 try:
6998
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
1451 # eval for exception side effect
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
1452 id = self.lookup(node[key]) # noqa: F841
1523
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1453 except KeyError:
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1454 pass
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1455 else:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1456 raise KeyError("Key property (%s) of retired node clashes "
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1457 "with existing one (%s)" % (key, node[key]))
1523
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1458 # Now we can safely restore node
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1459 self.fireAuditors('restore', nodeid, None)
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1460 del node[self.db.RETIRED_FLAG]
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1461 self.db.setnode(self.classname, nodeid, node)
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1462 if self.do_journal:
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1463 self.db.addjournal(self.classname, nodeid, 'restored', None)
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1464
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1465 self.fireReactors('restore', nodeid, None)
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1466
8305
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
1467 def is_retired(self, nodeid, cldb=None, allow_abort=True):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1468 """Return true if the node is retired.
8305
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
1469 'allow_abort' is used only in sql backends.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1470 """
1476
5a01e90b7dc9 fixed export/import of retired nodes [SF#685273]
Richard Jones <richard@users.sourceforge.net>
parents: 1467
diff changeset
1471 node = self.db.getnode(self.classname, nodeid, cldb)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1472 if self.db.RETIRED_FLAG in node:
940
301a02ea6020 added is_retired query to Class
Richard Jones <richard@users.sourceforge.net>
parents: 930
diff changeset
1473 return 1
301a02ea6020 added is_retired query to Class
Richard Jones <richard@users.sourceforge.net>
parents: 930
diff changeset
1474 return 0
301a02ea6020 added is_retired query to Class
Richard Jones <richard@users.sourceforge.net>
parents: 930
diff changeset
1475
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1476 def destroy(self, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1477 """Destroy a node.
1333
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
1478
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1479 WARNING: this method should never be used except in extremely rare
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1480 situations where there could never be links to the node being
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1481 deleted
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1482
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1483 WARNING: use retire() instead
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1484
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1485 WARNING: the properties of this node will not be available ever again
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1486
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1487 WARNING: really, use retire() instead
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1488
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1489 Well, I think that's enough warnings. This method exists mostly to
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1490 support the session storage of the cgi interface.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1491 """
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1492 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1493 raise hyperdb.DatabaseError(_('Database open read-only'))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1494 self.db.destroynode(self.classname, nodeid)
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1495
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1496 # Locating nodes:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1497 def hasnode(self, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1498 """Determine if the given nodeid actually exists
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1499 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1500 return self.db.hasnode(self.classname, nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1501
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1502 def setkey(self, propname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1503 """Select a String property of this class to be the key property.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1504
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1505 'propname' must be the name of a String property of this class or
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1506 None, or a TypeError is raised. The values of the key property on
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
1507 all existing nodes must be unique or a ValueError is raised. If the
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
1508 property doesn't exist, KeyError is raised.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1509 """
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
1510 prop = self.getprops()[propname]
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1511 if not isinstance(prop, hyperdb.String):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1512 raise TypeError('key properties must be String')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1513 self.key = propname
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1514
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1515 def getkey(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1516 """Return the name of the key property for this class or None."""
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1517 return self.key
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1518
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1519 # TODO: set up a separate index db file for this? profile?
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1520 def lookup(self, keyvalue):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1521 """Locate a particular node by its key property and return its id.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1522
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1523 If this class has no key property, a TypeError is raised. If the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1524 'keyvalue' matches one of the values for the key property among
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1525 the nodes in this class, the matching node's id is returned;
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1526 otherwise a KeyError is raised.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1527 """
969
ddcb527491ba Consistent quoting
Richard Jones <richard@users.sourceforge.net>
parents: 967
diff changeset
1528 if not self.key:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1529 raise TypeError('No key property set for '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1530 'class %s' % self.classname)
7936
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1531
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1532 # special notation for looking up the current database user
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1533 if keyvalue == '@current_user' and self.classname == 'user':
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1534 keyvalue = self.db.user.get(self.db.getuid(), self.key)
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1535
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1536 cldb = self.db.getclassdb(self.classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1537 try:
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1538 for nodeid in self.getnodeids(cldb):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1539 node = self.db.getnode(self.classname, nodeid, cldb)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1540 if self.db.RETIRED_FLAG in node:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1541 continue
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1542 if self.key not in node:
2606
17eb5aeada7f fix bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2601
diff changeset
1543 continue
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1544 if node[self.key] == keyvalue:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1545 return nodeid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1546 finally:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1547 cldb.close()
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1548 raise KeyError('No key (%s) value "%s" for "%s"' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1549 self.key, keyvalue, self.classname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1550
1103
db787cef1385 handled some XXXs
Richard Jones <richard@users.sourceforge.net>
parents: 1099
diff changeset
1551 # change from spec - allows multiple props to match
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1552 def find(self, **propspec):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1553 """Get the ids of nodes in this class which link to the given nodes.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1554
3239
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3166
diff changeset
1555 'propspec' consists of keyword args propname=nodeid or
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3166
diff changeset
1556 propname={nodeid:1, }
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
1557 'propname' must be the name of a property in this class, or a
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
1558 KeyError is raised. That property must be a Link or
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
1559 Multilink property, or a TypeError is raised.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1560
3239
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3166
diff changeset
1561 Any node in this class whose 'propname' property links to any of
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3166
diff changeset
1562 the nodeids will be returned. Examples::
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
1563
3239
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3166
diff changeset
1564 db.issue.find(messages='1')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1565 db.issue.find(messages={'1':1,'3':1}, files={'7':1})
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1566 db.issue.find(messages=('1','3'), files=('7',))
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1567 db.issue.find(messages=['1','3'], files=['7'])
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1568 """
6427
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1569 # shortcut
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1570 if not propspec:
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1571 return []
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1572
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1573 # validate the args
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1574 props = self.getprops()
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
1575 for propname, itemids in propspec.items():
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1576 # check the prop is OK
6427
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1577 prop = props[propname]
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1578 if not isinstance(prop, hyperdb.Link) and not isinstance(prop, hyperdb.Multilink):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1579 raise TypeError("'%s' not a Link/Multilink "
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1580 "property" % propname)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1581
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1582 # ok, now do the find
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1583 cldb = self.db.getclassdb(self.classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1584 l = []
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1585 rev_multilinks = []
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1586 try:
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1587 for id in self.getnodeids(db=cldb):
1563
e2a8ce4d2317 Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents: 1558
diff changeset
1588 item = self.db.getnode(self.classname, id, db=cldb)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1589 if self.db.RETIRED_FLAG in item:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1590 continue
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
1591 for propname, itemids in propspec.items():
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1592 if not isinstance(itemids, dict):
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1593 if itemids is None or isinstance(itemids, type("")):
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1594 itemids = {itemids: 1}
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1595 else:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1596 itemids = dict.fromkeys(itemids)
1563
e2a8ce4d2317 Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents: 1558
diff changeset
1597
2450
c45ed2413044 fixed lookup of "missing" Link values for new props in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2437
diff changeset
1598 # special case if the item doesn't have this property
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1599 if propname not in item:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1600 if None in itemids:
2450
c45ed2413044 fixed lookup of "missing" Link values for new props in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2437
diff changeset
1601 l.append(id)
c45ed2413044 fixed lookup of "missing" Link values for new props in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2437
diff changeset
1602 break
c45ed2413044 fixed lookup of "missing" Link values for new props in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2437
diff changeset
1603 continue
c45ed2413044 fixed lookup of "missing" Link values for new props in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2437
diff changeset
1604
1563
e2a8ce4d2317 Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents: 1558
diff changeset
1605 # grab the property definition and its value on this item
6427
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1606 prop = props[propname]
1563
e2a8ce4d2317 Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents: 1558
diff changeset
1607 value = item[propname]
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1608 if isinstance(prop, hyperdb.Link) and value in itemids:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1609 l.append(id)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1610 break
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1611 elif isinstance(prop, hyperdb.Multilink):
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1612 if prop.rev_property:
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
1613 rev_multilinks.append((prop, itemids))
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1614 continue
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1615 hit = 0
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1616 for v in value:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1617 if v in itemids:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1618 l.append(id)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1619 hit = 1
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1620 break
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1621 if hit:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1622 break
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1623 for prop, itemids in rev_multilinks:
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1624 rprop = prop.rev_property
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1625 fun = l.append
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
1626 if isinstance(rprop, hyperdb.Multilink):
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1627 fun = l.extend
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1628 for id in itemids:
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1629 fun(rprop.cls.get(id, rprop.name))
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1630 if rev_multilinks:
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1631 l = list(sorted(set(l)))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1632 finally:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1633 cldb.close()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1634 return l
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1635
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1636 def stringFind(self, **requirements):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1637 """Locate a particular node by matching a set of its String
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1638 properties in a caseless search.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1639
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1640 If the property is not a String property, a TypeError is raised.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1641
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1642 The return is a list of the id of all nodes that match.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1643 """
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1644 for propname in requirements:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1645 prop = self.properties[propname]
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1646 if not isinstance(prop, hyperdb.String):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1647 raise TypeError("'%s' not a String property" % propname)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1648 requirements[propname] = requirements[propname].lower()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1649 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1650 cldb = self.db.getclassdb(self.classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1651 try:
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1652 for nodeid in self.getnodeids(cldb):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1653 node = self.db.getnode(self.classname, nodeid, cldb)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1654 if self.db.RETIRED_FLAG in node:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1655 continue
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
1656 for key, value in requirements.items():
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1657 if key not in node:
1296
34b1e6490e65 hardening for stringFind with missing props
Richard Jones <richard@users.sourceforge.net>
parents: 1252
diff changeset
1658 break
905
502a5ae11cc5 Very close now. The cgi and mailgw now use the new security API.
Richard Jones <richard@users.sourceforge.net>
parents: 902
diff changeset
1659 if node[key] is None or node[key].lower() != value:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1660 break
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1661 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1662 l.append(nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1663 finally:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1664 cldb.close()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1665 return l
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1666
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1667 def list(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1668 """ Return a list of the ids of the active nodes in this class.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1669 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1670 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1671 cn = self.classname
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1672 cldb = self.db.getclassdb(cn)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1673 try:
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1674 for nodeid in self.getnodeids(cldb):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1675 node = self.db.getnode(cn, nodeid, cldb)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1676 if self.db.RETIRED_FLAG in node:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1677 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1678 l.append(nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1679 finally:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1680 cldb.close()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1681 l.sort()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1682 return l
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1683
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1684 def getnodeids(self, db=None, retired=None):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1685 """ Return a list of ALL nodeids
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1686
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1687 Set retired=None to get all nodes. Otherwise it'll get all the
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1688 retired or non-retired nodes, depending on the flag.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1689 """
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1690 res = []
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1691
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1692 # start off with the new nodes
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1693 if self.classname in self.db.newnodes:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1694 res.extend(self.db.newnodes[self.classname])
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1695
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1696 must_close = False
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1697 if db is None:
1486
f5f60c75a458 added test for error in sqlite backend, and fixed *dbm backend error
Richard Jones <richard@users.sourceforge.net>
parents: 1484
diff changeset
1698 db = self.db.getclassdb(self.classname)
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
1699 must_close = True
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1700 try:
5444
167f0d25ea8e Python 3 preparation: convert dbm keys back from bytes to strings.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5395
diff changeset
1701 res.extend(map(b2s, db.keys()))
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1702
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1703 # remove the uncommitted, destroyed nodes
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1704 if self.classname in self.db.destroyednodes:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1705 for nodeid in self.db.destroyednodes[self.classname]:
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
1706 if nodeid in db:
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1707 res.remove(nodeid)
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1708
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1709 # check retired flag
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1710 if retired is False or retired is True:
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1711 l = []
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1712 for nodeid in res:
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1713 node = self.db.getnode(self.classname, nodeid, db)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1714 is_ret = self.db.RETIRED_FLAG in node
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1715 if retired == is_ret:
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1716 l.append(nodeid)
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1717 res = l
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1718 finally:
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1719 if must_close:
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1720 db.close()
7547
c8c4514f4c3e issue685275 - show retired/unretire commands
John Rouillard <rouilj@ieee.org>
parents: 7393
diff changeset
1721
c8c4514f4c3e issue685275 - show retired/unretire commands
John Rouillard <rouilj@ieee.org>
parents: 7393
diff changeset
1722 res.sort()
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1723 return res
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1724
7006
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1725 num_re = re.compile(r'^\d+$')
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1726
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1727 def _filter(self, search_matches, filterspec, proptree,
7006
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1728 num_re=num_re, retired=False,
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1729 exact_match_spec=_marker):
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1730 """Return a list of the ids of the nodes in this class that
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1731 match the 'filter' spec, sorted by the group spec and then the
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1732 sort spec.
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1733
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1734 "filterspec" is {propname: value(s)}
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1735 same for "exact_match_spec". The latter specifies exact matching
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1736 for String type while String types in "filterspec" are searched
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1737 for as case insensitive substring match.
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1738
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1739 "sort" and "group" are (dir, prop) where dir is '+', '-' or None
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1740 and prop is a prop name or None
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1741
4044
27a9906cd8d1 Fix issue2550505.
Stefan Seefeld <stefan@seefeld.name>
parents: 3979
diff changeset
1742 "search_matches" is a sequence type or None
1170
af104fa52746 Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents: 1159
diff changeset
1743
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1744 "retired" specifies if we should find only non-retired nodes
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1745 (default) or only retired node (value True) or all nodes.
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1746
3455
e01bc6d65fa9 fixed documentation of filter()...
Richard Jones <richard@users.sourceforge.net>
parents: 3419
diff changeset
1747 The filter must match all properties specificed. If the property
e01bc6d65fa9 fixed documentation of filter()...
Richard Jones <richard@users.sourceforge.net>
parents: 3419
diff changeset
1748 value to match is a list:
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
1749
3455
e01bc6d65fa9 fixed documentation of filter()...
Richard Jones <richard@users.sourceforge.net>
parents: 3419
diff changeset
1750 1. String properties must match all elements in the list, and
e01bc6d65fa9 fixed documentation of filter()...
Richard Jones <richard@users.sourceforge.net>
parents: 3419
diff changeset
1751 2. Other properties must match any of the elements in the list.
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1752 """
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
1753 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
1754 start_t = time.time()
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
1755
7006
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1756 if exact_match_spec is _marker:
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1757 exact_match_spec = {}
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1758
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1759 cn = self.classname
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1760
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1761 # optimise filterspec
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1762 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1763 props = self.getprops()
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1764 LINK = 'spec:link'
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1765 MULTILINK = 'spec:multilink'
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1766 STRING = 'spec:string'
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1767 DATE = 'spec:date'
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1768 INTERVAL = 'spec:interval'
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1769 OTHER = 'spec:other'
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1770
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1771 for exact, filtertype in enumerate((filterspec, exact_match_spec)):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1772 for k, v in filtertype.items():
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1773 propclass = props[k]
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1774 if isinstance(propclass, hyperdb.Link):
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1775 if not isinstance(v, list):
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1776 v = [v]
7936
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1777 if propclass.classname == 'user' and '@current_user' in v:
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1778 cu = self.db.getuid()
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1779 v = [x if x != "@current_user" else cu for x in v]
6412
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
1780 l.append((LINK, k, v))
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1781 elif isinstance(propclass, hyperdb.Multilink):
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1782 # If it's a reverse multilink, we've already
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1783 # computed the ids of our own class.
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1784 if propclass.rev_property:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1785 l.append((OTHER, 'id', v))
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1786 else:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1787 # the value -1 is a special "not set" sentinel
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1788 if v in ('-1', ['-1']):
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1789 v = []
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1790 elif not isinstance(v, list):
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1791 v = [v]
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1792 l.append((MULTILINK, k, v))
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1793 elif isinstance(propclass, hyperdb.String) and k != 'id':
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1794 if not isinstance(v, list):
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1795 v = [v]
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1796 for x in v:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1797 if exact:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1798 l.append((STRING, k, x))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1799 else:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1800 # simple glob searching
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1801 x = re.sub(r'([\|\{\}\\\.\+\[\]\(\)])', r'\\\1', x)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1802 x = x.replace('?', '.')
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1803 x = x.replace('*', '.*?')
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1804 l.append((STRING, k, re.compile(x, re.I)))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1805 elif isinstance(propclass, hyperdb.Date):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1806 try:
6118
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1807 ranges = []
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1808 for d in v.split(','):
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1809 if d == '-':
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1810 ranges.append(None)
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1811 continue
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1812 date_rng = propclass.range_from_raw(d, self.db)
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1813 ranges.append(date_rng)
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1814 l.append((DATE, k, ranges))
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1815 except ValueError:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1816 # If range creation fails - ignore that search parameter
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1817 pass
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1818 elif isinstance(propclass, hyperdb.Interval):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1819 try:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1820 intv_rng = date.Range(v, date.Interval)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1821 l.append((INTERVAL, k, intv_rng))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1822 except ValueError:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1823 # If range creation fails - ignore that search parameter
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1824 pass
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1825
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1826 elif isinstance(propclass, hyperdb.Boolean):
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1827 if isinstance(v, str):
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1828 v = v.split(',')
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1829 if not isinstance(v, list):
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1830 v = [v]
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1831 bv = []
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1832 for val in v:
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1833 if isinstance(val, str):
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
1834 bv.append(propclass.from_raw(val))
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1835 else:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1836 bv.append(val)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1837 l.append((OTHER, k, bv))
2453
0e2a0c2c8142 allow list of values for id, Number and Boolean filtering in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2450
diff changeset
1838
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1839 elif k == 'id':
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1840 if not isinstance(v, list):
4364
0e81742d0e2f - unify number searching across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4357
diff changeset
1841 v = v.split(',')
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1842 l.append((OTHER, k, [str(int(val)) for val in v]))
2453
0e2a0c2c8142 allow list of values for id, Number and Boolean filtering in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2450
diff changeset
1843
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1844 elif isinstance(propclass, hyperdb.Number):
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1845 if not isinstance(v, list):
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1846 try:
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1847 v = v.split(',')
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1848 except AttributeError:
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1849 v = [v]
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1850 l.append((OTHER, k, [float(val) for val in v]))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1851
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1852 elif isinstance(propclass, hyperdb.Integer):
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1853 if not isinstance(v, list):
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1854 try:
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1855 v = v.split(',')
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1856 except AttributeError:
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1857 v = [v]
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1858 l.append((OTHER, k, [int(val) for val in v]))
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1859
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1860 filterspec = l
4480
1613754d2646 Fix first part of Password handling security issue2550688
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1861
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1862 # now, find all the nodes that pass filtering
2239
c8a06e10e2c6 much faster anydbm filter(), but it breaks most filtering tests
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
1863 matches = []
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1864 cldb = self.db.getclassdb(cn)
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1865 t = 0
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1866 try:
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1867 # TODO: only full-scan once (use items())
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1868 for nodeid in self.getnodeids(cldb, retired=retired):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1869 node = self.db.getnode(cn, nodeid, cldb)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1870 # apply filter
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1871 for t, k, v in filterspec:
1303
71be6588904f fixed filtering by id in anydbm
Richard Jones <richard@users.sourceforge.net>
parents: 1296
diff changeset
1872 # handle the id prop
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1873 if k == 'id':
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1874 if nodeid not in v:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1875 break
1303
71be6588904f fixed filtering by id in anydbm
Richard Jones <richard@users.sourceforge.net>
parents: 1296
diff changeset
1876 continue
71be6588904f fixed filtering by id in anydbm
Richard Jones <richard@users.sourceforge.net>
parents: 1296
diff changeset
1877
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1878 # get the node value
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1879 nv = node.get(k, None)
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1880
2617
33fffbf7ae68 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2608
diff changeset
1881 match = 0
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1882
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1883 # now apply the property filter
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1884 if t == LINK:
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1885 # link - if this node's property doesn't appear in the
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1886 # filterspec's nodeid list, skip it
8241
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1887 try:
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1888 expr = Expression(v, is_link=True)
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1889 except ExpressionError as e:
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1890 e.context['class'] = cn
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1891 e.context['attr'] = k
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1892 raise
6412
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
1893 if expr.evaluate(nv):
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
1894 match = 1
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1895 elif t == MULTILINK:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1896 # multilink - if any of the nodeids required by the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1897 # filterspec aren't in this node's property, then skip
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1898 # it
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1899 nv = node.get(k, [])
1555
948c7764d46c implemented ability to search for multilink properties with no value (not in mk)
Richard Jones <richard@users.sourceforge.net>
parents: 1523
diff changeset
1900
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1901 # check for matching the absence of multilink values
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1902 if not v:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1903 match = not nv
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1904 else:
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4447
diff changeset
1905 # otherwise, make sure this node has each of the
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1906 # required values
8241
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1907 try:
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1908 expr = Expression(v)
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1909 except ExpressionError as e:
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1910 e.context['class'] = cn
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1911 e.context['attr'] = k
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1912 raise
6396
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6352
diff changeset
1913 if expr.evaluate(nv):
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6352
diff changeset
1914 match = 1
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1915 elif t == STRING:
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1916 if nv is None:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1917 nv = ''
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1918 if is_us(v):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1919 # Exact match
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1920 match = (nv == v)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1921 else:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1922 # RE search
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1923 match = v.search(nv)
6118
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1924 elif t == DATE:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1925 for x in v:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1926 if x is None or nv is None:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1927 if nv is None and x is None:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1928 match = 1
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1929 break
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1930 continue
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1931 elif x.to_value:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1932 if x.from_value <= nv <= x.to_value:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1933 match = 1
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1934 break
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1935 else:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1936 if x.from_value <= nv:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1937 match = 1
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1938 break
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1939 elif t == INTERVAL:
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1940 if nv is None:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1941 match = v is None
1499
8ee69708da0c added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1496
diff changeset
1942 else:
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1943 if v.to_value:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1944 if v.from_value <= nv and v.to_value >= nv:
2617
33fffbf7ae68 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2608
diff changeset
1945 match = 1
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1946 else:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1947 if v.from_value <= nv:
2617
33fffbf7ae68 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2608
diff changeset
1948 match = 1
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1949 elif t == OTHER:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1950 # straight value comparison for the other types
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1951 match = nv in v
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1952 if not match:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1953 break
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1954 else:
2239
c8a06e10e2c6 much faster anydbm filter(), but it breaks most filtering tests
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
1955 matches.append([nodeid, node])
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1956
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1957 # filter based on full text search
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1958 if search_matches is not None:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1959 k = []
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1960 for v in matches:
4044
27a9906cd8d1 Fix issue2550505.
Stefan Seefeld <stefan@seefeld.name>
parents: 3979
diff changeset
1961 if v[0] in search_matches:
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1962 k.append(v)
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1963 matches = k
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1964
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1965 # add sorting information to the proptree
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1966 JPROPS = {'actor': 1, 'activity': 1, 'creator': 1, 'creation': 1}
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1967 children = []
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1968 if proptree:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1969 children = proptree.sortable_children()
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1970 for pt in children:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1971 dir = pt.sort_direction
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1972 prop = pt.name
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1973 assert (dir and prop)
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1974 propclass = props[prop]
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1975 pt.sort_ids = []
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1976 is_pointer = isinstance(propclass, (hyperdb.Link,
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1977 hyperdb.Multilink))
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1978 if not is_pointer:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1979 pt.sort_result = []
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1980 try:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1981 # cache the opened link class db, if needed.
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1982 lcldb = None
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1983 # cache the linked class items too
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1984 lcache = {}
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1985
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1986 for entry in matches:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1987 itemid = entry[-2]
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1988 item = entry[-1]
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1989 # handle the properties that might be "faked"
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1990 # also, handle possible missing properties
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1991 try:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1992 v = item[prop]
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1993 except KeyError:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1994 if prop in JPROPS:
2437
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1995 # force lookup of the special journal prop
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1996 v = self.get(itemid, prop)
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1997 else:
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1998 # the node doesn't have a value for this
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1999 # property
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2000 v = None
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2001 if isinstance(propclass, hyperdb.Multilink):
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2002 v = []
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2003 if prop == 'id':
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
2004 v = int(itemid)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2005 pt.sort_ids.append(v)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2006 if not is_pointer:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2007 pt.sort_result.append(v)
2437
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
2008 continue
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2009
2285
c7f780c24a87 fixed anydbm sorting with None values [SF#952853]
Richard Jones <richard@users.sourceforge.net>
parents: 2252
diff changeset
2010 # missing (None) values are always sorted first
c7f780c24a87 fixed anydbm sorting with None values [SF#952853]
Richard Jones <richard@users.sourceforge.net>
parents: 2252
diff changeset
2011 if v is None:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2012 pt.sort_ids.append(v)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2013 if not is_pointer:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2014 pt.sort_result.append(v)
2285
c7f780c24a87 fixed anydbm sorting with None values [SF#952853]
Richard Jones <richard@users.sourceforge.net>
parents: 2252
diff changeset
2015 continue
c7f780c24a87 fixed anydbm sorting with None values [SF#952853]
Richard Jones <richard@users.sourceforge.net>
parents: 2252
diff changeset
2016
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2017 if isinstance(propclass, hyperdb.Link):
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2018 lcn = propclass.classname
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2019 link = self.db.classes[lcn]
3476
1142dafe0d7f added setorderprop() and setlabelprop() to Class (lots of patches)
Richard Jones <richard@users.sourceforge.net>
parents: 3455
diff changeset
2020 key = link.orderprop()
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2021 child = pt.propdict[key]
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
2022 if key != 'id':
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2023 if v not in lcache:
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2024 # open the link class db if it's not already
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2025 if lcldb is None:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2026 lcldb = self.db.getclassdb(lcn)
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2027 lcache[v] = self.db.getnode(lcn, v, lcldb)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2028 r = lcache[v][key]
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2029 child.propdict[key].sort_ids.append(r)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2030 else:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2031 child.propdict[key].sort_ids.append(v)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2032 pt.sort_ids.append(v)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2033 if not is_pointer:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2034 r = propclass.sort_repr(pt.parent.cls, v, pt.name)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2035 pt.sort_result.append(r)
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2036 finally:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2037 # if we opened the link class db, close it now
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2038 if lcldb is not None:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2039 lcldb.close()
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2040 del lcache
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2041 finally:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2042 cldb.close()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2043
2239
c8a06e10e2c6 much faster anydbm filter(), but it breaks most filtering tests
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
2044 # pull the id out of the individual entries
c8a06e10e2c6 much faster anydbm filter(), but it breaks most filtering tests
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
2045 matches = [entry[-2] for entry in matches]
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
2046 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
2047 self.db.stats['filtering'] += (time.time() - start_t)
2239
c8a06e10e2c6 much faster anydbm filter(), but it breaks most filtering tests
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
2048 return matches
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2049
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2050 def count(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2051 """Get the number of nodes in this class.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2052
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2053 If the returned integer is 'numnodes', the ids of all the nodes
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2054 in this class run from 1 to numnodes, and numnodes+1 will be the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2055 id of the next node to be created in this class.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2056 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2057 return self.db.countnodes(self.classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2058
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2059 # Manipulating properties:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2060
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2061 def getprops(self, protected=1):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2062 """Return a dictionary mapping property names to property objects.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2063 If the "protected" flag is true, we include protected properties -
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2064 those which may not be modified.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2065
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2066 In addition to the actual properties on the node, these
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2067 methods provide the "creation" and "activity" properties. If the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2068 "protected" flag is true, we include protected properties - those
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2069 which may not be modified.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2070 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2071 d = self.properties.copy()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2072 if protected:
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
2073 d['id'] = hyperdb.String()
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2074 d['creation'] = hyperdb.Date()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2075 d['activity'] = hyperdb.Date()
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
2076 d['creator'] = hyperdb.Link('user')
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
2077 d['actor'] = hyperdb.Link('user')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2078 return d
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2079
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2080 def addprop(self, **properties):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2081 """Add properties to this class.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2082
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2083 The keyword arguments in 'properties' must map names to property
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2084 objects, or a TypeError is raised. None of the keys in 'properties'
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2085 may collide with the names of existing properties, or a ValueError
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2086 is raised before any properties have been added.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2087 """
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2088 for key in properties:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2089 if key in self.properties:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2090 raise ValueError(key)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2091 self.properties.update(properties)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2092
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2093 def index(self, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2094 """ Add (or refresh) the node to search indexes """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2095 # find all the String properties that have indexme
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
2096 for prop, propclass in self.getprops().items():
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2097 if isinstance(propclass, hyperdb.String) and propclass.indexme:
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2098 # index them under (classname, nodeid, property)
7393
9e612a39547a Make anydbm indexer handle missing items the same as rdbms backend
John Rouillard <rouilj@ieee.org>
parents: 7211
diff changeset
2099 self.db.indexer.add_text((self.classname, nodeid, prop),
9e612a39547a Make anydbm indexer handle missing items the same as rdbms backend
John Rouillard <rouilj@ieee.org>
parents: 7211
diff changeset
2100 str(self.get(nodeid, prop)))
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2101
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2102 #
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2103 # import / export support
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2104 #
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2105 def export_list(self, propnames, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2106 """ Export a node - generate a list of CSV-able data in the order
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2107 specified by propnames for the given node.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2108 """
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2109 properties = self.getprops()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2110 l = []
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2111 for prop in propnames:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2112 proptype = properties[prop]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2113 value = self.get(nodeid, prop)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2114 # "marshal" data where needed
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2115 if value is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2116 pass
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2117 elif isinstance(proptype, hyperdb.Date):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2118 value = value.get_tuple()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2119 elif isinstance(proptype, hyperdb.Interval):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2120 value = value.get_tuple()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2121 elif isinstance(proptype, hyperdb.Password):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2122 value = str(value)
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2123 l.append(repr_export(value))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2124
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2125 # append retired flag
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2126 l.append(repr_export(self.is_retired(nodeid)))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2127
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2128 return l
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2129
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2130 def import_list(self, propnames, proplist):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2131 """ Import a node - all information including "id" is present and
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2132 should not be sanity checked. Triggers are not triggered. The
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2133 journal should be initialised using the "creator" and "created"
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2134 information.
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2135
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2136 Return the nodeid of the node imported.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2137 """
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2138 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2139 raise hyperdb.DatabaseError(_('Database open read-only'))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2140 properties = self.getprops()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2141
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2142 # make the new node's property map
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2143 d = {}
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2144 newid = None
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2145 for i in range(len(propnames)):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2146 # Figure the property for this column
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2147 propname = propnames[i]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2148
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2149 # Use eval_import to reverse the repr_export() used to
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2150 # output the CSV
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2151 value = eval_import(proplist[i])
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2152
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2153 # "unmarshal" where necessary
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2154 if propname == 'id':
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2155 newid = value
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2156 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2157 elif propname == 'is retired':
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2158 # is the item retired?
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2159 if int(value):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2160 d[self.db.RETIRED_FLAG] = 1
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2161 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2162 elif value is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2163 d[propname] = None
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2164 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2165
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2166 prop = properties[propname]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2167 if isinstance(prop, hyperdb.Date):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2168 value = date.Date(value)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2169 elif isinstance(prop, hyperdb.Interval):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2170 value = date.Interval(value)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2171 elif isinstance(prop, hyperdb.Password):
7211
506c86823abb Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents: 7038
diff changeset
2172 value = password.Password(encrypted=value,
506c86823abb Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents: 7038
diff changeset
2173 config=self.db.config)
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2174 d[propname] = value
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2176 # get a new id if necessary
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2177 if newid is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2178 newid = self.db.newid(self.classname)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2179
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2180 # add the node and journal
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2181 self.db.addnode(self.classname, newid, d)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2182 return newid
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2183
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2184 def export_journals(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2185 """Export a class's journal - generate a list of lists of
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2186 CSV-able data:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2187
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2188 nodeid, date, user, action, params
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2189
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2190 No heading here - the columns are fixed.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2191 """
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2192 properties = self.getprops()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2193 r = []
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2194 for nodeid in self.getnodeids():
6998
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
2195 for nodeid, date_, user, action, params in self.history(
6997
60b37f632601 Flake8 fix whitespace/indent issues.
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
2196 nodeid, enforceperm=False, skipquiet=False):
6998
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
2197 date_ = date_.get_tuple()
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2198 if action == 'set':
3003
855e9f99c96c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 2966
diff changeset
2199 export_data = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
2200 for propname, value in params.items():
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2201 if propname not in properties:
3003
855e9f99c96c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 2966
diff changeset
2202 # property no longer in the schema
855e9f99c96c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 2966
diff changeset
2203 continue
855e9f99c96c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 2966
diff changeset
2204
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2205 prop = properties[propname]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2206 # make sure the params are eval()'able
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2207 if value is None:
3826
bf2e9535da00 Journal and database testing.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3687
diff changeset
2208 pass
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
2209 elif isinstance(prop, hyperdb.Date):
3166
3204488d20b5 hack to fix some anydbm export problems [SF#1081454]
Richard Jones <richard@users.sourceforge.net>
parents: 3155
diff changeset
2210 # this is a hack - some dates are stored as strings
3204488d20b5 hack to fix some anydbm export problems [SF#1081454]
Richard Jones <richard@users.sourceforge.net>
parents: 3155
diff changeset
2211 if not isinstance(value, type('')):
3204488d20b5 hack to fix some anydbm export problems [SF#1081454]
Richard Jones <richard@users.sourceforge.net>
parents: 3155
diff changeset
2212 value = value.get_tuple()
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
2213 elif isinstance(prop, hyperdb.Interval):
3390
a2e4b8ab5b51 include hack from [SF#1238571]
Richard Jones <richard@users.sourceforge.net>
parents: 3383
diff changeset
2214 # hack too - some intervals are stored as strings
a2e4b8ab5b51 include hack from [SF#1238571]
Richard Jones <richard@users.sourceforge.net>
parents: 3383
diff changeset
2215 if not isinstance(value, type('')):
a2e4b8ab5b51 include hack from [SF#1238571]
Richard Jones <richard@users.sourceforge.net>
parents: 3383
diff changeset
2216 value = value.get_tuple()
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
2217 elif isinstance(prop, hyperdb.Password):
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2218 value = str(value)
3003
855e9f99c96c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 2966
diff changeset
2219 export_data[propname] = value
3052
230f7394cce9 fix to fix
Richard Jones <richard@users.sourceforge.net>
parents: 3003
diff changeset
2220 params = export_data
6998
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
2221 r.append([repr_export(nodeid), repr_export(date_),
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2222 repr_export(user), repr_export(action),
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2223 repr_export(params)])
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2224 return r
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2225
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
2226
2597
c86b2179085b fix journal export of files to remove content from CSV files
Richard Jones <richard@users.sourceforge.net>
parents: 2514
diff changeset
2227 class FileClass(hyperdb.FileClass, Class):
8304
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2228 # Use for explicit upcalls in generic code, for py2 compat we cannot
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2229 # use super() without making everything a new-style class.
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2230 subclass = Class
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2231 def __init__(self, db, classname, **properties):
8304
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2232 self._update_properties(properties)
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2233 Class.__init__(self, db, classname, **properties)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2234
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2235 class IssueClass(Class, roundupdb.IssueClass):
8304
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2236 # Use for explicit upcalls in generic code, for py2 compat we cannot
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2237 # use super() without making everything a new-style class.
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2238 subclass = Class
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2239 def __init__(self, db, classname, **properties):
8304
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2240 self._update_properties(classname, properties)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2241 Class.__init__(self, db, classname, **properties)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2242
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
2243 # vim: set et sts=4 sw=4 :

Roundup Issue Tracker: http://roundup-tracker.org/