Mercurial > p > roundup > code

annotate roundup/backends/back_anydbm.py @ 8540:e8d1da6e3571

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
bug: fix traceback in roundup-admin init with bad config values initialize accepts setting values for config.ini file settings. If they are not valid, you got a python traceback. ConfigurationError exceptions are now trapped. The admin.py's usageError_feedback method is used to inform the user. Also the feedback message now starts with a newline making it easier to read by separating it from command that caused the issue.
author John Rouillard <rouilj@ieee.org>
date Mon, 23 Mar 2026 13:18:41 -0400
parents 1f8492d68aca
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
213
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
1 #
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/)
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
3 # This module is free software, and you may redistribute it and/or modify
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
4 # under the same terms as Python, so long as this copyright message and
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
5 # disclaimer are retained in their original form.
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
6 #
214
18134bffab37 stupid typo
Richard Jones <richard@users.sourceforge.net>
parents: 213
diff changeset
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR
213
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
10 # POSSIBILITY OF SUCH DAMAGE.
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
11 #
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS"
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
17 #
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
18 """This module defines a backend that saves the hyperdatabase in a
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
19 database chosen by anydbm. It is guaranteed to always be available in python
440
de5bf4191f11 Enabled transaction support in the bsddb backend.
Richard Jones <richard@users.sourceforge.net>
parents: 431
diff changeset
20 versions >2.1.1 (the dumbdbm fallback in 2.1.1 and earlier has several
de5bf4191f11 Enabled transaction support in the bsddb backend.
Richard Jones <richard@users.sourceforge.net>
parents: 431
diff changeset
21 serious bugs, and is not available)
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
22 """
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
23 __docformat__ = 'restructuredtext'
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
24
6998
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
25 import copy
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
26 import logging
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
27 import marshal
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
28 import os
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
29 import re
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
30 import shutil
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
31 import time
1809
bd127cafe3a8 Simplify backend importing, by moving the imports into the backend modules.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 1800
diff changeset
32
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
33 from roundup.anypy.dbm_ import anydbm, whichdb
8304
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
34 from roundup.anypy.strings import b2s, repr_export, eval_import, is_us
2908
95813789cf70 translate UI message
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2906
diff changeset
35
3589
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
36 from roundup import hyperdb, date, password, roundupdb, security, support
8241
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
37 from roundup.mlink_expr import Expression, ExpressionError
2908
95813789cf70 translate UI message
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2906
diff changeset
38 from roundup.backends import locking
95813789cf70 translate UI message
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2906
diff changeset
39 from roundup.i18n import _
95813789cf70 translate UI message
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2906
diff changeset
40
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
41 from roundup.backends.blobfiles import FileStorage
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
42 from roundup.backends import sessions_dbm
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
43
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
44 try:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
45 from roundup.backends import sessions_redis
6820
feb175271f3e redis import failure on python2 causes crash
John Rouillard <rouilj@ieee.org>
parents: 6814
diff changeset
46 except ImportError:
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
47 sessions_redis = None
3295
a615cc230160 added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents: 3239
diff changeset
48
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
49 from roundup.backends.indexer_common import get_indexer
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
50
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
51
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
52 def db_exists(config):
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
53 # check for the user db
7884
88a2a6079a89 fix: support dumbdbm on windows.
John Rouillard <rouilj@ieee.org>
parents: 7750
diff changeset
54 for db in 'nodes.user nodes.user.db nodes.user.dat'.split():
2736
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
55 if os.path.exists(os.path.join(config.DATABASE, db)):
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
56 return 1
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
57 return 0
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
58
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
59
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
60 def db_nuke(config):
2736
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
61 shutil.rmtree(config.DATABASE)
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
62
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
63
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
64 # python 3 doesn't have a unicode type
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
65 try:
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
66 unicode # noqa: F821
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
67 except NameError:
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
68 unicode = str
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
69
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
70
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
71 # marker used for an unspecified keyword argument
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
72 _marker = []
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
73
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
74
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
75 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
76 # Now the database
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
77 #
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
78
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
79
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
80 class Database(FileStorage, hyperdb.Database, roundupdb.Database):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
81 """A database for storing records containing flexible data types.
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
82
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
83 Transaction stuff TODO:
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
84
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
85 - check the timestamp of the class file and nuke the cache if it's
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
86 modified. Do some sort of conflict checking on the dirty stuff.
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
87 - perhaps detect write collisions (related to above)?
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
88
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
89 attributes:
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
90 dbtype:
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
91 holds the value for the type of db. It is used by indexer to
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
92 identify the database type so it can import the correct indexer
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
93 module when using native text search mode.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
94 """
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
95
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
96 dbtype = "anydbm"
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
97
5100
5775959d81a3 issue1926124: fix crash in roundup_admin migrate option wih anydbm
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
98 # used by migrate roundup_admin command. Is a no-op for anydbm.
5775959d81a3 issue1926124: fix crash in roundup_admin migrate option wih anydbm
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
99 # but needed to stop traceback in admin.
5775959d81a3 issue1926124: fix crash in roundup_admin migrate option wih anydbm
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
100 db_version_updated = False
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
101
524
dce4c75bef5a changed all config accesses...
Richard Jones <richard@users.sourceforge.net>
parents: 475
diff changeset
102 def __init__(self, config, journaltag=None):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
103 """Open a hyperdatabase given a specifier to some storage.
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
104
524
dce4c75bef5a changed all config accesses...
Richard Jones <richard@users.sourceforge.net>
parents: 475
diff changeset
105 The 'storagelocator' is obtained from config.DATABASE.
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
106 The meaning of 'storagelocator' depends on the particular
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
107 implementation of the hyperdatabase. It could be a file name,
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
108 a directory path, a socket descriptor for a connection to a
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
109 database over the network, etc.
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
110
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
111 The 'journaltag' is a token that will be attached to the journal
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
112 entries for any edits done on the database. If 'journaltag' is
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
113 None, the database is opened in read-only mode: the Class.create(),
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
114 Class.set(), Class.retire(), and Class.restore() methods are
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
115 disabled.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
116 """
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3872
diff changeset
117 FileStorage.__init__(self, config.UMASK)
6658
408fd477761f Add i18n object to roundupdb.Database
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6427
diff changeset
118 roundupdb.Database.__init__(self)
524
dce4c75bef5a changed all config accesses...
Richard Jones <richard@users.sourceforge.net>
parents: 475
diff changeset
119 self.config, self.journaltag = config, journaltag
dce4c75bef5a changed all config accesses...
Richard Jones <richard@users.sourceforge.net>
parents: 475
diff changeset
120 self.dir = config.DATABASE
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
121 self.classes = {}
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
122 self.cache = {} # cache of nodes loaded or created
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
123 self.stats = {'cache_hits': 0, 'cache_misses': 0, 'get_items': 0,
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
124 'filtering': 0}
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
125 self.dirtynodes = {} # keep track of the dirty nodes by class
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
126 self.newnodes = {} # keep track of the new nodes by class
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
127 self.destroyednodes = {} # keep track of the destroyed nodes by class
252
76c6994aa4e8 CGI interfaces now spit up a top-level index of all instances they can serve.
Richard Jones <richard@users.sourceforge.net>
parents: 224
diff changeset
128 self.transactions = []
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
129 self.indexer = get_indexer(config, self)
902
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
130 self.security = security.Security(self)
3609
f2fda3e6fc8b umask is now configurable (with the same 0002 default)
Richard Jones <richard@users.sourceforge.net>
parents: 3601
diff changeset
131 os.umask(config.UMASK)
440
de5bf4191f11 Enabled transaction support in the bsddb backend.
Richard Jones <richard@users.sourceforge.net>
parents: 431
diff changeset
132
5041
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 5011
diff changeset
133 # make sure the database directory exists
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 5011
diff changeset
134 if not os.path.isdir(self.config.DATABASE):
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 5011
diff changeset
135 os.makedirs(self.config.DATABASE)
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 5011
diff changeset
136
1333
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
137 # lock it
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
138 lockfilenm = os.path.join(self.dir, 'lock')
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
139 self.lockfile = locking.acquire_lock(lockfilenm)
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
140 self.lockfile.write(str(os.getpid()))
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
141 self.lockfile.flush()
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
142
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
143 self.Session = None
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
144 self.Otk = None
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
145
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
146 def post_init(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
147 """Called once the schema initialisation has finished.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
148 """
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
149 super(Database, self).post_init()
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
150 # reindex the db if necessary
826
6d7a45c8464a Added reindex command to roundup-admin.
Richard Jones <richard@users.sourceforge.net>
parents: 825
diff changeset
151 if self.indexer.should_reindex():
6d7a45c8464a Added reindex command to roundup-admin.
Richard Jones <richard@users.sourceforge.net>
parents: 825
diff changeset
152 self.reindex()
6d7a45c8464a Added reindex command to roundup-admin.
Richard Jones <richard@users.sourceforge.net>
parents: 825
diff changeset
153
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1809
diff changeset
154 def refresh_database(self):
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
155 """Rebuild the database
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
156 """
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1809
diff changeset
157 self.reindex()
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1809
diff changeset
158
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
159 def getSessionManager(self):
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
160 if not self.Session:
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
161 if self.config.SESSIONDB_BACKEND == "redis":
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
162 if sessions_redis is None:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
163 self.Session = sessions_dbm.Sessions(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
164 raise ValueError("[redis] session is set, but "
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
165 "redis is not found")
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
166 self.Session = sessions_redis.Sessions(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
167 else:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
168 self.Session = sessions_dbm.Sessions(self)
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
169 return self.Session
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
170
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
171 def getOTKManager(self):
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
172 if not self.Otk:
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
173 if self.config.SESSIONDB_BACKEND == "redis":
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
174 if sessions_redis is None:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
175 self.Session = sessions_dbm.OneTimeKeys(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
176 raise ValueError("[redis] session is set, but "
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
177 "redis is not found")
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
178 self.Otk = sessions_redis.OneTimeKeys(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
179 else:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
180 self.Otk = sessions_dbm.OneTimeKeys(self)
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
181 return self.Otk
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
182
3589
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
183 def reindex(self, classname=None, show_progress=False):
2650
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
184 if classname:
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
185 classes = [self.getclass(classname)]
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
186 else:
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
187 classes = self.classes.values()
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
188 for klass in classes:
3589
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
189 if show_progress:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
190 for nodeid in support.Progress('Reindex %s' %
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
191 klass.classname, klass.list()):
3589
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
192 klass.index(nodeid)
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
193 else:
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
194 for nodeid in klass.list():
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
195 klass.index(nodeid)
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
196 self.indexer.save_index()
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
197
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
198 def __repr__(self):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
199 return '<back_anydbm instance at %x>' % id(self)
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
200
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
201 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
202 # Classes
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
203 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
204 def __getattr__(self, classname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
205 """A convenient way of calling self.getclass(classname)."""
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
206 if classname in self.classes:
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
207 return self.classes[classname]
5378
35ea9b1efc14 Python 3 preparation: "raise" syntax.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5319
diff changeset
208 raise AttributeError(classname)
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
209
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
210 def addclass(self, cl):
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
211 cn = cl.classname
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
212 if cn in self.classes:
7750
216662fbaaee fix(i18n): fix incorrect lookup of some translations
John Rouillard <rouilj@ieee.org>
parents: 7547
diff changeset
213 raise ValueError(_('Class "%s" already defined.') % cn)
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
214 self.classes[cn] = cl
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
215
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
216 # add default Edit and View permissions
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
217 self.security.addPermission(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
218 name="Create", klass=cn,
2649
1df7d4a41da4 Buncha stuff (sorry about the large checkin):
Richard Jones <richard@users.sourceforge.net>
parents: 2640
diff changeset
219 description="User is allowed to create "+cn)
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
220 self.security.addPermission(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
221 name="Edit", klass=cn,
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
222 description="User is allowed to edit "+cn)
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
223 self.security.addPermission(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
224 name="View", klass=cn,
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
225 description="User is allowed to access "+cn)
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
226 self.security.addPermission(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
227 name="Retire", klass=cn,
4517
f8e85cf5f0fe how odd, the Retire permission wasn't being registered;
Richard Jones <richard@users.sourceforge.net>
parents: 4483
diff changeset
228 description="User is allowed to retire "+cn)
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
229
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
230 def getclasses(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
231 """Return a list of the names of all existing classes."""
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
232 return sorted(self.classes.keys())
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
233
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
234 def getclass(self, classname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
235 """Get the Class object representing a particular class.
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
236
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
237 If 'classname' is not a valid class name, a KeyError is raised.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
238 """
1145
81941abedb0a nicer error message for invalid class lookup
Richard Jones <richard@users.sourceforge.net>
parents: 1143
diff changeset
239 try:
81941abedb0a nicer error message for invalid class lookup
Richard Jones <richard@users.sourceforge.net>
parents: 1143
diff changeset
240 return self.classes[classname]
81941abedb0a nicer error message for invalid class lookup
Richard Jones <richard@users.sourceforge.net>
parents: 1143
diff changeset
241 except KeyError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
242 raise KeyError('There is no class called "%s"' % classname)
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
243
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
244 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
245 # Class DBs
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
246 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
247 def clear(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
248 """Delete all database contents
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
249 """
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
250 logging.getLogger('roundup.hyperdb.backend').info('clear')
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
251 for cn in self.classes:
7008
f92c56e74f82 in clear() clear journals and node files.
John Rouillard <rouilj@ieee.org>
parents: 7006
diff changeset
252 for data_type in 'nodes', 'journals':
f92c56e74f82 in clear() clear journals and node files.
John Rouillard <rouilj@ieee.org>
parents: 7006
diff changeset
253 path = os.path.join(self.dir, '%s.%s' % (data_type, cn))
443
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
254 if os.path.exists(path):
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
255 os.remove(path)
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
256 elif os.path.exists(path+'.db'): # dbm appends .db
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
257 os.remove(path+'.db')
2736
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
258 # reset id sequences
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
259 path = os.path.join(os.getcwd(), self.dir, '_ids')
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
260 if os.path.exists(path):
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
261 os.remove(path)
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
262 elif os.path.exists(path+'.db'): # dbm appends .db
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
263 os.remove(path+'.db')
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
264
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
265 def getclassdb(self, classname, mode='r'):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
266 """ grab a connection to the class db that will be used for
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
267 multiple actions
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
268 """
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
269 return self.opendb('nodes.%s' % classname, mode)
444
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
270
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
271 def determine_db_type(self, path):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
272 """ determine which DB wrote the class file
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
273 """
444
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
274 db_type = ''
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
275 if os.path.exists(path):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
276 db_type = whichdb(path)
444
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
277 if not db_type:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
278 raise hyperdb.DatabaseError(_("Couldn't identify database type"))
444
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
279 elif os.path.exists(path+'.db'):
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
280 # if the path ends in '.db', it's a dbm database, whether
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
281 # anydbm says it's dbhash or not!
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
282 db_type = 'dbm'
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
283 return db_type
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
284
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
285 def opendb(self, name, mode):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
286 """Low-level database opener that gets around anydbm/dbm
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
287 eccentricities.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
288 """
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
289 # figure the class db type
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
290 path = os.path.join(os.getcwd(), self.dir, name)
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
291 db_type = self.determine_db_type(path)
443
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
292
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
293 # new database? let anydbm pick the best dbm
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
294 # in Python 3+ the "dbm" ("anydbm" to us) module already uses the
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
295 # whichdb() function to do this
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
296 if not db_type or hasattr(anydbm, 'whichdb'):
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
297 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
298 logging.getLogger('roundup.hyperdb.backend').debug(
8403
be227ab4c2e1 perf: lazy evaluate debug logger command.
John Rouillard <rouilj@ieee.org>
parents: 8305
diff changeset
299 "opendb anydbm.open(%(path)r, 'c')",
be227ab4c2e1 perf: lazy evaluate debug logger command.
John Rouillard <rouilj@ieee.org>
parents: 8305
diff changeset
300 {"path": path,})
1028
16498e77e3ff allow overiding of the index args roundup/cgi/templating.py
Richard Jones <richard@users.sourceforge.net>
parents: 1022
diff changeset
301 return anydbm.open(path, 'c')
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
302
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
303 # in Python <3 it anydbm was a little dumb so manually open the
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
304 # database with the correct module
443
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
305 try:
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
306 dbm = __import__(db_type)
444
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
307 except ImportError:
6209
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
308 if db_type == 'gdbm':
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
309 try:
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
310 dbm = __import__('dbm.gnu')
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
311 except ImportError:
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
312 raise hyperdb.DatabaseError(_(
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
313 "Couldn't open database - the required module '%s' "
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
314 "(as dbm.gnu) is not available") % db_type)
6209
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
315 else:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
316 raise hyperdb.DatabaseError(_(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
317 "Couldn't open database - the "
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
318 "required module '%s' is not available") % db_type)
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
319 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
320 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
321 "opendb %r.open(%r, %r)" % (db_type, path, mode))
443
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
322 return dbm.open(path, mode)
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
323
690
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
324 #
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
325 # Node IDs
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
326 #
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
327 def newid(self, classname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
328 """ Generate a new id for the given class
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
329 """
690
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
330 # open the ids DB - create if if doesn't exist
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
331 db = self.opendb('_ids', 'c')
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
332 if classname in db:
690
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
333 newid = db[classname] = str(int(db[classname]) + 1)
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
334 else:
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
335 # the count() bit is transitional - older dbs won't start at 1
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
336 newid = str(self.getclass(classname).count()+1)
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
337 db[classname] = newid
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
338 db.close()
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
339 return newid
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
340
952
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
341 def setid(self, classname, setid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
342 """ Set the id counter: used during import of database
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
343 """
952
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
344 # open the ids DB - create if if doesn't exist
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
345 db = self.opendb('_ids', 'c')
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
346 db[classname] = str(setid)
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
347 db.close()
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
348
48
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
349 #
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
350 # Nodes
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
351 #
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
352 def addnode(self, classname, nodeid, node):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
353 """ add the specified node to its class's db
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
354 """
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
355 # we'll be supplied these props if we're doing an import
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
356 if 'creator' not in node:
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
357 # add in the "calculated" properties (dupe so we don't affect
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
358 # calling code's node assumptions)
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
359 node = node.copy()
1800
a3b1b1dcf639 Use getuid(), not figure_curuserid()
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 1794
diff changeset
360 node['creator'] = self.getuid()
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
361 node['actor'] = self.getuid()
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
362 node['creation'] = node['activity'] = date.Date()
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
363
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
364 self.newnodes.setdefault(classname, {})[nodeid] = 1
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
365 self.cache.setdefault(classname, {})[nodeid] = node
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
366 self.savenode(classname, nodeid, node)
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
367
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
368 def setnode(self, classname, nodeid, node):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
369 """ change the specified node
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
370 """
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
371 self.dirtynodes.setdefault(classname, {})[nodeid] = 1
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
372
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
373 # can't set without having already loaded the node
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
374 self.cache[classname][nodeid] = node
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
375 self.savenode(classname, nodeid, node)
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
376
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
377 def savenode(self, classname, nodeid, node):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
378 """ perform the saving of data specified by the set/addnode
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
379 """
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
380 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
381 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
382 'save %s%s %r' % (classname, nodeid, node))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
383 self.transactions.append((self.doSaveNode, (classname, nodeid, node)))
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
384
8305
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
385 def getnode(self, classname, nodeid, db=None, cache=1, allow_abort=True):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
386 """ get a node from the database
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
387
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
388 Note the "cache" parameter is not used, and exists purely for
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
389 backward compatibility!
8305
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
390
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
391 'allow_abort' is used only in sql backends.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
392 """
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
393 # try the cache
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
394 cache_dict = self.cache.setdefault(classname, {})
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
395 if nodeid in cache_dict:
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
396 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
397 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
398 'get %s%s cached' % (classname, nodeid))
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
399 self.stats['cache_hits'] += 1
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
400 return cache_dict[nodeid]
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
401
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
402 if __debug__:
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
403 self.stats['cache_misses'] += 1
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
404 start_t = time.time()
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
405 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
406 'get %s%s' % (classname, nodeid))
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
407
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
408 # get from the database and save in the cache
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
409 if db is None:
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
410 db = self.getclassdb(classname)
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
411 if nodeid not in db:
8130
9056b0c8fcd6 Fix file descriptor leak in anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents: 7936
diff changeset
412 db.close()
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
413 raise IndexError("no such %s %s" % (classname, nodeid))
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
414
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
415 # check the uncommitted, destroyed nodes
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
416 if (classname in self.destroyednodes and
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
417 nodeid in self.destroyednodes[classname]):
8130
9056b0c8fcd6 Fix file descriptor leak in anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents: 7936
diff changeset
418 db.close()
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
419 raise IndexError("no such %s %s" % (classname, nodeid))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
420
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
421 # decode
48
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
422 res = marshal.loads(db[nodeid])
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
423
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
424 # reverse the serialisation
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
425 res = self.unserialise(classname, res)
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
426
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
427 # store off in the cache dict
475
a1a44636bace Fix breakage caused by transaction changes.
Richard Jones <richard@users.sourceforge.net>
parents: 464
diff changeset
428 if cache:
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
429 cache_dict[nodeid] = res
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
430
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
431 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
432 self.stats['get_items'] += (time.time() - start_t)
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
433
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
434 return res
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
435
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
436 def destroynode(self, classname, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
437 """Remove a node from the database. Called exclusively by the
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
438 destroy() method on Class.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
439 """
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
440 logging.getLogger('roundup.hyperdb.backend').info(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
441 'destroy %s%s' % (classname, nodeid))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
442
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
443 # remove from cache and newnodes if it's there
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
444 if (classname in self.cache and nodeid in self.cache[classname]):
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
445 del self.cache[classname][nodeid]
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
446 if (classname in self.newnodes and nodeid in self.newnodes[classname]):
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
447 del self.newnodes[classname][nodeid]
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
448
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
449 # see if there's any obvious commit actions that we should get rid of
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
450 for entry in self.transactions[:]:
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
451 if entry[1][:2] == (classname, nodeid):
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
452 self.transactions.remove(entry)
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
453
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
454 # add to the destroyednodes map
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
455 self.destroyednodes.setdefault(classname, {})[nodeid] = 1
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
456
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
457 # add the destroy commit action
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
458 self.transactions.append((self.doDestroyNode, (classname, nodeid)))
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
459 self.transactions.append((FileStorage.destroy, (self, classname, nodeid)))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
460
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
461 def serialise(self, classname, node):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
462 """Copy the node contents, converting non-marshallable data into
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
463 marshallable data.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
464 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
465 properties = self.getclass(classname).getprops()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
466 d = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
467 for k, v in node.items():
2964
b71f6d0a463d merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2908
diff changeset
468 if k == self.RETIRED_FLAG:
b71f6d0a463d merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2908
diff changeset
469 d[k] = v
b71f6d0a463d merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2908
diff changeset
470 continue
b71f6d0a463d merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2908
diff changeset
471
b71f6d0a463d merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2908
diff changeset
472 # if the property doesn't exist then we really don't care
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
473 if k not in properties:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
474 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
475
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
476 # get the property spec
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
477 prop = properties[k]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
478
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
479 if isinstance(prop, hyperdb.Password) and v is not None:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
480 d[k] = str(v)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
481 elif isinstance(prop, hyperdb.Date) and v is not None:
964
832d1209aaa2 Preparing to turn back on link/unlink journal events.
Richard Jones <richard@users.sourceforge.net>
parents: 952
diff changeset
482 d[k] = v.serialise()
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
483 elif isinstance(prop, hyperdb.Interval) and v is not None:
964
832d1209aaa2 Preparing to turn back on link/unlink journal events.
Richard Jones <richard@users.sourceforge.net>
parents: 952
diff changeset
484 d[k] = v.serialise()
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
485 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
486 d[k] = v
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
487 return d
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
488
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
489 def unserialise(self, classname, node):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
490 """Decode the marshalled node data
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
491 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
492 properties = self.getclass(classname).getprops()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
493 d = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
494 for k, v in node.items():
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
495 # if the property doesn't exist, or is the "retired" flag then
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
496 # it won't be in the properties dict
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
497 if k not in properties:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
498 d[k] = v
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
499 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
500
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
501 # get the property spec
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
502 prop = properties[k]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
503
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
504 if isinstance(prop, hyperdb.Date) and v is not None:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
505 d[k] = date.Date(v)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
506 elif isinstance(prop, hyperdb.Interval) and v is not None:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
507 d[k] = date.Interval(v)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
508 elif isinstance(prop, hyperdb.Password) and v is not None:
7211
506c86823abb Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents: 7038
diff changeset
509 d[k] = password.Password(encrypted=v, config=self.config)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
510 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
511 d[k] = v
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
512 return d
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
513
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
514 def hasnode(self, classname, nodeid, db=None):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
515 """ determine if the database has a given node
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
516 """
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
517 # try the cache
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
518 cache = self.cache.setdefault(classname, {})
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
519 if nodeid in cache:
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
520 return 1
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
521
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
522 # not in the cache - check the database
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
523 if db is None:
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
524 db = self.getclassdb(classname)
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
525 return nodeid in db
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
526
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
527 def countnodes(self, classname, db=None):
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
528 count = 0
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
529
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
530 # include the uncommitted nodes
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
531 if classname in self.newnodes:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
532 count += len(self.newnodes[classname])
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
533 if classname in self.destroyednodes:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
534 count -= len(self.destroyednodes[classname])
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
535
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
536 # and count those in the DB
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
537 if db is None:
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
538 db = self.getclassdb(classname)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
539 return count + len(db)
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
540
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
541 #
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
542 # Files - special node properties
646
07abfe8f0c01 use blobfiles in back_anydbm which is used in back_bsddb.
Engelbert Gruber <grubert@users.sourceforge.net>
parents: 624
diff changeset
543 # inherited from FileStorage
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
544
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
545 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
546 # Journal
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
547 #
1143
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
548 def addjournal(self, classname, nodeid, action, params, creator=None,
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
549 creation=None):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
550 """ Journal the Action
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
551 'action' may be:
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
552
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
553 'set' -- 'params' is a dictionary of property values
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
554 'create' -- 'params' is an empty dictionary as of
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
555 Wed Nov 06 11:38:43 2002 +0000
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
556 'link' or 'unlink' -- 'params' is (classname, nodeid, propname)
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
557 'retired' or 'restored' -- 'params' is None
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
558
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
559 'creator' -- the user performing the action, which defaults to
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
560 the current user.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
561 """
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
562 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
563 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
564 'addjournal %s%s %s %r %s %r' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
565 classname, nodeid, action, params, creator, creation))
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
566 if creator is None:
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
567 creator = self.getuid()
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
568 self.transactions.append((self.doSaveJournal, (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
569 classname, nodeid, action, params, creator, creation)))
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
570
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
571 def setjournal(self, classname, nodeid, journal):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
572 """Set the journal to the "journal" list."""
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
573 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
574 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
575 'setjournal %s%s %r' % (classname, nodeid, journal))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
576 self.transactions.append((self.doSetJournal, (classname, nodeid,
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
577 journal)))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
578
4534
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
579 def fix_journal(self, classname, journal):
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
580 """ fix password entries to correct type """
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
581 pwprops = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
582 for pn, prop in self.getclass(classname).properties.items():
4534
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
583 if isinstance(prop, hyperdb.Password):
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
584 pwprops[pn] = 1
4534
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
585 if not pwprops:
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
586 return journal
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
587 for j in journal:
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
588 if j[3] == 'set':
7038
f524ddc27af8 replace for a,b in x.items() with for a in x.keys()
John Rouillard <rouilj@ieee.org>
parents: 7012
diff changeset
589 for k in j[4].keys():
4538
fd972e18b21a - fix import/export regression test for anydbm for latest journal fix
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4534
diff changeset
590 if k in pwprops and j[4][k]:
4534
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
591 j[4][k] = password.JournalPassword(j[4][k])
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
592 return journal
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
593
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
594 def getjournal(self, classname, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
595 """ get the journal for id
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
596
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
597 Raise IndexError if the node doesn't exist (as per history()'s
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
598 API)
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
599 """
1567
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
600 # our journal result
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
601 res = []
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
602
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
603 # add any journal entries for transactions not committed to the
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
604 # database
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
605 for method, args in self.transactions:
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
606 if method != self.doSaveJournal:
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
607 continue
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
608 (cache_classname, cache_nodeid, cache_action, cache_params,
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
609 cache_creator, cache_creation) = args
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
610 if cache_classname == classname and cache_nodeid == nodeid:
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
611 if not cache_creator:
1800
a3b1b1dcf639 Use getuid(), not figure_curuserid()
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 1794
diff changeset
612 cache_creator = self.getuid()
1567
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
613 if not cache_creation:
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
614 cache_creation = date.Date()
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
615 res.append((cache_nodeid, cache_creation, cache_creator,
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
616 cache_action, cache_params))
1567
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
617
48
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
618 # attempt to open the journal - in some rare cases, the journal may
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
619 # not exist
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
620 try:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
621 db = self.opendb('journals.%s' % classname, 'r')
5248
198b6e810c67 Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents: 5232
diff changeset
622 except anydbm.error as error:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
623 if str(error) == "need 'c' or 'n' flag to open new db":
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
624 raise IndexError('no such %s %s' % (classname, nodeid))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
625 elif error.args[0] != 2:
1920
f9316d2cd5ba Fixed retirement of items in rdbms imports [SF#841355]
Richard Jones <richard@users.sourceforge.net>
parents: 1904
diff changeset
626 # this isn't a "not found" error, be alarmed!
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
627 raise
1920
f9316d2cd5ba Fixed retirement of items in rdbms imports [SF#841355]
Richard Jones <richard@users.sourceforge.net>
parents: 1904
diff changeset
628 if res:
f9316d2cd5ba Fixed retirement of items in rdbms imports [SF#841355]
Richard Jones <richard@users.sourceforge.net>
parents: 1904
diff changeset
629 # we have unsaved journal entries, return them
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
630 return self.fix_journal(classname, res)
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
631 raise IndexError('no such %s %s' % (classname, nodeid))
787
b6b0a92e0738 More informative error message
Richard Jones <richard@users.sourceforge.net>
parents: 778
diff changeset
632 try:
b6b0a92e0738 More informative error message
Richard Jones <richard@users.sourceforge.net>
parents: 778
diff changeset
633 journal = marshal.loads(db[nodeid])
b6b0a92e0738 More informative error message
Richard Jones <richard@users.sourceforge.net>
parents: 778
diff changeset
634 except KeyError:
843
01f9d02faea1 ...except of course it's nice to use valid Python syntax
Richard Jones <richard@users.sourceforge.net>
parents: 842
diff changeset
635 db.close()
1567
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
636 if res:
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
637 # we have some unsaved journal entries, be happy!
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
638 return self.fix_journal(classname, res)
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
639 raise IndexError('no such %s %s' % (classname, nodeid))
843
01f9d02faea1 ...except of course it's nice to use valid Python syntax
Richard Jones <richard@users.sourceforge.net>
parents: 842
diff changeset
640 db.close()
1567
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
641
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
642 # add all the saved journal entries for this node
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
643 for nodeid, date_stamp, user, action, params in journal:
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
644 res.append((nodeid, date.Date(date_stamp), user, action, params))
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
645 return self.fix_journal(classname, res)
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
646
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
647 def pack(self, pack_before):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
648 """ Delete all journal entries except "create" before 'pack_before'.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
649 """
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
650 pack_before = pack_before.serialise()
990
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
651 for classname in self.getclasses():
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
652 packed = 0
990
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
653 # get the journal db
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
654 db_name = 'journals.%s' % classname
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
655 path = os.path.join(os.getcwd(), self.dir, classname)
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
656 db_type = self.determine_db_type(path)
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
657 db = self.opendb(db_name, 'w')
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
658
5444
167f0d25ea8e Python 3 preparation: convert dbm keys back from bytes to strings.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5395
diff changeset
659 for key in map(b2s, db.keys()):
990
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
660 # get the journal for this db entry
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
661 journal = marshal.loads(db[key])
7010
1bb7aa8414b8 change variable name l -> kept_entries
John Rouillard <rouilj@ieee.org>
parents: 7009
diff changeset
662 kept_entries = []
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
663 for entry in journal:
990
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
664 # unpack the entry
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
665 (nodeid, date_stamp, self.journaltag, action,
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
666 params) = entry
990
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
667 # if the entry is after the pack date, _or_ the initial
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
668 # create entry, then it stays
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
669 if date_stamp > pack_before or action == 'create':
7010
1bb7aa8414b8 change variable name l -> kept_entries
John Rouillard <rouilj@ieee.org>
parents: 7009
diff changeset
670 kept_entries.append(entry)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
671 else:
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
672 packed += 1
7010
1bb7aa8414b8 change variable name l -> kept_entries
John Rouillard <rouilj@ieee.org>
parents: 7009
diff changeset
673 db[key] = marshal.dumps(kept_entries)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
674
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
675 logging.getLogger('roundup.hyperdb.backend').info(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
676 'packed %d %s items' % (packed, classname))
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
677
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
678 if db_type == 'gdbm':
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
679 db.reorganize()
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
680 db.close()
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
681
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
682 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
683 # Basic transaction support
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
684 #
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
685 def commit(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
686 """ Commit the current transactions.
3687
ff9f4ca42454 Postgres backend allows transaction collisions to be ignored when...
Richard Jones <richard@users.sourceforge.net>
parents: 3682
diff changeset
687
ff9f4ca42454 Postgres backend allows transaction collisions to be ignored when...
Richard Jones <richard@users.sourceforge.net>
parents: 3682
diff changeset
688 Save all data changed since the database was opened or since the
ff9f4ca42454 Postgres backend allows transaction collisions to be ignored when...
Richard Jones <richard@users.sourceforge.net>
parents: 3682
diff changeset
689 last commit() or rollback().
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
690 """
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
691 logging.getLogger('roundup.hyperdb.backend').info(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
692 'commit %s transactions' % (len(self.transactions)))
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
693
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
694 # keep a handle to all the database files opened
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
695 self.databases = {}
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
696
1904
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
697 try:
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
698 # now, do all the transactions
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
699 reindex = {}
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
700 for method, args in self.transactions:
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
701 reindex[method(*args)] = 1
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
702 finally:
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
703 # make sure we close all the database files
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
704 for db in self.databases.values():
1904
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
705 db.close()
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
706 del self.databases
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
707
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3955
diff changeset
708 # clear the transactions list now so the blobfile implementation
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3955
diff changeset
709 # doesn't think there's still pending file commits when it tries
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3955
diff changeset
710 # to access the file data
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3955
diff changeset
711 self.transactions = []
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3955
diff changeset
712
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
713 # reindex the nodes that request it
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
714 for classname, nodeid in [k for k in reindex if k]:
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
715 self.getclass(classname).index(nodeid)
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
716
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
717 # save the indexer state
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
718 self.indexer.save_index()
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
719
1181
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
720 self.clearCache()
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
721
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
722 def clearCache(self):
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
723 # all transactions committed, back to normal
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
724 self.cache = {}
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
725 self.dirtynodes = {}
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
726 self.newnodes = {}
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
727 self.destroyednodes = {}
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
728 self.transactions = []
4652
dfbc0cfa9811 Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4538
diff changeset
729 # upcall is necessary!
dfbc0cfa9811 Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4538
diff changeset
730 roundupdb.Database.clearCache(self)
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
731
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
732 def getCachedClassDB(self, classname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
733 """ get the class db, looking in our cache of databases for commit
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
734 """
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
735 # get the database handle
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
736 db_name = 'nodes.%s' % classname
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
737 if db_name not in self.databases:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
738 self.databases[db_name] = self.getclassdb(classname, 'c')
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
739 return self.databases[db_name]
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
740
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
741 def doSaveNode(self, classname, nodeid, node):
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
742 db = self.getCachedClassDB(classname)
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
743
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
744 # now save the marshalled data
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
745 db[nodeid] = marshal.dumps(self.serialise(classname, node))
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
746
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
747 # return the classname, nodeid so we reindex this content
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
748 return (classname, nodeid)
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
749
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
750 def getCachedJournalDB(self, classname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
751 """ get the journal db, looking in our cache of databases for commit
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
752 """
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
753 # get the database handle
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
754 db_name = 'journals.%s' % classname
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
755 if db_name not in self.databases:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
756 self.databases[db_name] = self.opendb(db_name, 'c')
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
757 return self.databases[db_name]
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
758
1143
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
759 def doSaveJournal(self, classname, nodeid, action, params, creator,
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
760 creation):
1143
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
761 # serialise the parameters now if necessary
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
762 if isinstance(params, type({})):
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
763 if action in ('set', 'create'):
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
764 params = self.serialise(classname, params)
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
765
952
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
766 # handle supply of the special journalling parameters (usually
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
767 # supplied on importing an existing database)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
768 journaltag = creator
1143
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
769 if creation:
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
770 journaldate = creation.serialise()
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
771 else:
964
832d1209aaa2 Preparing to turn back on link/unlink journal events.
Richard Jones <richard@users.sourceforge.net>
parents: 952
diff changeset
772 journaldate = date.Date().serialise()
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
773
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
774 # create the journal entry
952
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
775 entry = (nodeid, journaldate, journaltag, action, params)
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
776
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
777 db = self.getCachedJournalDB(classname)
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
778
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
779 # now insert the journal entry
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
780 if nodeid in db:
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
781 # append to existing
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
782 s = db[nodeid]
650
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 646
diff changeset
783 l = marshal.loads(s)
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
784 l.append(entry)
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
785 else:
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
786 l = [entry]
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
787
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
788 db[nodeid] = marshal.dumps(l)
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
789
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
790 def doSetJournal(self, classname, nodeid, journal):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
791 l = []
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
792 for nodeid, journaldate, journaltag, action, params in journal:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
793 # serialise the parameters now if necessary
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
794 if isinstance(params, type({})):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
795 if action in ('set', 'create'):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
796 params = self.serialise(classname, params)
2512
f5542d92307a fix anydbm journal import [SF#983166]
Richard Jones <richard@users.sourceforge.net>
parents: 2505
diff changeset
797 journaldate = journaldate.serialise()
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
798 l.append((nodeid, journaldate, journaltag, action, params))
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
799 db = self.getCachedJournalDB(classname)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
800 db[nodeid] = marshal.dumps(l)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
801
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
802 def doDestroyNode(self, classname, nodeid):
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
803 # delete from the class database
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
804 db = self.getCachedClassDB(classname)
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
805 if nodeid in db:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
806 del db[nodeid]
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
807
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
808 # delete from the database
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
809 db = self.getCachedJournalDB(classname)
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
810 if nodeid in db:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
811 del db[nodeid]
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
812
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
813 def rollback(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
814 """ Reverse all actions from the current transaction.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
815 """
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
816 logging.getLogger('roundup.hyperdb.backend').info(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
817 'rollback %s transactions' % (len(self.transactions)))
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
818
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
819 for method, args in self.transactions:
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
820 # delete temporary files
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
821 if method == self.doStoreFile:
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
822 self.rollbackStoreFile(*args)
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
823 self.cache = {}
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
824 self.dirtynodes = {}
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
825 self.newnodes = {}
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
826 self.destroyednodes = {}
252
76c6994aa4e8 CGI interfaces now spit up a top-level index of all instances they can serve.
Richard Jones <richard@users.sourceforge.net>
parents: 224
diff changeset
827 self.transactions = []
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
828
1131
92e92ae58494 add close() methods where they are missing!
Richard Jones <richard@users.sourceforge.net>
parents: 1127
diff changeset
829 def close(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
830 """ Nothing to do
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
831 """
1333
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
832 if self.lockfile is not None:
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
833 locking.release_lock(self.lockfile)
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
834 self.lockfile.close()
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
835 self.lockfile = None
1131
92e92ae58494 add close() methods where they are missing!
Richard Jones <richard@users.sourceforge.net>
parents: 1127
diff changeset
836
7002
1c56bd744be4 More whiespace fixes
John Rouillard <rouilj@ieee.org>
parents: 7001
diff changeset
837
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
838 class Class(hyperdb.Class):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
839 """The handle to a particular class of nodes in a hyperdatabase."""
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
840
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
841 def enableJournalling(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
842 """Turn journalling on for this class
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
843 """
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
844 self.do_journal = 1
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
845
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
846 def disableJournalling(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
847 """Turn journalling off for this class
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
848 """
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
849 self.do_journal = 0
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
850
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
851 # Editing nodes:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
852
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
853 def create(self, **propvalues):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
854 """Create a new node of this class and return its id.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
855
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
856 The keyword arguments in 'propvalues' map property names to values.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
857
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
858 The values of arguments must be acceptable for the types of their
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
859 corresponding properties or a TypeError is raised.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
860
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
861 If this class has a key property, it must be present and its value
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
862 must not collide with other key strings or a ValueError is raised.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
863
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
864 Any other properties on this class that are missing from the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
865 'propvalues' dictionary are set to None.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
866
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
867 If an id in a link or multilink property does not refer to a valid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
868 node, an IndexError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
869
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
870 These operations trigger detectors and can be vetoed. Attempts
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
871 to modify the "creation" or "activity" properties cause a KeyError.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
872 """
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4342
diff changeset
873 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
874 raise hyperdb.DatabaseError(_('Database open read-only'))
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
875 self.fireAuditors('create', None, propvalues)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
876 newid = self.create_inner(**propvalues)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
877 self.fireReactors('create', newid, None)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
878 return newid
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
879
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
880 def create_inner(self, **propvalues):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
881 """ Called by create, in-between the audit and react calls.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
882 """
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
883 if 'id' in propvalues:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
884 raise KeyError('"id" is reserved')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
885
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
886 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
887 raise hyperdb.DatabaseError(_('Database open read-only'))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
888
7002
1c56bd744be4 More whiespace fixes
John Rouillard <rouilj@ieee.org>
parents: 7001
diff changeset
889 if ('creator' in propvalues or 'actor' in propvalues or
1c56bd744be4 More whiespace fixes
John Rouillard <rouilj@ieee.org>
parents: 7001
diff changeset
890 'creation' in propvalues or 'activity' in propvalues):
5704
aa13a517cc63 Anydbm back end didn't list creator or actor as protected properties.
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
891 raise KeyError('"creator", "actor", "creation" and '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
892 '"activity" are reserved')
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
893
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
894 for p in propvalues:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
895 prop = self.properties[p]
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
896 if prop.computed:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
897 raise KeyError('"%s" is a computed property' % p)
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
898
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
899 # new node's id
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
900 newid = self.db.newid(self.classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
901
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
902 # validate propvalues
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
903 num_re = re.compile(r'^\d+$')
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
904 for key, value in propvalues.items():
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
905 if key == self.key:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
906 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
907 self.lookup(value)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
908 except KeyError:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
909 pass
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
910 else:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
911 raise ValueError('node with key "%s" exists' % value)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
912
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
913 # try to handle this property
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
914 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
915 prop = self.properties[key]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
916 except KeyError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
917 raise KeyError('"%s" has no property "%s"' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
918 self.classname, key))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
919
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
920 if value is not None and isinstance(prop, hyperdb.Link):
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
921 if not isinstance(value, str):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
922 raise ValueError('link value must be String')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
923 link_class = self.properties[key].classname
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
924 # if it isn't a number, it's a key
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
925 if not num_re.match(value):
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
926 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
927 value = self.db.classes[link_class].lookup(value)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
928 except (TypeError, KeyError):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
929 raise IndexError('new property "%s": %s not a %s' % (
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
930 key, value, link_class))
902
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
931 elif not self.db.getclass(link_class).hasnode(value):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
932 raise IndexError('%s has no node %s' % (link_class, value))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
933
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
934 # save off the value
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
935 propvalues[key] = value
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
936
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
937 # register the link with the newly linked node
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
938 if self.do_journal and self.properties[key].do_journal:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
939 self.db.addjournal(link_class, value, 'link',
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
940 (self.classname, newid, key))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
941
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
942 elif isinstance(prop, hyperdb.Multilink):
3872
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
943 if value is None:
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
944 value = []
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
945 if not hasattr(value, '__iter__') or isinstance(value, str):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
946 raise TypeError(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
947 'new property "%s" not an iterable of ids' % key)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
948
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
949 # clean up and validate the list of links
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
950 link_class = self.properties[key].classname
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
951 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
952 for entry in value:
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
953 if not isinstance(entry, str):
7004
d3d099432ba2 Remove uneeded \
John Rouillard <rouilj@ieee.org>
parents: 7003
diff changeset
954 raise ValueError('"%s" multilink value (%r) '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
955 'must contain Strings' % (key, value))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
956 # if it isn't a number, it's a key
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
957 if not num_re.match(entry):
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
958 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
959 entry = self.db.classes[link_class].lookup(entry)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
960 except (TypeError, KeyError):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
961 raise IndexError(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
962 'new property "%s": %s not a %s' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
963 key, entry,
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
964 self.properties[key].classname))
6999
e5a28b8ded72 backout checkin 60b37f632601
John Rouillard <rouilj@ieee.org>
parents: 6998
diff changeset
965 l.append(entry)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
966 value = l
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
967 propvalues[key] = value
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
968
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
969 # handle additions
902
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
970 for nodeid in value:
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
971 if not self.db.getclass(link_class).hasnode(nodeid):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
972 raise IndexError('%s has no node %s' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
973 link_class, nodeid))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
974 # register the link with the newly linked node
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
975 if self.do_journal and self.properties[key].do_journal:
902
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
976 self.db.addjournal(link_class, nodeid, 'link',
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
977 (self.classname, newid, key))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
978
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
979 elif isinstance(prop, hyperdb.String):
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
980 if not isinstance(value, (str, unicode)):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
981 raise TypeError('new property "%s" not a string' % key)
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2736
diff changeset
982 if prop.indexme:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
983 self.db.indexer.add_text(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
984 (self.classname, newid, key), value)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
985
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
986 elif isinstance(prop, hyperdb.Password):
8534
1f8492d68aca bug: using 'null' value for attributes causes error.
John Rouillard <rouilj@ieee.org>
parents: 8403
diff changeset
987 if value is not None and not isinstance(value, password.Password):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
988 raise TypeError('new property "%s" not a Password' % key)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
989
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
990 elif isinstance(prop, hyperdb.Date):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
991 if value is not None and not isinstance(value, date.Date):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
992 raise TypeError('new property "%s" not a Date' % key)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
993
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
994 elif isinstance(prop, hyperdb.Interval):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
995 if value is not None and not isinstance(value, date.Interval):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
996 raise TypeError('new property "%s" not an Interval' % key)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
997
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
998 elif value is not None and isinstance(prop, hyperdb.Number):
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
999 try:
890
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1000 float(value)
887
e7169d6e6e45 added tests for number type too
Richard Jones <richard@users.sourceforge.net>
parents: 886
diff changeset
1001 except ValueError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1002 raise TypeError('new property "%s" not numeric' % key)
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
1003
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1004 elif value is not None and isinstance(prop, hyperdb.Integer):
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1005 try:
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1006 int(value)
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1007 except ValueError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1008 raise TypeError('new property "%s" not an integer' % key)
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1009
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1010 elif value is not None and isinstance(prop, hyperdb.Boolean):
890
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1011 try:
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1012 int(value)
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1013 except ValueError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1014 raise TypeError('new property "%s" not boolean' % key)
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
1015
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1016 # make sure there's data where there needs to be
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
1017 for key, prop in self.properties.items():
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1018 if key in propvalues:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1019 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1020 if key == self.key:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1021 raise ValueError('key property "%s" is required' % key)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1022 if isinstance(prop, hyperdb.Multilink):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1023 propvalues[key] = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1024
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1025 # done
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1026 self.db.addnode(self.classname, newid, propvalues)
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1027 if self.do_journal:
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1303
diff changeset
1028 self.db.addjournal(self.classname, newid, 'create', {})
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1029
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1030 return newid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1031
8305
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
1032 def get(self, nodeid, propname, default=_marker, cache=1, allow_abort=True):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1033 """Get the value of a property on an existing node of this class.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1034
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1035 'nodeid' must be the id of an existing node of this class or an
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1036 IndexError is raised. 'propname' must be the name of a property
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1037 of this class or a KeyError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1038
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
1039 'cache' exists for backward compatibility, and is not used.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1040
8305
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
1041 'allow_abort' is used only in sql backends.
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
1042
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1043 Attempts to get the "creation" or "activity" properties should
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1044 do the right thing.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1045 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1046 if propname == 'id':
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1047 return nodeid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1048
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1049 # get the node's dict
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
1050 d = self.db.getnode(self.classname, nodeid)
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1051
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1052 # check for one of the special props
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1053 if propname == 'creation':
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1054 if 'creation' in d:
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1055 return d['creation']
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1056 if not self.do_journal:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1057 raise ValueError('Journalling is disabled for this class')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1058 journal = self.db.getjournal(self.classname, nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1059 if journal:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents: 4075
diff changeset
1060 return journal[0][1]
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1061 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1062 # on the strange chance that there's no journal
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1063 return date.Date()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1064 if propname == 'activity':
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1065 if 'activity' in d:
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1066 return d['activity']
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1067 if not self.do_journal:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1068 raise ValueError('Journalling is disabled for this class')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1069 journal = self.db.getjournal(self.classname, nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1070 if journal:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1071 return self.db.getjournal(self.classname, nodeid)[-1][1]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1072 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1073 # on the strange chance that there's no journal
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1074 return date.Date()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1075 if propname == 'creator':
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1076 if 'creator' in d:
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1077 return d['creator']
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1078 if not self.do_journal:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1079 raise ValueError('Journalling is disabled for this class')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1080 journal = self.db.getjournal(self.classname, nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1081 if journal:
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
1082 num_re = re.compile(r'^\d+$')
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1083 value = journal[0][2]
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1084 if num_re.match(value):
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1085 return value
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1086 else:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1087 # old-style "username" journal tag
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1088 try:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1089 return self.db.user.lookup(value)
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1090 except KeyError:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1091 # user's been retired, return admin
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1092 return '1'
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1093 else:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1094 return self.db.getuid()
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1095 if propname == 'actor':
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1096 if 'actor' in d:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1097 return d['actor']
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1098 if not self.do_journal:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1099 raise ValueError('Journalling is disabled for this class')
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1100 journal = self.db.getjournal(self.classname, nodeid)
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1101 if journal:
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
1102 num_re = re.compile(r'^\d+$')
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1103 value = journal[-1][2]
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1104 if num_re.match(value):
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1105 return value
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1106 else:
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1107 # old-style "username" journal tag
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1108 try:
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1109 return self.db.user.lookup(value)
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1110 except KeyError:
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1111 # user's been retired, return admin
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1112 return '1'
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1113 else:
1800
a3b1b1dcf639 Use getuid(), not figure_curuserid()
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 1794
diff changeset
1114 return self.db.getuid()
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1115
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1116 # get the property (raises KeyErorr if invalid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1117 prop = self.properties[propname]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1118
6179
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
1119 if isinstance(prop, hyperdb.Multilink) and prop.computed:
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
1120 cls = self.db.getclass(prop.rev_classname)
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
1121 ids = cls.find(**{prop.rev_propname: nodeid})
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
1122 return ids
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
1123
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1124 if propname not in d:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1125 if default is _marker:
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1126 if isinstance(prop, hyperdb.Multilink):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1127 return []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1128 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1129 return None
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1130 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1131 return default
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1132
1014
8816534e6a1a Fixed nasty bug that was preventing changes to multilinks going through.
Richard Jones <richard@users.sourceforge.net>
parents: 1002
diff changeset
1133 # return a dupe of the list so code doesn't get confused
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1134 if isinstance(prop, hyperdb.Multilink):
6351
0db59cc2cd37 Enable testMultilinkOrdering check. Fix back_anydbm to pass.
John Rouillard <rouilj@ieee.org>
parents: 6238
diff changeset
1135 ids = d[propname][:]
0db59cc2cd37 Enable testMultilinkOrdering check. Fix back_anydbm to pass.
John Rouillard <rouilj@ieee.org>
parents: 6238
diff changeset
1136 ids.sort(key=lambda x: int(x))
0db59cc2cd37 Enable testMultilinkOrdering check. Fix back_anydbm to pass.
John Rouillard <rouilj@ieee.org>
parents: 6238
diff changeset
1137 return ids
1014
8816534e6a1a Fixed nasty bug that was preventing changes to multilinks going through.
Richard Jones <richard@users.sourceforge.net>
parents: 1002
diff changeset
1138
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1139 return d[propname]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1140
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1141 def set(self, nodeid, **propvalues):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1142 """Modify a property on an existing node of this class.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1143
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1144 'nodeid' must be the id of an existing node of this class or an
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1145 IndexError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1146
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1147 Each key in 'propvalues' must be the name of a property of this
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1148 class or a KeyError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1149
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1150 All values in 'propvalues' must be acceptable types for their
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1151 corresponding properties or a TypeError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1152
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1153 If the value of the key property is set, it must not collide with
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1154 other key strings or a ValueError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1155
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1156 If the value of a Link or Multilink property contains an invalid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1157 node id, a ValueError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1158
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1159 These operations trigger detectors and can be vetoed. Attempts
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1160 to modify the "creation" or "activity" properties cause a KeyError.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1161 """
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4342
diff changeset
1162 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1163 raise hyperdb.DatabaseError(_('Database open read-only'))
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4342
diff changeset
1164
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1165 self.fireAuditors('set', nodeid, propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1166 oldvalues = copy.deepcopy(self.db.getnode(self.classname, nodeid))
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
1167 for name, prop in self.getprops(protected=0).items():
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1168 if name in oldvalues:
2608
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
1169 continue
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1170 if isinstance(prop, hyperdb.Multilink):
2608
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
1171 oldvalues[name] = []
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
1172 else:
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
1173 oldvalues[name] = None
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1174 propvalues = self.set_inner(nodeid, **propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1175 self.fireReactors('set', nodeid, oldvalues)
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1176 return propvalues
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1177
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1178 def set_inner(self, nodeid, **propvalues):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1179 """ Called by set, in-between the audit and react calls.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1180 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1181 if not propvalues:
930
3c344e942055 Use same regex to split search terms as used to index text.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 927
diff changeset
1182 return propvalues
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1183
7002
1c56bd744be4 More whiespace fixes
John Rouillard <rouilj@ieee.org>
parents: 7001
diff changeset
1184 if ('creator' in propvalues or 'actor' in propvalues or
1c56bd744be4 More whiespace fixes
John Rouillard <rouilj@ieee.org>
parents: 7001
diff changeset
1185 'creation' in propvalues or 'activity' in propvalues):
5704
aa13a517cc63 Anydbm back end didn't list creator or actor as protected properties.
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
1186 raise KeyError('"creator", "actor", "creation" and '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1187 '"activity" are reserved')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1188
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1189 if 'id' in propvalues:
5378
35ea9b1efc14 Python 3 preparation: "raise" syntax.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5319
diff changeset
1190 raise KeyError('"id" is reserved')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1191
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1192 for p in propvalues:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1193 prop = self.properties[p]
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1194 if prop.computed:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1195 raise KeyError('"%s" is a computed property' % p)
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1196
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1197 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1198 raise hyperdb.DatabaseError(_('Database open read-only'))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1199
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1200 node = self.db.getnode(self.classname, nodeid)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1201 if self.db.RETIRED_FLAG in node:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1202 raise IndexError
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
1203 num_re = re.compile(r'^\d+$')
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1204
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1205 # if the journal value is to be different, store it in here
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1206 journalvalues = {}
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1207
5112
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1208 # omit quiet properties from history/changelog
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1209 quiet_props = []
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1210
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1211 # list() propvalues 'cos it might be modified by the loop
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1212 for propname, value in list(propvalues.items()):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1213 # check to make sure we're not duplicating an existing key
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1214 if propname == self.key and node[propname] != value:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1215 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1216 self.lookup(value)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1217 except KeyError:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1218 pass
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1219 else:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1220 raise ValueError('node with key "%s" exists' % value)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1221
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1222 # this will raise the KeyError if the property isn't valid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1223 # ... we don't use getprops() here because we only care about
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1224 # the writeable properties.
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1225 try:
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1226 prop = self.properties[propname]
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1227 except KeyError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1228 raise KeyError('"%s" has no property named "%s"' % (
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1229 self.classname, propname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1230
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1231 # if the value's the same as the existing value, no sense in
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1232 # doing anything
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1303
diff changeset
1233 current = node.get(propname, None)
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1303
diff changeset
1234 if value == current:
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1235 del propvalues[propname]
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1236 continue
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1303
diff changeset
1237 journalvalues[propname] = current
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1238
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1239 # do stuff based on the prop type
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1240 if isinstance(prop, hyperdb.Link):
927
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1241 link_class = prop.classname
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1242 # if it isn't a number, it's a key
927
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1243 if value is not None and not isinstance(value, type('')):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1244 raise ValueError('property "%s" link value be a string' % (
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1245 propname))
927
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1246 if isinstance(value, type('')) and not num_re.match(value):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1247 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1248 value = self.db.classes[link_class].lookup(value)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1249 except (TypeError, KeyError):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1250 raise IndexError('new property "%s": %s not a %s' % (
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1251 propname, value, prop.classname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1252
927
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1253 if (value is not None and
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1254 not self.db.getclass(link_class).hasnode(value)):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1255 raise IndexError('%s has no node %s' % (link_class,
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1256 value))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1257
927
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1258 if self.do_journal and prop.do_journal:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1259 # register the unlink with the old linked node
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1260 if propname in node and node[propname] is not None:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1261 self.db.addjournal(link_class, node[propname],
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1262 'unlink',
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1263 (self.classname, nodeid, propname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1264
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1265 # register the link with the newly linked node
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1266 if value is not None:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1267 self.db.addjournal(link_class, value, 'link',
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1268 (self.classname, nodeid, propname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1269
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1270 elif isinstance(prop, hyperdb.Multilink):
3872
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
1271 if value is None:
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
1272 value = []
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
1273 if not hasattr(value, '__iter__') or isinstance(value, str):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1274 raise TypeError('new property "%s" not an iterable of'
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1275 ' ids' % propname)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1276 link_class = self.properties[propname].classname
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1277 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1278 for entry in value:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1279 # if it isn't a number, it's a key
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
1280 if not isinstance(entry, str):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1281 raise ValueError('new property "%s" link value '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1282 'must be a string' % propname)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1283 if not num_re.match(entry):
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1284 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1285 entry = self.db.classes[link_class].lookup(entry)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1286 except (TypeError, KeyError):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1287 raise IndexError(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1288 'new property "%s": %s not a %s' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1289 propname, entry,
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1290 self.properties[propname].classname))
6999
e5a28b8ded72 backout checkin 60b37f632601
John Rouillard <rouilj@ieee.org>
parents: 6998
diff changeset
1291 l.append(entry)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1292 value = l
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1293 propvalues[propname] = value
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1294
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1295 # figure the journal entry for this property
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1296 add = []
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1297 remove = []
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1298
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1299 # handle removals
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1300 if propname in node:
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1301 l = node[propname]
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1302 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1303 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1304 for id in l[:]:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1305 if id in value:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1306 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1307 # register the unlink with the old linked node
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1308 if self.do_journal and self.properties[propname].do_journal:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1309 self.db.addjournal(link_class, id, 'unlink',
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1310 (self.classname, nodeid, propname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1311 l.remove(id)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1312 remove.append(id)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1313
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1314 # handle additions
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1315 for id in value:
902
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
1316 if not self.db.getclass(link_class).hasnode(id):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1317 raise IndexError('%s has no node %s' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1318 link_class, id))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1319 if id in l:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1320 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1321 # register the link with the newly linked node
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1322 if self.do_journal and self.properties[propname].do_journal:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1323 self.db.addjournal(link_class, id, 'link',
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1324 (self.classname, nodeid, propname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1325 l.append(id)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1326 add.append(id)
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1327
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1328 # figure the journal entry
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1329 l = []
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1330 if add:
967
dd35bab19dd9 use more robust date stamp comparisons in pack(), make journal smaller too
Richard Jones <richard@users.sourceforge.net>
parents: 964
diff changeset
1331 l.append(('+', add))
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1332 if remove:
967
dd35bab19dd9 use more robust date stamp comparisons in pack(), make journal smaller too
Richard Jones <richard@users.sourceforge.net>
parents: 964
diff changeset
1333 l.append(('-', remove))
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1334 if l:
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1335 journalvalues[propname] = tuple(l)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1336
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1337 elif isinstance(prop, hyperdb.String):
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1338 if value is not None and not isinstance(value, (str, unicode)):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1339 raise TypeError('new property "%s" not a '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1340 'string' % propname)
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2736
diff changeset
1341 if prop.indexme:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1342 self.db.indexer.add_text(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1343 (self.classname, nodeid, propname),
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2736
diff changeset
1344 value)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1345
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1346 elif isinstance(prop, hyperdb.Password):
8534
1f8492d68aca bug: using 'null' value for attributes causes error.
John Rouillard <rouilj@ieee.org>
parents: 8403
diff changeset
1347 if value is not None and not isinstance(value, password.Password):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1348 raise TypeError('new property "%s" not a '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1349 'Password' % propname)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1350 propvalues[propname] = value
4483
22bc0426e348 Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4480
diff changeset
1351 journalvalues[propname] = \
22bc0426e348 Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4480
diff changeset
1352 current and password.JournalPassword(current)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1353
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1354 elif value is not None and isinstance(prop, hyperdb.Date):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1355 if not isinstance(value, date.Date):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1356 raise TypeError('new property "%s" not a '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1357 'Date' % propname)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1358 propvalues[propname] = value
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1359
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1360 elif value is not None and isinstance(prop, hyperdb.Interval):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1361 if not isinstance(value, date.Interval):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1362 raise TypeError('new property "%s" not an '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1363 'Interval' % propname)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1364 propvalues[propname] = value
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1365
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1366 elif value is not None and isinstance(prop, hyperdb.Number):
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
1367 try:
890
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1368 float(value)
887
e7169d6e6e45 added tests for number type too
Richard Jones <richard@users.sourceforge.net>
parents: 886
diff changeset
1369 except ValueError:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1370 raise TypeError('new property "%s" not '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1371 'numeric' % propname)
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
1372
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1373 elif value is not None and isinstance(prop, hyperdb.Integer):
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1374 try:
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1375 int(value)
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1376 except ValueError:
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1377 raise TypeError('new property "%s" not '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1378 'numeric' % propname)
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1379
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1380 elif value is not None and isinstance(prop, hyperdb.Boolean):
890
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1381 try:
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1382 int(value)
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1383 except ValueError:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1384 raise TypeError('new property "%s" not '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1385 'boolean' % propname)
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
1386
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1387 node[propname] = value
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1388
5112
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1389 # record quiet properties to omit from history/changelog
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1390 if prop.quiet:
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1391 quiet_props.append(propname)
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1392
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1393 # nothing to do?
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1394 if not propvalues:
930
3c344e942055 Use same regex to split search terms as used to index text.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 927
diff changeset
1395 return propvalues
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1396
3083
b9a819ef7554 actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3052
diff changeset
1397 # update the activity time
b9a819ef7554 actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3052
diff changeset
1398 node['activity'] = date.Date()
b9a819ef7554 actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3052
diff changeset
1399 node['actor'] = self.db.getuid()
b9a819ef7554 actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3052
diff changeset
1400
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1401 # do the set, and journal it
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1402 self.db.setnode(self.classname, nodeid, node)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1403
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1404 if self.do_journal:
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1303
diff changeset
1405 self.db.addjournal(self.classname, nodeid, 'set', journalvalues)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1406
5112
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1407 # remove quiet properties from output
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1408 for propname in quiet_props:
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1409 if propname in propvalues:
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1410 del propvalues[propname]
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1411
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1412 return propvalues
930
3c344e942055 Use same regex to split search terms as used to index text.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 927
diff changeset
1413
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1414 def retire(self, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1415 """Retire a node.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1416
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1417 The properties on the node remain available from the get() method,
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1418 and the node's id is never reused.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1419
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1420 Retired nodes are not returned by the find(), list(), or lookup()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1421 methods, and other nodes may reuse the values of their key properties.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1422
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1423 These operations trigger detectors and can be vetoed. Attempts
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1424 to modify the "creation" or "activity" properties cause a KeyError.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1425 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1426 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1427 raise hyperdb.DatabaseError(_('Database open read-only'))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1428
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1429 self.fireAuditors('retire', nodeid, None)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1430
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1431 node = self.db.getnode(self.classname, nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1432 node[self.db.RETIRED_FLAG] = 1
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1433 self.db.setnode(self.classname, nodeid, node)
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1434 if self.do_journal:
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1435 self.db.addjournal(self.classname, nodeid, 'retired', None)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1436
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1437 self.fireReactors('retire', nodeid, None)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1438
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1439 def restore(self, nodeid):
5107
33ac07a7ef96 spelling error fixed in comment
John Rouillard <rouilj@ieee.org>
parents: 5100
diff changeset
1440 """Restore a retired node.
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1441
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1442 Make node available for all operations like it was before retirement.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1443 """
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1444 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1445 raise hyperdb.DatabaseError(_('Database open read-only'))
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1446
1523
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1447 node = self.db.getnode(self.classname, nodeid)
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1448 # check if key property was overrided
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1449 key = self.getkey()
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1450 try:
6998
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
1451 # eval for exception side effect
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
1452 id = self.lookup(node[key]) # noqa: F841
1523
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1453 except KeyError:
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1454 pass
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1455 else:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1456 raise KeyError("Key property (%s) of retired node clashes "
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1457 "with existing one (%s)" % (key, node[key]))
1523
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1458 # Now we can safely restore node
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1459 self.fireAuditors('restore', nodeid, None)
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1460 del node[self.db.RETIRED_FLAG]
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1461 self.db.setnode(self.classname, nodeid, node)
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1462 if self.do_journal:
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1463 self.db.addjournal(self.classname, nodeid, 'restored', None)
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1464
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1465 self.fireReactors('restore', nodeid, None)
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1466
8305
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
1467 def is_retired(self, nodeid, cldb=None, allow_abort=True):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1468 """Return true if the node is retired.
8305
a81a3cd067fa Generate savepoint only if necessary
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8304
diff changeset
1469 'allow_abort' is used only in sql backends.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1470 """
1476
5a01e90b7dc9 fixed export/import of retired nodes [SF#685273]
Richard Jones <richard@users.sourceforge.net>
parents: 1467
diff changeset
1471 node = self.db.getnode(self.classname, nodeid, cldb)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1472 if self.db.RETIRED_FLAG in node:
940
301a02ea6020 added is_retired query to Class
Richard Jones <richard@users.sourceforge.net>
parents: 930
diff changeset
1473 return 1
301a02ea6020 added is_retired query to Class
Richard Jones <richard@users.sourceforge.net>
parents: 930
diff changeset
1474 return 0
301a02ea6020 added is_retired query to Class
Richard Jones <richard@users.sourceforge.net>
parents: 930
diff changeset
1475
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1476 def destroy(self, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1477 """Destroy a node.
1333
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
1478
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1479 WARNING: this method should never be used except in extremely rare
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1480 situations where there could never be links to the node being
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1481 deleted
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1482
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1483 WARNING: use retire() instead
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1484
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1485 WARNING: the properties of this node will not be available ever again
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1486
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1487 WARNING: really, use retire() instead
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1488
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1489 Well, I think that's enough warnings. This method exists mostly to
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1490 support the session storage of the cgi interface.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1491 """
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1492 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1493 raise hyperdb.DatabaseError(_('Database open read-only'))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1494 self.db.destroynode(self.classname, nodeid)
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1495
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1496 # Locating nodes:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1497 def hasnode(self, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1498 """Determine if the given nodeid actually exists
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1499 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1500 return self.db.hasnode(self.classname, nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1501
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1502 def setkey(self, propname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1503 """Select a String property of this class to be the key property.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1504
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1505 'propname' must be the name of a String property of this class or
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1506 None, or a TypeError is raised. The values of the key property on
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
1507 all existing nodes must be unique or a ValueError is raised. If the
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
1508 property doesn't exist, KeyError is raised.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1509 """
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
1510 prop = self.getprops()[propname]
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1511 if not isinstance(prop, hyperdb.String):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1512 raise TypeError('key properties must be String')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1513 self.key = propname
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1514
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1515 def getkey(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1516 """Return the name of the key property for this class or None."""
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1517 return self.key
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1518
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1519 # TODO: set up a separate index db file for this? profile?
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1520 def lookup(self, keyvalue):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1521 """Locate a particular node by its key property and return its id.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1522
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1523 If this class has no key property, a TypeError is raised. If the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1524 'keyvalue' matches one of the values for the key property among
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1525 the nodes in this class, the matching node's id is returned;
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1526 otherwise a KeyError is raised.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1527 """
969
ddcb527491ba Consistent quoting
Richard Jones <richard@users.sourceforge.net>
parents: 967
diff changeset
1528 if not self.key:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1529 raise TypeError('No key property set for '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1530 'class %s' % self.classname)
7936
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1531
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1532 # special notation for looking up the current database user
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1533 if keyvalue == '@current_user' and self.classname == 'user':
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1534 keyvalue = self.db.user.get(self.db.getuid(), self.key)
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1535
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1536 cldb = self.db.getclassdb(self.classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1537 try:
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1538 for nodeid in self.getnodeids(cldb):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1539 node = self.db.getnode(self.classname, nodeid, cldb)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1540 if self.db.RETIRED_FLAG in node:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1541 continue
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1542 if self.key not in node:
2606
17eb5aeada7f fix bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2601
diff changeset
1543 continue
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1544 if node[self.key] == keyvalue:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1545 return nodeid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1546 finally:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1547 cldb.close()
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1548 raise KeyError('No key (%s) value "%s" for "%s"' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1549 self.key, keyvalue, self.classname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1550
1103
db787cef1385 handled some XXXs
Richard Jones <richard@users.sourceforge.net>
parents: 1099
diff changeset
1551 # change from spec - allows multiple props to match
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1552 def find(self, **propspec):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1553 """Get the ids of nodes in this class which link to the given nodes.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1554
3239
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3166
diff changeset
1555 'propspec' consists of keyword args propname=nodeid or
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3166
diff changeset
1556 propname={nodeid:1, }
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
1557 'propname' must be the name of a property in this class, or a
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
1558 KeyError is raised. That property must be a Link or
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
1559 Multilink property, or a TypeError is raised.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1560
3239
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3166
diff changeset
1561 Any node in this class whose 'propname' property links to any of
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3166
diff changeset
1562 the nodeids will be returned. Examples::
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
1563
3239
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3166
diff changeset
1564 db.issue.find(messages='1')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1565 db.issue.find(messages={'1':1,'3':1}, files={'7':1})
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1566 db.issue.find(messages=('1','3'), files=('7',))
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1567 db.issue.find(messages=['1','3'], files=['7'])
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1568 """
6427
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1569 # shortcut
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1570 if not propspec:
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1571 return []
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1572
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1573 # validate the args
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1574 props = self.getprops()
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
1575 for propname, itemids in propspec.items():
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1576 # check the prop is OK
6427
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1577 prop = props[propname]
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1578 if not isinstance(prop, hyperdb.Link) and not isinstance(prop, hyperdb.Multilink):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1579 raise TypeError("'%s' not a Link/Multilink "
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1580 "property" % propname)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1581
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1582 # ok, now do the find
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1583 cldb = self.db.getclassdb(self.classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1584 l = []
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1585 rev_multilinks = []
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1586 try:
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1587 for id in self.getnodeids(db=cldb):
1563
e2a8ce4d2317 Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents: 1558
diff changeset
1588 item = self.db.getnode(self.classname, id, db=cldb)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1589 if self.db.RETIRED_FLAG in item:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1590 continue
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
1591 for propname, itemids in propspec.items():
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1592 if not isinstance(itemids, dict):
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1593 if itemids is None or isinstance(itemids, type("")):
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1594 itemids = {itemids: 1}
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1595 else:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1596 itemids = dict.fromkeys(itemids)
1563
e2a8ce4d2317 Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents: 1558
diff changeset
1597
2450
c45ed2413044 fixed lookup of "missing" Link values for new props in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2437
diff changeset
1598 # special case if the item doesn't have this property
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1599 if propname not in item:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1600 if None in itemids:
2450
c45ed2413044 fixed lookup of "missing" Link values for new props in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2437
diff changeset
1601 l.append(id)
c45ed2413044 fixed lookup of "missing" Link values for new props in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2437
diff changeset
1602 break
c45ed2413044 fixed lookup of "missing" Link values for new props in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2437
diff changeset
1603 continue
c45ed2413044 fixed lookup of "missing" Link values for new props in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2437
diff changeset
1604
1563
e2a8ce4d2317 Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents: 1558
diff changeset
1605 # grab the property definition and its value on this item
6427
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1606 prop = props[propname]
1563
e2a8ce4d2317 Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents: 1558
diff changeset
1607 value = item[propname]
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1608 if isinstance(prop, hyperdb.Link) and value in itemids:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1609 l.append(id)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1610 break
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1611 elif isinstance(prop, hyperdb.Multilink):
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1612 if prop.rev_property:
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
1613 rev_multilinks.append((prop, itemids))
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1614 continue
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1615 hit = 0
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1616 for v in value:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1617 if v in itemids:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1618 l.append(id)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1619 hit = 1
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1620 break
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1621 if hit:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1622 break
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1623 for prop, itemids in rev_multilinks:
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1624 rprop = prop.rev_property
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1625 fun = l.append
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
1626 if isinstance(rprop, hyperdb.Multilink):
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1627 fun = l.extend
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1628 for id in itemids:
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1629 fun(rprop.cls.get(id, rprop.name))
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1630 if rev_multilinks:
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1631 l = list(sorted(set(l)))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1632 finally:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1633 cldb.close()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1634 return l
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1635
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1636 def stringFind(self, **requirements):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1637 """Locate a particular node by matching a set of its String
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1638 properties in a caseless search.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1639
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1640 If the property is not a String property, a TypeError is raised.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1641
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1642 The return is a list of the id of all nodes that match.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1643 """
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1644 for propname in requirements:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1645 prop = self.properties[propname]
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1646 if not isinstance(prop, hyperdb.String):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1647 raise TypeError("'%s' not a String property" % propname)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1648 requirements[propname] = requirements[propname].lower()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1649 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1650 cldb = self.db.getclassdb(self.classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1651 try:
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1652 for nodeid in self.getnodeids(cldb):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1653 node = self.db.getnode(self.classname, nodeid, cldb)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1654 if self.db.RETIRED_FLAG in node:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1655 continue
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
1656 for key, value in requirements.items():
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1657 if key not in node:
1296
34b1e6490e65 hardening for stringFind with missing props
Richard Jones <richard@users.sourceforge.net>
parents: 1252
diff changeset
1658 break
905
502a5ae11cc5 Very close now. The cgi and mailgw now use the new security API.
Richard Jones <richard@users.sourceforge.net>
parents: 902
diff changeset
1659 if node[key] is None or node[key].lower() != value:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1660 break
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1661 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1662 l.append(nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1663 finally:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1664 cldb.close()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1665 return l
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1666
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1667 def list(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1668 """ Return a list of the ids of the active nodes in this class.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1669 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1670 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1671 cn = self.classname
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1672 cldb = self.db.getclassdb(cn)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1673 try:
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1674 for nodeid in self.getnodeids(cldb):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1675 node = self.db.getnode(cn, nodeid, cldb)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1676 if self.db.RETIRED_FLAG in node:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1677 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1678 l.append(nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1679 finally:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1680 cldb.close()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1681 l.sort()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1682 return l
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1683
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1684 def getnodeids(self, db=None, retired=None):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1685 """ Return a list of ALL nodeids
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1686
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1687 Set retired=None to get all nodes. Otherwise it'll get all the
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1688 retired or non-retired nodes, depending on the flag.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1689 """
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1690 res = []
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1691
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1692 # start off with the new nodes
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1693 if self.classname in self.db.newnodes:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1694 res.extend(self.db.newnodes[self.classname])
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1695
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1696 must_close = False
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1697 if db is None:
1486
f5f60c75a458 added test for error in sqlite backend, and fixed *dbm backend error
Richard Jones <richard@users.sourceforge.net>
parents: 1484
diff changeset
1698 db = self.db.getclassdb(self.classname)
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
1699 must_close = True
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1700 try:
5444
167f0d25ea8e Python 3 preparation: convert dbm keys back from bytes to strings.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5395
diff changeset
1701 res.extend(map(b2s, db.keys()))
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1702
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1703 # remove the uncommitted, destroyed nodes
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1704 if self.classname in self.db.destroyednodes:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1705 for nodeid in self.db.destroyednodes[self.classname]:
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
1706 if nodeid in db:
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1707 res.remove(nodeid)
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1708
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1709 # check retired flag
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1710 if retired is False or retired is True:
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1711 l = []
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1712 for nodeid in res:
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1713 node = self.db.getnode(self.classname, nodeid, db)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1714 is_ret = self.db.RETIRED_FLAG in node
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1715 if retired == is_ret:
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1716 l.append(nodeid)
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1717 res = l
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1718 finally:
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1719 if must_close:
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1720 db.close()
7547
c8c4514f4c3e issue685275 - show retired/unretire commands
John Rouillard <rouilj@ieee.org>
parents: 7393
diff changeset
1721
c8c4514f4c3e issue685275 - show retired/unretire commands
John Rouillard <rouilj@ieee.org>
parents: 7393
diff changeset
1722 res.sort()
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1723 return res
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1724
7006
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1725 num_re = re.compile(r'^\d+$')
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1726
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1727 def _filter(self, search_matches, filterspec, proptree,
7006
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1728 num_re=num_re, retired=False,
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1729 exact_match_spec=_marker):
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1730 """Return a list of the ids of the nodes in this class that
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1731 match the 'filter' spec, sorted by the group spec and then the
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1732 sort spec.
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1733
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1734 "filterspec" is {propname: value(s)}
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1735 same for "exact_match_spec". The latter specifies exact matching
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1736 for String type while String types in "filterspec" are searched
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1737 for as case insensitive substring match.
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1738
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1739 "sort" and "group" are (dir, prop) where dir is '+', '-' or None
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1740 and prop is a prop name or None
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1741
4044
27a9906cd8d1 Fix issue2550505.
Stefan Seefeld <stefan@seefeld.name>
parents: 3979
diff changeset
1742 "search_matches" is a sequence type or None
1170
af104fa52746 Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents: 1159
diff changeset
1743
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1744 "retired" specifies if we should find only non-retired nodes
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1745 (default) or only retired node (value True) or all nodes.
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1746
3455
e01bc6d65fa9 fixed documentation of filter()...
Richard Jones <richard@users.sourceforge.net>
parents: 3419
diff changeset
1747 The filter must match all properties specificed. If the property
e01bc6d65fa9 fixed documentation of filter()...
Richard Jones <richard@users.sourceforge.net>
parents: 3419
diff changeset
1748 value to match is a list:
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
1749
3455
e01bc6d65fa9 fixed documentation of filter()...
Richard Jones <richard@users.sourceforge.net>
parents: 3419
diff changeset
1750 1. String properties must match all elements in the list, and
e01bc6d65fa9 fixed documentation of filter()...
Richard Jones <richard@users.sourceforge.net>
parents: 3419
diff changeset
1751 2. Other properties must match any of the elements in the list.
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1752 """
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
1753 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
1754 start_t = time.time()
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
1755
7006
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1756 if exact_match_spec is _marker:
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1757 exact_match_spec = {}
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1758
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1759 cn = self.classname
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1760
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1761 # optimise filterspec
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1762 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1763 props = self.getprops()
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1764 LINK = 'spec:link'
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1765 MULTILINK = 'spec:multilink'
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1766 STRING = 'spec:string'
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1767 DATE = 'spec:date'
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1768 INTERVAL = 'spec:interval'
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1769 OTHER = 'spec:other'
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1770
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1771 for exact, filtertype in enumerate((filterspec, exact_match_spec)):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1772 for k, v in filtertype.items():
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1773 propclass = props[k]
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1774 if isinstance(propclass, hyperdb.Link):
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1775 if not isinstance(v, list):
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1776 v = [v]
7936
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1777 if propclass.classname == 'user' and '@current_user' in v:
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1778 cu = self.db.getuid()
a9b136565838 feat: issue1525113 - notation to filter by logged-in user
John Rouillard <rouilj@ieee.org>
parents: 7884
diff changeset
1779 v = [x if x != "@current_user" else cu for x in v]
6412
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
1780 l.append((LINK, k, v))
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1781 elif isinstance(propclass, hyperdb.Multilink):
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1782 # If it's a reverse multilink, we've already
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1783 # computed the ids of our own class.
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1784 if propclass.rev_property:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1785 l.append((OTHER, 'id', v))
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1786 else:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1787 # the value -1 is a special "not set" sentinel
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1788 if v in ('-1', ['-1']):
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1789 v = []
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1790 elif not isinstance(v, list):
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1791 v = [v]
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1792 l.append((MULTILINK, k, v))
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1793 elif isinstance(propclass, hyperdb.String) and k != 'id':
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1794 if not isinstance(v, list):
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1795 v = [v]
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1796 for x in v:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1797 if exact:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1798 l.append((STRING, k, x))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1799 else:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1800 # simple glob searching
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1801 x = re.sub(r'([\|\{\}\\\.\+\[\]\(\)])', r'\\\1', x)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1802 x = x.replace('?', '.')
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1803 x = x.replace('*', '.*?')
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1804 l.append((STRING, k, re.compile(x, re.I)))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1805 elif isinstance(propclass, hyperdb.Date):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1806 try:
6118
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1807 ranges = []
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1808 for d in v.split(','):
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1809 if d == '-':
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1810 ranges.append(None)
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1811 continue
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1812 date_rng = propclass.range_from_raw(d, self.db)
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1813 ranges.append(date_rng)
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1814 l.append((DATE, k, ranges))
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1815 except ValueError:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1816 # If range creation fails - ignore that search parameter
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1817 pass
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1818 elif isinstance(propclass, hyperdb.Interval):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1819 try:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1820 intv_rng = date.Range(v, date.Interval)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1821 l.append((INTERVAL, k, intv_rng))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1822 except ValueError:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1823 # If range creation fails - ignore that search parameter
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1824 pass
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1825
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1826 elif isinstance(propclass, hyperdb.Boolean):
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1827 if isinstance(v, str):
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1828 v = v.split(',')
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1829 if not isinstance(v, list):
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1830 v = [v]
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1831 bv = []
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1832 for val in v:
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1833 if isinstance(val, str):
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
1834 bv.append(propclass.from_raw(val))
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1835 else:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1836 bv.append(val)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1837 l.append((OTHER, k, bv))
2453
0e2a0c2c8142 allow list of values for id, Number and Boolean filtering in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2450
diff changeset
1838
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1839 elif k == 'id':
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1840 if not isinstance(v, list):
4364
0e81742d0e2f - unify number searching across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4357
diff changeset
1841 v = v.split(',')
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1842 l.append((OTHER, k, [str(int(val)) for val in v]))
2453
0e2a0c2c8142 allow list of values for id, Number and Boolean filtering in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2450
diff changeset
1843
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1844 elif isinstance(propclass, hyperdb.Number):
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1845 if not isinstance(v, list):
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1846 try:
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1847 v = v.split(',')
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1848 except AttributeError:
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1849 v = [v]
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1850 l.append((OTHER, k, [float(val) for val in v]))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1851
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1852 elif isinstance(propclass, hyperdb.Integer):
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1853 if not isinstance(v, list):
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1854 try:
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1855 v = v.split(',')
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1856 except AttributeError:
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1857 v = [v]
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1858 l.append((OTHER, k, [int(val) for val in v]))
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1859
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1860 filterspec = l
4480
1613754d2646 Fix first part of Password handling security issue2550688
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1861
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1862 # now, find all the nodes that pass filtering
2239
c8a06e10e2c6 much faster anydbm filter(), but it breaks most filtering tests
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
1863 matches = []
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1864 cldb = self.db.getclassdb(cn)
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1865 t = 0
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1866 try:
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1867 # TODO: only full-scan once (use items())
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1868 for nodeid in self.getnodeids(cldb, retired=retired):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1869 node = self.db.getnode(cn, nodeid, cldb)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1870 # apply filter
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1871 for t, k, v in filterspec:
1303
71be6588904f fixed filtering by id in anydbm
Richard Jones <richard@users.sourceforge.net>
parents: 1296
diff changeset
1872 # handle the id prop
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1873 if k == 'id':
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1874 if nodeid not in v:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1875 break
1303
71be6588904f fixed filtering by id in anydbm
Richard Jones <richard@users.sourceforge.net>
parents: 1296
diff changeset
1876 continue
71be6588904f fixed filtering by id in anydbm
Richard Jones <richard@users.sourceforge.net>
parents: 1296
diff changeset
1877
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1878 # get the node value
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1879 nv = node.get(k, None)
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1880
2617
33fffbf7ae68 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2608
diff changeset
1881 match = 0
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1882
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1883 # now apply the property filter
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1884 if t == LINK:
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1885 # link - if this node's property doesn't appear in the
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1886 # filterspec's nodeid list, skip it
8241
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1887 try:
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1888 expr = Expression(v, is_link=True)
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1889 except ExpressionError as e:
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1890 e.context['class'] = cn
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1891 e.context['attr'] = k
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1892 raise
6412
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
1893 if expr.evaluate(nv):
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
1894 match = 1
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1895 elif t == MULTILINK:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1896 # multilink - if any of the nodeids required by the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1897 # filterspec aren't in this node's property, then skip
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1898 # it
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1899 nv = node.get(k, [])
1555
948c7764d46c implemented ability to search for multilink properties with no value (not in mk)
Richard Jones <richard@users.sourceforge.net>
parents: 1523
diff changeset
1900
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1901 # check for matching the absence of multilink values
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1902 if not v:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1903 match = not nv
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1904 else:
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4447
diff changeset
1905 # otherwise, make sure this node has each of the
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1906 # required values
8241
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1907 try:
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1908 expr = Expression(v)
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1909 except ExpressionError as e:
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1910 e.context['class'] = cn
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1911 e.context['attr'] = k
741ea8a86012 fix: issue2551374. Error handling for filter expressions.
John Rouillard <rouilj@ieee.org>
parents: 8130
diff changeset
1912 raise
6396
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6352
diff changeset
1913 if expr.evaluate(nv):
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6352
diff changeset
1914 match = 1
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1915 elif t == STRING:
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1916 if nv is None:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1917 nv = ''
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1918 if is_us(v):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1919 # Exact match
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1920 match = (nv == v)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1921 else:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1922 # RE search
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1923 match = v.search(nv)
6118
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1924 elif t == DATE:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1925 for x in v:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1926 if x is None or nv is None:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1927 if nv is None and x is None:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1928 match = 1
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1929 break
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1930 continue
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1931 elif x.to_value:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1932 if x.from_value <= nv <= x.to_value:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1933 match = 1
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1934 break
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1935 else:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1936 if x.from_value <= nv:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1937 match = 1
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1938 break
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1939 elif t == INTERVAL:
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1940 if nv is None:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1941 match = v is None
1499
8ee69708da0c added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1496
diff changeset
1942 else:
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1943 if v.to_value:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1944 if v.from_value <= nv and v.to_value >= nv:
2617
33fffbf7ae68 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2608
diff changeset
1945 match = 1
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1946 else:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1947 if v.from_value <= nv:
2617
33fffbf7ae68 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2608
diff changeset
1948 match = 1
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1949 elif t == OTHER:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1950 # straight value comparison for the other types
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1951 match = nv in v
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1952 if not match:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1953 break
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1954 else:
2239
c8a06e10e2c6 much faster anydbm filter(), but it breaks most filtering tests
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
1955 matches.append([nodeid, node])
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1956
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1957 # filter based on full text search
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1958 if search_matches is not None:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1959 k = []
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1960 for v in matches:
4044
27a9906cd8d1 Fix issue2550505.
Stefan Seefeld <stefan@seefeld.name>
parents: 3979
diff changeset
1961 if v[0] in search_matches:
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1962 k.append(v)
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1963 matches = k
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1964
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1965 # add sorting information to the proptree
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1966 JPROPS = {'actor': 1, 'activity': 1, 'creator': 1, 'creation': 1}
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1967 children = []
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1968 if proptree:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1969 children = proptree.sortable_children()
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1970 for pt in children:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1971 dir = pt.sort_direction
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1972 prop = pt.name
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1973 assert (dir and prop)
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1974 propclass = props[prop]
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1975 pt.sort_ids = []
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1976 is_pointer = isinstance(propclass, (hyperdb.Link,
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1977 hyperdb.Multilink))
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1978 if not is_pointer:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1979 pt.sort_result = []
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1980 try:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1981 # cache the opened link class db, if needed.
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1982 lcldb = None
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1983 # cache the linked class items too
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1984 lcache = {}
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1985
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1986 for entry in matches:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1987 itemid = entry[-2]
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1988 item = entry[-1]
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1989 # handle the properties that might be "faked"
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1990 # also, handle possible missing properties
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1991 try:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1992 v = item[prop]
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1993 except KeyError:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1994 if prop in JPROPS:
2437
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1995 # force lookup of the special journal prop
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1996 v = self.get(itemid, prop)
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1997 else:
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1998 # the node doesn't have a value for this
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1999 # property
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2000 v = None
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2001 if isinstance(propclass, hyperdb.Multilink):
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2002 v = []
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2003 if prop == 'id':
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
2004 v = int(itemid)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2005 pt.sort_ids.append(v)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2006 if not is_pointer:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2007 pt.sort_result.append(v)
2437
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
2008 continue
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2009
2285
c7f780c24a87 fixed anydbm sorting with None values [SF#952853]
Richard Jones <richard@users.sourceforge.net>
parents: 2252
diff changeset
2010 # missing (None) values are always sorted first
c7f780c24a87 fixed anydbm sorting with None values [SF#952853]
Richard Jones <richard@users.sourceforge.net>
parents: 2252
diff changeset
2011 if v is None:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2012 pt.sort_ids.append(v)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2013 if not is_pointer:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2014 pt.sort_result.append(v)
2285
c7f780c24a87 fixed anydbm sorting with None values [SF#952853]
Richard Jones <richard@users.sourceforge.net>
parents: 2252
diff changeset
2015 continue
c7f780c24a87 fixed anydbm sorting with None values [SF#952853]
Richard Jones <richard@users.sourceforge.net>
parents: 2252
diff changeset
2016
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2017 if isinstance(propclass, hyperdb.Link):
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2018 lcn = propclass.classname
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2019 link = self.db.classes[lcn]
3476
1142dafe0d7f added setorderprop() and setlabelprop() to Class (lots of patches)
Richard Jones <richard@users.sourceforge.net>
parents: 3455
diff changeset
2020 key = link.orderprop()
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2021 child = pt.propdict[key]
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
2022 if key != 'id':
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2023 if v not in lcache:
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2024 # open the link class db if it's not already
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2025 if lcldb is None:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2026 lcldb = self.db.getclassdb(lcn)
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2027 lcache[v] = self.db.getnode(lcn, v, lcldb)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2028 r = lcache[v][key]
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2029 child.propdict[key].sort_ids.append(r)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2030 else:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2031 child.propdict[key].sort_ids.append(v)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2032 pt.sort_ids.append(v)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2033 if not is_pointer:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2034 r = propclass.sort_repr(pt.parent.cls, v, pt.name)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2035 pt.sort_result.append(r)
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2036 finally:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2037 # if we opened the link class db, close it now
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2038 if lcldb is not None:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2039 lcldb.close()
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2040 del lcache
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2041 finally:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2042 cldb.close()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2043
2239
c8a06e10e2c6 much faster anydbm filter(), but it breaks most filtering tests
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
2044 # pull the id out of the individual entries
c8a06e10e2c6 much faster anydbm filter(), but it breaks most filtering tests
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
2045 matches = [entry[-2] for entry in matches]
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
2046 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
2047 self.db.stats['filtering'] += (time.time() - start_t)
2239
c8a06e10e2c6 much faster anydbm filter(), but it breaks most filtering tests
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
2048 return matches
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2049
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2050 def count(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2051 """Get the number of nodes in this class.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2052
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2053 If the returned integer is 'numnodes', the ids of all the nodes
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2054 in this class run from 1 to numnodes, and numnodes+1 will be the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2055 id of the next node to be created in this class.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2056 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2057 return self.db.countnodes(self.classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2058
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2059 # Manipulating properties:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2060
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2061 def getprops(self, protected=1):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2062 """Return a dictionary mapping property names to property objects.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2063 If the "protected" flag is true, we include protected properties -
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2064 those which may not be modified.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2065
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2066 In addition to the actual properties on the node, these
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2067 methods provide the "creation" and "activity" properties. If the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2068 "protected" flag is true, we include protected properties - those
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2069 which may not be modified.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2070 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2071 d = self.properties.copy()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2072 if protected:
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
2073 d['id'] = hyperdb.String()
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2074 d['creation'] = hyperdb.Date()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2075 d['activity'] = hyperdb.Date()
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
2076 d['creator'] = hyperdb.Link('user')
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
2077 d['actor'] = hyperdb.Link('user')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2078 return d
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2079
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2080 def addprop(self, **properties):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2081 """Add properties to this class.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2082
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2083 The keyword arguments in 'properties' must map names to property
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2084 objects, or a TypeError is raised. None of the keys in 'properties'
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2085 may collide with the names of existing properties, or a ValueError
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2086 is raised before any properties have been added.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2087 """
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2088 for key in properties:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2089 if key in self.properties:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2090 raise ValueError(key)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2091 self.properties.update(properties)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2092
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2093 def index(self, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2094 """ Add (or refresh) the node to search indexes """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2095 # find all the String properties that have indexme
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
2096 for prop, propclass in self.getprops().items():
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2097 if isinstance(propclass, hyperdb.String) and propclass.indexme:
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2098 # index them under (classname, nodeid, property)
7393
9e612a39547a Make anydbm indexer handle missing items the same as rdbms backend
John Rouillard <rouilj@ieee.org>
parents: 7211
diff changeset
2099 self.db.indexer.add_text((self.classname, nodeid, prop),
9e612a39547a Make anydbm indexer handle missing items the same as rdbms backend
John Rouillard <rouilj@ieee.org>
parents: 7211
diff changeset
2100 str(self.get(nodeid, prop)))
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2101
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2102 #
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2103 # import / export support
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2104 #
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2105 def export_list(self, propnames, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2106 """ Export a node - generate a list of CSV-able data in the order
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2107 specified by propnames for the given node.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2108 """
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2109 properties = self.getprops()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2110 l = []
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2111 for prop in propnames:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2112 proptype = properties[prop]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2113 value = self.get(nodeid, prop)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2114 # "marshal" data where needed
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2115 if value is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2116 pass
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2117 elif isinstance(proptype, hyperdb.Date):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2118 value = value.get_tuple()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2119 elif isinstance(proptype, hyperdb.Interval):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2120 value = value.get_tuple()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2121 elif isinstance(proptype, hyperdb.Password):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2122 value = str(value)
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2123 l.append(repr_export(value))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2124
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2125 # append retired flag
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2126 l.append(repr_export(self.is_retired(nodeid)))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2127
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2128 return l
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2129
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2130 def import_list(self, propnames, proplist):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2131 """ Import a node - all information including "id" is present and
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2132 should not be sanity checked. Triggers are not triggered. The
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2133 journal should be initialised using the "creator" and "created"
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2134 information.
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2135
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2136 Return the nodeid of the node imported.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2137 """
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2138 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2139 raise hyperdb.DatabaseError(_('Database open read-only'))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2140 properties = self.getprops()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2141
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2142 # make the new node's property map
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2143 d = {}
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2144 newid = None
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2145 for i in range(len(propnames)):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2146 # Figure the property for this column
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2147 propname = propnames[i]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2148
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2149 # Use eval_import to reverse the repr_export() used to
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2150 # output the CSV
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2151 value = eval_import(proplist[i])
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2152
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2153 # "unmarshal" where necessary
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2154 if propname == 'id':
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2155 newid = value
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2156 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2157 elif propname == 'is retired':
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2158 # is the item retired?
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2159 if int(value):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2160 d[self.db.RETIRED_FLAG] = 1
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2161 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2162 elif value is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2163 d[propname] = None
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2164 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2165
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2166 prop = properties[propname]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2167 if isinstance(prop, hyperdb.Date):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2168 value = date.Date(value)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2169 elif isinstance(prop, hyperdb.Interval):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2170 value = date.Interval(value)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2171 elif isinstance(prop, hyperdb.Password):
7211
506c86823abb Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents: 7038
diff changeset
2172 value = password.Password(encrypted=value,
506c86823abb Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents: 7038
diff changeset
2173 config=self.db.config)
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2174 d[propname] = value
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2176 # get a new id if necessary
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2177 if newid is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2178 newid = self.db.newid(self.classname)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2179
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2180 # add the node and journal
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2181 self.db.addnode(self.classname, newid, d)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2182 return newid
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2183
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2184 def export_journals(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2185 """Export a class's journal - generate a list of lists of
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2186 CSV-able data:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2187
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2188 nodeid, date, user, action, params
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2189
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2190 No heading here - the columns are fixed.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2191 """
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2192 properties = self.getprops()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2193 r = []
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2194 for nodeid in self.getnodeids():
6998
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
2195 for nodeid, date_, user, action, params in self.history(
6997
60b37f632601 Flake8 fix whitespace/indent issues.
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
2196 nodeid, enforceperm=False, skipquiet=False):
6998
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
2197 date_ = date_.get_tuple()
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2198 if action == 'set':
3003
855e9f99c96c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 2966
diff changeset
2199 export_data = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
2200 for propname, value in params.items():
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2201 if propname not in properties:
3003
855e9f99c96c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 2966
diff changeset
2202 # property no longer in the schema
855e9f99c96c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 2966
diff changeset
2203 continue
855e9f99c96c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 2966
diff changeset
2204
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2205 prop = properties[propname]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2206 # make sure the params are eval()'able
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2207 if value is None:
3826
bf2e9535da00 Journal and database testing.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3687
diff changeset
2208 pass
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
2209 elif isinstance(prop, hyperdb.Date):
3166
3204488d20b5 hack to fix some anydbm export problems [SF#1081454]
Richard Jones <richard@users.sourceforge.net>
parents: 3155
diff changeset
2210 # this is a hack - some dates are stored as strings
3204488d20b5 hack to fix some anydbm export problems [SF#1081454]
Richard Jones <richard@users.sourceforge.net>
parents: 3155
diff changeset
2211 if not isinstance(value, type('')):
3204488d20b5 hack to fix some anydbm export problems [SF#1081454]
Richard Jones <richard@users.sourceforge.net>
parents: 3155
diff changeset
2212 value = value.get_tuple()
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
2213 elif isinstance(prop, hyperdb.Interval):
3390
a2e4b8ab5b51 include hack from [SF#1238571]
Richard Jones <richard@users.sourceforge.net>
parents: 3383
diff changeset
2214 # hack too - some intervals are stored as strings
a2e4b8ab5b51 include hack from [SF#1238571]
Richard Jones <richard@users.sourceforge.net>
parents: 3383
diff changeset
2215 if not isinstance(value, type('')):
a2e4b8ab5b51 include hack from [SF#1238571]
Richard Jones <richard@users.sourceforge.net>
parents: 3383
diff changeset
2216 value = value.get_tuple()
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
2217 elif isinstance(prop, hyperdb.Password):
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2218 value = str(value)
3003
855e9f99c96c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 2966
diff changeset
2219 export_data[propname] = value
3052
230f7394cce9 fix to fix
Richard Jones <richard@users.sourceforge.net>
parents: 3003
diff changeset
2220 params = export_data
6998
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
2221 r.append([repr_export(nodeid), repr_export(date_),
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2222 repr_export(user), repr_export(action),
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2223 repr_export(params)])
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2224 return r
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2225
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
2226
2597
c86b2179085b fix journal export of files to remove content from CSV files
Richard Jones <richard@users.sourceforge.net>
parents: 2514
diff changeset
2227 class FileClass(hyperdb.FileClass, Class):
8304
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2228 # Use for explicit upcalls in generic code, for py2 compat we cannot
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2229 # use super() without making everything a new-style class.
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2230 subclass = Class
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2231 def __init__(self, db, classname, **properties):
8304
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2232 self._update_properties(properties)
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2233 Class.__init__(self, db, classname, **properties)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2234
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2235 class IssueClass(Class, roundupdb.IssueClass):
8304
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2236 # Use for explicit upcalls in generic code, for py2 compat we cannot
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2237 # use super() without making everything a new-style class.
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2238 subclass = Class
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2239 def __init__(self, db, classname, **properties):
8304
24549122f9b1 Factor common code to hyperdb/roundupdb
Ralf Schlatterbeck <rsc@runtux.com>
parents: 8241
diff changeset
2240 self._update_properties(classname, properties)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2241 Class.__init__(self, db, classname, **properties)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2242
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
2243 # vim: set et sts=4 sw=4 :
Roundup Issue Tracker: http://roundup-tracker.org/