Mercurial > p > roundup > code

annotate roundup/backends/back_anydbm.py @ 7393:9e612a39547a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Make anydbm indexer handle missing items the same as rdbms backend The anydbm Class::index() method was capturing the IndexError rather than letting it bubble up. This is different from the rdbms backends. In implementing batch/incremental indexing via roundup-admin, I wanted to report any missing items that were specified in the batch. This worked by capturing the IndexError. It worked with rdbms backends but not anydbm. This change makes it work with anydbm as well. Focused testing doesn't show any regressions, so committing the change.
author John Rouillard <rouilj@ieee.org>
date Wed, 24 May 2023 11:44:35 -0400
parents 506c86823abb
children 82bbb95e5690 c8c4514f4c3e
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
213
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
1 #
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/)
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
3 # This module is free software, and you may redistribute it and/or modify
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
4 # under the same terms as Python, so long as this copyright message and
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
5 # disclaimer are retained in their original form.
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
6 #
214
18134bffab37 stupid typo
Richard Jones <richard@users.sourceforge.net>
parents: 213
diff changeset
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR
213
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
10 # POSSIBILITY OF SUCH DAMAGE.
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
11 #
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS"
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 149
diff changeset
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
17 #
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
18 """This module defines a backend that saves the hyperdatabase in a
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
19 database chosen by anydbm. It is guaranteed to always be available in python
440
de5bf4191f11 Enabled transaction support in the bsddb backend.
Richard Jones <richard@users.sourceforge.net>
parents: 431
diff changeset
20 versions >2.1.1 (the dumbdbm fallback in 2.1.1 and earlier has several
de5bf4191f11 Enabled transaction support in the bsddb backend.
Richard Jones <richard@users.sourceforge.net>
parents: 431
diff changeset
21 serious bugs, and is not available)
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
22 """
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
23 __docformat__ = 'restructuredtext'
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
24
6998
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
25 import copy
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
26 import logging
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
27 import marshal
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
28 import os
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
29 import re
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
30 import shutil
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
31 import time
1809
bd127cafe3a8 Simplify backend importing, by moving the imports into the backend modules.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 1800
diff changeset
32
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
33 from roundup.anypy.dbm_ import anydbm, whichdb
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
34 from roundup.anypy.strings import b2s, bs2b, repr_export, eval_import, is_us
2908
95813789cf70 translate UI message
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2906
diff changeset
35
3589
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
36 from roundup import hyperdb, date, password, roundupdb, security, support
6401
8bc5faeb7677 Make rev multilink expressions work for anydbm
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6399
diff changeset
37 from roundup.mlink_expr import Expression
2908
95813789cf70 translate UI message
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2906
diff changeset
38 from roundup.backends import locking
95813789cf70 translate UI message
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2906
diff changeset
39 from roundup.i18n import _
95813789cf70 translate UI message
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2906
diff changeset
40
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
41 from roundup.backends.blobfiles import FileStorage
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
42 from roundup.backends import sessions_dbm
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
43
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
44 try:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
45 from roundup.backends import sessions_redis
6820
feb175271f3e redis import failure on python2 causes crash
John Rouillard <rouilj@ieee.org>
parents: 6814
diff changeset
46 except ImportError:
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
47 sessions_redis = None
3295
a615cc230160 added Xapian indexer; replaces standard indexers if Xapian is available
Richard Jones <richard@users.sourceforge.net>
parents: 3239
diff changeset
48
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
49 from roundup.backends.indexer_common import get_indexer
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
50
5725
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5704
diff changeset
51 from hashlib import md5
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5704
diff changeset
52
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
53
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
54 def db_exists(config):
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
55 # check for the user db
2637
11811b313459 The demo.py script works again using the new configuration system.
Richard Jones <richard@users.sourceforge.net>
parents: 2633
diff changeset
56 for db in 'nodes.user nodes.user.db'.split():
2736
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
57 if os.path.exists(os.path.join(config.DATABASE, db)):
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
58 return 1
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
59 return 0
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
60
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
61
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
62 def db_nuke(config):
2736
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
63 shutil.rmtree(config.DATABASE)
2633
a9e1fff1e793 I thought I committed this last night. Ho hum.
Richard Jones <richard@users.sourceforge.net>
parents: 2617
diff changeset
64
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
65
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
66 # python 3 doesn't have a unicode type
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
67 try:
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
68 unicode # noqa: F821
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
69 except NameError:
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
70 unicode = str
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
71
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
72
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
73 # marker used for an unspecified keyword argument
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
74 _marker = []
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
75
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
76
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
77 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
78 # Now the database
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
79 #
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
80
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
81
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
82 class Database(FileStorage, hyperdb.Database, roundupdb.Database):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
83 """A database for storing records containing flexible data types.
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
84
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
85 Transaction stuff TODO:
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
86
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
87 - check the timestamp of the class file and nuke the cache if it's
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
88 modified. Do some sort of conflict checking on the dirty stuff.
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
89 - perhaps detect write collisions (related to above)?
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
90
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
91 attributes:
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
92 dbtype:
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
93 holds the value for the type of db. It is used by indexer to
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
94 identify the database type so it can import the correct indexer
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
95 module when using native text search mode.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
96 """
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
97
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
98 dbtype = "anydbm"
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
99
5100
5775959d81a3 issue1926124: fix crash in roundup_admin migrate option wih anydbm
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
100 # used by migrate roundup_admin command. Is a no-op for anydbm.
5775959d81a3 issue1926124: fix crash in roundup_admin migrate option wih anydbm
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
101 # but needed to stop traceback in admin.
5775959d81a3 issue1926124: fix crash in roundup_admin migrate option wih anydbm
John Rouillard <rouilj@ieee.org>
parents: 5096
diff changeset
102 db_version_updated = False
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
103
524
dce4c75bef5a changed all config accesses...
Richard Jones <richard@users.sourceforge.net>
parents: 475
diff changeset
104 def __init__(self, config, journaltag=None):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
105 """Open a hyperdatabase given a specifier to some storage.
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
106
524
dce4c75bef5a changed all config accesses...
Richard Jones <richard@users.sourceforge.net>
parents: 475
diff changeset
107 The 'storagelocator' is obtained from config.DATABASE.
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
108 The meaning of 'storagelocator' depends on the particular
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
109 implementation of the hyperdatabase. It could be a file name,
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
110 a directory path, a socket descriptor for a connection to a
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
111 database over the network, etc.
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
112
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
113 The 'journaltag' is a token that will be attached to the journal
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
114 entries for any edits done on the database. If 'journaltag' is
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
115 None, the database is opened in read-only mode: the Class.create(),
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
116 Class.set(), Class.retire(), and Class.restore() methods are
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
117 disabled.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
118 """
3897
e2c2d91932ad respect umask on filestorage backend (fixes [SF#744328])
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3872
diff changeset
119 FileStorage.__init__(self, config.UMASK)
6658
408fd477761f Add i18n object to roundupdb.Database
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6427
diff changeset
120 roundupdb.Database.__init__(self)
524
dce4c75bef5a changed all config accesses...
Richard Jones <richard@users.sourceforge.net>
parents: 475
diff changeset
121 self.config, self.journaltag = config, journaltag
dce4c75bef5a changed all config accesses...
Richard Jones <richard@users.sourceforge.net>
parents: 475
diff changeset
122 self.dir = config.DATABASE
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
123 self.classes = {}
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
124 self.cache = {} # cache of nodes loaded or created
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
125 self.stats = {'cache_hits': 0, 'cache_misses': 0, 'get_items': 0,
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
126 'filtering': 0}
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
127 self.dirtynodes = {} # keep track of the dirty nodes by class
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
128 self.newnodes = {} # keep track of the new nodes by class
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
129 self.destroyednodes = {} # keep track of the destroyed nodes by class
252
76c6994aa4e8 CGI interfaces now spit up a top-level index of all instances they can serve.
Richard Jones <richard@users.sourceforge.net>
parents: 224
diff changeset
130 self.transactions = []
5096
e74c3611b138 - issue2550636, issue2550909: Added support for Whoosh indexer.
John Rouillard <rouilj@ieee.org>
parents: 5067
diff changeset
131 self.indexer = get_indexer(config, self)
902
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
132 self.security = security.Security(self)
3609
f2fda3e6fc8b umask is now configurable (with the same 0002 default)
Richard Jones <richard@users.sourceforge.net>
parents: 3601
diff changeset
133 os.umask(config.UMASK)
440
de5bf4191f11 Enabled transaction support in the bsddb backend.
Richard Jones <richard@users.sourceforge.net>
parents: 431
diff changeset
134
5041
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 5011
diff changeset
135 # make sure the database directory exists
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 5011
diff changeset
136 if not os.path.isdir(self.config.DATABASE):
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 5011
diff changeset
137 os.makedirs(self.config.DATABASE)
5251e97b1de0 Configure the database backend in the config.ini
John Kristensen <john@jerrykan.com>
parents: 5011
diff changeset
138
1333
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
139 # lock it
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
140 lockfilenm = os.path.join(self.dir, 'lock')
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
141 self.lockfile = locking.acquire_lock(lockfilenm)
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
142 self.lockfile.write(str(os.getpid()))
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
143 self.lockfile.flush()
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
144
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
145 self.Session = None
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
146 self.Otk = None
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
147
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
148 def post_init(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
149 """Called once the schema initialisation has finished.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
150 """
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
151 super(Database, self).post_init()
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
152 # reindex the db if necessary
826
6d7a45c8464a Added reindex command to roundup-admin.
Richard Jones <richard@users.sourceforge.net>
parents: 825
diff changeset
153 if self.indexer.should_reindex():
6d7a45c8464a Added reindex command to roundup-admin.
Richard Jones <richard@users.sourceforge.net>
parents: 825
diff changeset
154 self.reindex()
6d7a45c8464a Added reindex command to roundup-admin.
Richard Jones <richard@users.sourceforge.net>
parents: 825
diff changeset
155
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1809
diff changeset
156 def refresh_database(self):
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
157 """Rebuild the database
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
158 """
1840
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1809
diff changeset
159 self.reindex()
91a4619b1a14 hyperdb grows a refresh_database() method.
Anthony Baxter <anthonybaxter@users.sourceforge.net>
parents: 1809
diff changeset
160
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
161 def getSessionManager(self):
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
162 if not self.Session:
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
163 if self.config.SESSIONDB_BACKEND == "redis":
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
164 if sessions_redis is None:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
165 self.Session = sessions_dbm.Sessions(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
166 raise ValueError("[redis] session is set, but "
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
167 "redis is not found")
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
168 self.Session = sessions_redis.Sessions(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
169 else:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
170 self.Session = sessions_dbm.Sessions(self)
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
171 return self.Session
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
172
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
173 def getOTKManager(self):
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
174 if not self.Otk:
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
175 if self.config.SESSIONDB_BACKEND == "redis":
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
176 if sessions_redis is None:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
177 self.Session = sessions_dbm.OneTimeKeys(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
178 raise ValueError("[redis] session is set, but "
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
179 "redis is not found")
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
180 self.Otk = sessions_redis.OneTimeKeys(self)
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
181 else:
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
182 self.Otk = sessions_dbm.OneTimeKeys(self)
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
183 return self.Otk
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 2077
diff changeset
184
3589
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
185 def reindex(self, classname=None, show_progress=False):
2650
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
186 if classname:
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
187 classes = [self.getclass(classname)]
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
188 else:
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
189 classes = self.classes.values()
d68a444fcce3 roundup-admin reindex command may now work on single items or classes
Richard Jones <richard@users.sourceforge.net>
parents: 2649
diff changeset
190 for klass in classes:
3589
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
191 if show_progress:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
192 for nodeid in support.Progress('Reindex %s' %
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
193 klass.classname, klass.list()):
3589
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
194 klass.index(nodeid)
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
195 else:
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
196 for nodeid in klass.list():
1be293265e61 woo, that was quick
Richard Jones <richard@users.sourceforge.net>
parents: 3586
diff changeset
197 klass.index(nodeid)
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
198 self.indexer.save_index()
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
199
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
200 def __repr__(self):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
201 return '<back_anydbm instance at %x>' % id(self)
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
202
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
203 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
204 # Classes
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
205 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
206 def __getattr__(self, classname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
207 """A convenient way of calling self.getclass(classname)."""
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
208 if classname in self.classes:
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
209 return self.classes[classname]
5378
35ea9b1efc14 Python 3 preparation: "raise" syntax.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5319
diff changeset
210 raise AttributeError(classname)
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
211
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
212 def addclass(self, cl):
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
213 cn = cl.classname
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
214 if cn in self.classes:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
215 raise ValueError(_('Class "%s" already defined.' % cn))
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
216 self.classes[cn] = cl
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
217
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
218 # add default Edit and View permissions
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
219 self.security.addPermission(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
220 name="Create", klass=cn,
2649
1df7d4a41da4 Buncha stuff (sorry about the large checkin):
Richard Jones <richard@users.sourceforge.net>
parents: 2640
diff changeset
221 description="User is allowed to create "+cn)
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
222 self.security.addPermission(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
223 name="Edit", klass=cn,
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
224 description="User is allowed to edit "+cn)
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
225 self.security.addPermission(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
226 name="View", klass=cn,
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
227 description="User is allowed to access "+cn)
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
228 self.security.addPermission(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
229 name="Retire", klass=cn,
4517
f8e85cf5f0fe how odd, the Retire permission wasn't being registered;
Richard Jones <richard@users.sourceforge.net>
parents: 4483
diff changeset
230 description="User is allowed to retire "+cn)
2076
2a4309450202 security fixes and doc updates
Richard Jones <richard@users.sourceforge.net>
parents: 2005
diff changeset
231
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
232 def getclasses(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
233 """Return a list of the names of all existing classes."""
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
234 return sorted(self.classes.keys())
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
235
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
236 def getclass(self, classname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
237 """Get the Class object representing a particular class.
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
238
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
239 If 'classname' is not a valid class name, a KeyError is raised.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
240 """
1145
81941abedb0a nicer error message for invalid class lookup
Richard Jones <richard@users.sourceforge.net>
parents: 1143
diff changeset
241 try:
81941abedb0a nicer error message for invalid class lookup
Richard Jones <richard@users.sourceforge.net>
parents: 1143
diff changeset
242 return self.classes[classname]
81941abedb0a nicer error message for invalid class lookup
Richard Jones <richard@users.sourceforge.net>
parents: 1143
diff changeset
243 except KeyError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
244 raise KeyError('There is no class called "%s"' % classname)
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
245
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
246 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
247 # Class DBs
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
248 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
249 def clear(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
250 """Delete all database contents
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
251 """
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
252 logging.getLogger('roundup.hyperdb.backend').info('clear')
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
253 for cn in self.classes:
7008
f92c56e74f82 in clear() clear journals and node files.
John Rouillard <rouilj@ieee.org>
parents: 7006
diff changeset
254 for data_type in 'nodes', 'journals':
f92c56e74f82 in clear() clear journals and node files.
John Rouillard <rouilj@ieee.org>
parents: 7006
diff changeset
255 path = os.path.join(self.dir, '%s.%s' % (data_type, cn))
443
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
256 if os.path.exists(path):
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
257 os.remove(path)
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
258 elif os.path.exists(path+'.db'): # dbm appends .db
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
259 os.remove(path+'.db')
2736
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
260 # reset id sequences
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
261 path = os.path.join(os.getcwd(), self.dir, '_ids')
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
262 if os.path.exists(path):
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
263 os.remove(path)
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
264 elif os.path.exists(path+'.db'): # dbm appends .db
402d6d556558 postgres backend open doesn't hide corruption in schema [SF#956375]
Richard Jones <richard@users.sourceforge.net>
parents: 2699
diff changeset
265 os.remove(path+'.db')
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
266
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
267 def getclassdb(self, classname, mode='r'):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
268 """ grab a connection to the class db that will be used for
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
269 multiple actions
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
270 """
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
271 return self.opendb('nodes.%s' % classname, mode)
444
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
272
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
273 def determine_db_type(self, path):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
274 """ determine which DB wrote the class file
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
275 """
444
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
276 db_type = ''
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
277 if os.path.exists(path):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
278 db_type = whichdb(path)
444
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
279 if not db_type:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
280 raise hyperdb.DatabaseError(_("Couldn't identify database type"))
444
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
281 elif os.path.exists(path+'.db'):
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
282 # if the path ends in '.db', it's a dbm database, whether
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
283 # anydbm says it's dbhash or not!
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
284 db_type = 'dbm'
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
285 return db_type
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
286
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
287 def opendb(self, name, mode):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
288 """Low-level database opener that gets around anydbm/dbm
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
289 eccentricities.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
290 """
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
291 # figure the class db type
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
292 path = os.path.join(os.getcwd(), self.dir, name)
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
293 db_type = self.determine_db_type(path)
443
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
294
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
295 # new database? let anydbm pick the best dbm
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
296 # in Python 3+ the "dbm" ("anydbm" to us) module already uses the
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
297 # whichdb() function to do this
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
298 if not db_type or hasattr(anydbm, 'whichdb'):
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
299 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
300 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
301 "opendb anydbm.open(%r, 'c')" % path)
1028
16498e77e3ff allow overiding of the index args roundup/cgi/templating.py
Richard Jones <richard@users.sourceforge.net>
parents: 1022
diff changeset
302 return anydbm.open(path, 'c')
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
303
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
304 # in Python <3 it anydbm was a little dumb so manually open the
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
305 # database with the correct module
443
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
306 try:
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
307 dbm = __import__(db_type)
444
3fa2268041a8 Cor blimey this anydbm/whichdb stuff is yecchy.
Richard Jones <richard@users.sourceforge.net>
parents: 443
diff changeset
308 except ImportError:
6209
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
309 if db_type == 'gdbm':
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
310 try:
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
311 dbm = __import__('dbm.gnu')
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
312 except ImportError:
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
313 raise hyperdb.DatabaseError(_(
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
314 "Couldn't open database - the required module '%s' "
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
315 "(as dbm.gnu) is not available") % db_type)
6209
e9d12d516517 If gdbm import fails try python3 fallback
John Rouillard <rouilj@ieee.org>
parents: 6179
diff changeset
316 else:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
317 raise hyperdb.DatabaseError(_(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
318 "Couldn't open database - the "
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
319 "required module '%s' is not available") % db_type)
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
320 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
321 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
322 "opendb %r.open(%r, %r)" % (db_type, path, mode))
443
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
323 return dbm.open(path, mode)
a0c598702f17 I fixed the problems with anydbm using the dbm module at the backend.
Richard Jones <richard@users.sourceforge.net>
parents: 440
diff changeset
324
690
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
325 #
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
326 # Node IDs
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
327 #
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
328 def newid(self, classname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
329 """ Generate a new id for the given class
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
330 """
690
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
331 # open the ids DB - create if if doesn't exist
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
332 db = self.opendb('_ids', 'c')
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
333 if classname in db:
690
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
334 newid = db[classname] = str(int(db[classname]) + 1)
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
335 else:
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
336 # the count() bit is transitional - older dbs won't start at 1
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
337 newid = str(self.getclass(classname).count()+1)
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
338 db[classname] = newid
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
339 db.close()
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
340 return newid
509a101305da node ids are now generated from a lockable store - no more race conditions
Richard Jones <richard@users.sourceforge.net>
parents: 676
diff changeset
341
952
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
342 def setid(self, classname, setid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
343 """ Set the id counter: used during import of database
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
344 """
952
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
345 # open the ids DB - create if if doesn't exist
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
346 db = self.opendb('_ids', 'c')
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
347 db[classname] = str(setid)
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
348 db.close()
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
349
48
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
350 #
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
351 # Nodes
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
352 #
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
353 def addnode(self, classname, nodeid, node):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
354 """ add the specified node to its class's db
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
355 """
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
356 # we'll be supplied these props if we're doing an import
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
357 if 'creator' not in node:
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
358 # add in the "calculated" properties (dupe so we don't affect
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
359 # calling code's node assumptions)
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
360 node = node.copy()
1800
a3b1b1dcf639 Use getuid(), not figure_curuserid()
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 1794
diff changeset
361 node['creator'] = self.getuid()
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
362 node['actor'] = self.getuid()
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
363 node['creation'] = node['activity'] = date.Date()
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
364
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
365 self.newnodes.setdefault(classname, {})[nodeid] = 1
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
366 self.cache.setdefault(classname, {})[nodeid] = node
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
367 self.savenode(classname, nodeid, node)
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
368
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
369 def setnode(self, classname, nodeid, node):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
370 """ change the specified node
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
371 """
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
372 self.dirtynodes.setdefault(classname, {})[nodeid] = 1
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
373
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
374 # can't set without having already loaded the node
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
375 self.cache[classname][nodeid] = node
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
376 self.savenode(classname, nodeid, node)
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
377
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
378 def savenode(self, classname, nodeid, node):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
379 """ perform the saving of data specified by the set/addnode
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
380 """
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
381 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
382 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
383 'save %s%s %r' % (classname, nodeid, node))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
384 self.transactions.append((self.doSaveNode, (classname, nodeid, node)))
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
385
475
a1a44636bace Fix breakage caused by transaction changes.
Richard Jones <richard@users.sourceforge.net>
parents: 464
diff changeset
386 def getnode(self, classname, nodeid, db=None, cache=1):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
387 """ get a node from the database
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
388
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
389 Note the "cache" parameter is not used, and exists purely for
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
390 backward compatibility!
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
391 """
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
392 # try the cache
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
393 cache_dict = self.cache.setdefault(classname, {})
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
394 if nodeid in cache_dict:
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
395 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
396 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
397 'get %s%s cached' % (classname, nodeid))
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
398 self.stats['cache_hits'] += 1
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
399 return cache_dict[nodeid]
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
400
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
401 if __debug__:
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
402 self.stats['cache_misses'] += 1
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
403 start_t = time.time()
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
404 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
405 'get %s%s' % (classname, nodeid))
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
406
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
407 # get from the database and save in the cache
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
408 if db is None:
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
409 db = self.getclassdb(classname)
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
410 if nodeid not in db:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
411 raise IndexError("no such %s %s" % (classname, nodeid))
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
412
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
413 # check the uncommitted, destroyed nodes
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
414 if (classname in self.destroyednodes and
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
415 nodeid in self.destroyednodes[classname]):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
416 raise IndexError("no such %s %s" % (classname, nodeid))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
417
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
418 # decode
48
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
419 res = marshal.loads(db[nodeid])
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
420
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
421 # reverse the serialisation
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
422 res = self.unserialise(classname, res)
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
423
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
424 # store off in the cache dict
475
a1a44636bace Fix breakage caused by transaction changes.
Richard Jones <richard@users.sourceforge.net>
parents: 464
diff changeset
425 if cache:
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
426 cache_dict[nodeid] = res
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
427
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
428 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
429 self.stats['get_items'] += (time.time() - start_t)
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
430
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
431 return res
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
432
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
433 def destroynode(self, classname, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
434 """Remove a node from the database. Called exclusively by the
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
435 destroy() method on Class.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
436 """
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
437 logging.getLogger('roundup.hyperdb.backend').info(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
438 'destroy %s%s' % (classname, nodeid))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
439
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
440 # remove from cache and newnodes if it's there
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
441 if (classname in self.cache and nodeid in self.cache[classname]):
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
442 del self.cache[classname][nodeid]
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
443 if (classname in self.newnodes and nodeid in self.newnodes[classname]):
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
444 del self.newnodes[classname][nodeid]
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
445
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
446 # see if there's any obvious commit actions that we should get rid of
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
447 for entry in self.transactions[:]:
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
448 if entry[1][:2] == (classname, nodeid):
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
449 self.transactions.remove(entry)
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
450
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
451 # add to the destroyednodes map
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
452 self.destroyednodes.setdefault(classname, {})[nodeid] = 1
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
453
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
454 # add the destroy commit action
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
455 self.transactions.append((self.doDestroyNode, (classname, nodeid)))
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
456 self.transactions.append((FileStorage.destroy, (self, classname, nodeid)))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
457
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
458 def serialise(self, classname, node):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
459 """Copy the node contents, converting non-marshallable data into
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
460 marshallable data.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
461 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
462 properties = self.getclass(classname).getprops()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
463 d = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
464 for k, v in node.items():
2964
b71f6d0a463d merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2908
diff changeset
465 if k == self.RETIRED_FLAG:
b71f6d0a463d merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2908
diff changeset
466 d[k] = v
b71f6d0a463d merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2908
diff changeset
467 continue
b71f6d0a463d merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2908
diff changeset
468
b71f6d0a463d merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2908
diff changeset
469 # if the property doesn't exist then we really don't care
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
470 if k not in properties:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
471 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
472
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
473 # get the property spec
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
474 prop = properties[k]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
475
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
476 if isinstance(prop, hyperdb.Password) and v is not None:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
477 d[k] = str(v)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
478 elif isinstance(prop, hyperdb.Date) and v is not None:
964
832d1209aaa2 Preparing to turn back on link/unlink journal events.
Richard Jones <richard@users.sourceforge.net>
parents: 952
diff changeset
479 d[k] = v.serialise()
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
480 elif isinstance(prop, hyperdb.Interval) and v is not None:
964
832d1209aaa2 Preparing to turn back on link/unlink journal events.
Richard Jones <richard@users.sourceforge.net>
parents: 952
diff changeset
481 d[k] = v.serialise()
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
482 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
483 d[k] = v
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
484 return d
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
485
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
486 def unserialise(self, classname, node):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
487 """Decode the marshalled node data
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
488 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
489 properties = self.getclass(classname).getprops()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
490 d = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
491 for k, v in node.items():
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
492 # if the property doesn't exist, or is the "retired" flag then
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
493 # it won't be in the properties dict
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
494 if k not in properties:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
495 d[k] = v
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
496 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
497
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
498 # get the property spec
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
499 prop = properties[k]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
500
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
501 if isinstance(prop, hyperdb.Date) and v is not None:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
502 d[k] = date.Date(v)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
503 elif isinstance(prop, hyperdb.Interval) and v is not None:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
504 d[k] = date.Interval(v)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
505 elif isinstance(prop, hyperdb.Password) and v is not None:
7211
506c86823abb Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents: 7038
diff changeset
506 d[k] = password.Password(encrypted=v, config=self.config)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
507 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
508 d[k] = v
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
509 return d
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
510
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
511 def hasnode(self, classname, nodeid, db=None):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
512 """ determine if the database has a given node
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
513 """
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
514 # try the cache
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
515 cache = self.cache.setdefault(classname, {})
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
516 if nodeid in cache:
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
517 return 1
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
518
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
519 # not in the cache - check the database
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
520 if db is None:
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
521 db = self.getclassdb(classname)
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
522 return nodeid in db
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
523
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
524 def countnodes(self, classname, db=None):
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
525 count = 0
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
526
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
527 # include the uncommitted nodes
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
528 if classname in self.newnodes:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
529 count += len(self.newnodes[classname])
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
530 if classname in self.destroyednodes:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
531 count -= len(self.destroyednodes[classname])
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
532
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
533 # and count those in the DB
452
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
534 if db is None:
7181efddce66 yuck, a gdbm instance tests false :(
Richard Jones <richard@users.sourceforge.net>
parents: 444
diff changeset
535 db = self.getclassdb(classname)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
536 return count + len(db)
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
537
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
538 #
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
539 # Files - special node properties
646
07abfe8f0c01 use blobfiles in back_anydbm which is used in back_bsddb.
Engelbert Gruber <grubert@users.sourceforge.net>
parents: 624
diff changeset
540 # inherited from FileStorage
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
541
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
542 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
543 # Journal
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
544 #
1143
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
545 def addjournal(self, classname, nodeid, action, params, creator=None,
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
546 creation=None):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
547 """ Journal the Action
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
548 'action' may be:
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
549
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
550 'set' -- 'params' is a dictionary of property values
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
551 'create' -- 'params' is an empty dictionary as of
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
552 Wed Nov 06 11:38:43 2002 +0000
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
553 'link' or 'unlink' -- 'params' is (classname, nodeid, propname)
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
554 'retired' or 'restored' -- 'params' is None
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
555
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
556 'creator' -- the user performing the action, which defaults to
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
557 the current user.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
558 """
719
fed4c363a7f3 node caching now works, and gives a small boost in performance
Richard Jones <richard@users.sourceforge.net>
parents: 697
diff changeset
559 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
560 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
561 'addjournal %s%s %s %r %s %r' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
562 classname, nodeid, action, params, creator, creation))
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
563 if creator is None:
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
564 creator = self.getuid()
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
565 self.transactions.append((self.doSaveJournal, (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
566 classname, nodeid, action, params, creator, creation)))
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
567
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
568 def setjournal(self, classname, nodeid, journal):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
569 """Set the journal to the "journal" list."""
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
570 if __debug__:
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
571 logging.getLogger('roundup.hyperdb.backend').debug(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
572 'setjournal %s%s %r' % (classname, nodeid, journal))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
573 self.transactions.append((self.doSetJournal, (classname, nodeid,
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
574 journal)))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
575
4534
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
576 def fix_journal(self, classname, journal):
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
577 """ fix password entries to correct type """
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
578 pwprops = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
579 for pn, prop in self.getclass(classname).properties.items():
4534
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
580 if isinstance(prop, hyperdb.Password):
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
581 pwprops[pn] = 1
4534
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
582 if not pwprops:
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
583 return journal
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
584 for j in journal:
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
585 if j[3] == 'set':
7038
f524ddc27af8 replace for a,b in x.items() with for a in x.keys()
John Rouillard <rouilj@ieee.org>
parents: 7012
diff changeset
586 for k in j[4].keys():
4538
fd972e18b21a - fix import/export regression test for anydbm for latest journal fix
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4534
diff changeset
587 if k in pwprops and j[4][k]:
4534
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
588 j[4][k] = password.JournalPassword(j[4][k])
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
589 return journal
0dd6bdeb2eb5 issue2550729: Fix password history display for anydbm backend...
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4518
diff changeset
590
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
591 def getjournal(self, classname, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
592 """ get the journal for id
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
593
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
594 Raise IndexError if the node doesn't exist (as per history()'s
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
595 API)
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
596 """
1567
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
597 # our journal result
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
598 res = []
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
599
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
600 # add any journal entries for transactions not committed to the
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
601 # database
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
602 for method, args in self.transactions:
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
603 if method != self.doSaveJournal:
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
604 continue
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
605 (cache_classname, cache_nodeid, cache_action, cache_params,
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
606 cache_creator, cache_creation) = args
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
607 if cache_classname == classname and cache_nodeid == nodeid:
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
608 if not cache_creator:
1800
a3b1b1dcf639 Use getuid(), not figure_curuserid()
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 1794
diff changeset
609 cache_creator = self.getuid()
1567
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
610 if not cache_creation:
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
611 cache_creation = date.Date()
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
612 res.append((cache_nodeid, cache_creation, cache_creator,
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
613 cache_action, cache_params))
1567
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
614
48
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
615 # attempt to open the journal - in some rare cases, the journal may
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
616 # not exist
671203878652 Moved over to using marshal in the bsddb and anydbm backends.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
617 try:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
618 db = self.opendb('journals.%s' % classname, 'r')
5248
198b6e810c67 Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents: 5232
diff changeset
619 except anydbm.error as error:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
620 if str(error) == "need 'c' or 'n' flag to open new db":
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
621 raise IndexError('no such %s %s' % (classname, nodeid))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
622 elif error.args[0] != 2:
1920
f9316d2cd5ba Fixed retirement of items in rdbms imports [SF#841355]
Richard Jones <richard@users.sourceforge.net>
parents: 1904
diff changeset
623 # this isn't a "not found" error, be alarmed!
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
624 raise
1920
f9316d2cd5ba Fixed retirement of items in rdbms imports [SF#841355]
Richard Jones <richard@users.sourceforge.net>
parents: 1904
diff changeset
625 if res:
f9316d2cd5ba Fixed retirement of items in rdbms imports [SF#841355]
Richard Jones <richard@users.sourceforge.net>
parents: 1904
diff changeset
626 # we have unsaved journal entries, return them
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
627 return self.fix_journal(classname, res)
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
628 raise IndexError('no such %s %s' % (classname, nodeid))
787
b6b0a92e0738 More informative error message
Richard Jones <richard@users.sourceforge.net>
parents: 778
diff changeset
629 try:
b6b0a92e0738 More informative error message
Richard Jones <richard@users.sourceforge.net>
parents: 778
diff changeset
630 journal = marshal.loads(db[nodeid])
b6b0a92e0738 More informative error message
Richard Jones <richard@users.sourceforge.net>
parents: 778
diff changeset
631 except KeyError:
843
01f9d02faea1 ...except of course it's nice to use valid Python syntax
Richard Jones <richard@users.sourceforge.net>
parents: 842
diff changeset
632 db.close()
1567
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
633 if res:
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
634 # we have some unsaved journal entries, be happy!
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
635 return self.fix_journal(classname, res)
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
636 raise IndexError('no such %s %s' % (classname, nodeid))
843
01f9d02faea1 ...except of course it's nice to use valid Python syntax
Richard Jones <richard@users.sourceforge.net>
parents: 842
diff changeset
637 db.close()
1567
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
638
fc8998ce6274 fixed missing (pre-commit) journal entries in *dbm backends [SF#679217]
Richard Jones <richard@users.sourceforge.net>
parents: 1563
diff changeset
639 # add all the saved journal entries for this node
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
640 for nodeid, date_stamp, user, action, params in journal:
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
641 res.append((nodeid, date.Date(date_stamp), user, action, params))
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
642 return self.fix_journal(classname, res)
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
643
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
644 def pack(self, pack_before):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
645 """ Delete all journal entries except "create" before 'pack_before'.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
646 """
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
647 pack_before = pack_before.serialise()
990
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
648 for classname in self.getclasses():
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
649 packed = 0
990
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
650 # get the journal db
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
651 db_name = 'journals.%s' % classname
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
652 path = os.path.join(os.getcwd(), self.dir, classname)
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
653 db_type = self.determine_db_type(path)
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
654 db = self.opendb(db_name, 'w')
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
655
5444
167f0d25ea8e Python 3 preparation: convert dbm keys back from bytes to strings.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5395
diff changeset
656 for key in map(b2s, db.keys()):
990
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
657 # get the journal for this db entry
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
658 journal = marshal.loads(db[key])
7010
1bb7aa8414b8 change variable name l -> kept_entries
John Rouillard <rouilj@ieee.org>
parents: 7009
diff changeset
659 kept_entries = []
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
660 for entry in journal:
990
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
661 # unpack the entry
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
662 (nodeid, date_stamp, self.journaltag, action,
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
663 params) = entry
990
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
664 # if the entry is after the pack date, _or_ the initial
d374545c8eb0 minor edits
Richard Jones <richard@users.sourceforge.net>
parents: 969
diff changeset
665 # create entry, then it stays
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
666 if date_stamp > pack_before or action == 'create':
7010
1bb7aa8414b8 change variable name l -> kept_entries
John Rouillard <rouilj@ieee.org>
parents: 7009
diff changeset
667 kept_entries.append(entry)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
668 else:
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
669 packed += 1
7010
1bb7aa8414b8 change variable name l -> kept_entries
John Rouillard <rouilj@ieee.org>
parents: 7009
diff changeset
670 db[key] = marshal.dumps(kept_entries)
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
671
5232
462b0f76fce8 issue2550864 - Potential information leakage via journal/history
John Rouillard <rouilj@ieee.org>
parents: 5112
diff changeset
672 logging.getLogger('roundup.hyperdb.backend').info(
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
673 'packed %d %s items' % (packed, classname))
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
674
562
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
675 if db_type == 'gdbm':
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
676 db.reorganize()
62febbd7ffec You can now use the roundup-admin tool to pack the database
Roche Compaan <rochecompaan@users.sourceforge.net>
parents: 553
diff changeset
677 db.close()
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
678
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
679 #
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
680 # Basic transaction support
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
681 #
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5318
diff changeset
682 def commit(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
683 """ Commit the current transactions.
3687
ff9f4ca42454 Postgres backend allows transaction collisions to be ignored when...
Richard Jones <richard@users.sourceforge.net>
parents: 3682
diff changeset
684
ff9f4ca42454 Postgres backend allows transaction collisions to be ignored when...
Richard Jones <richard@users.sourceforge.net>
parents: 3682
diff changeset
685 Save all data changed since the database was opened or since the
ff9f4ca42454 Postgres backend allows transaction collisions to be ignored when...
Richard Jones <richard@users.sourceforge.net>
parents: 3682
diff changeset
686 last commit() or rollback().
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
687 """
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
688 logging.getLogger('roundup.hyperdb.backend').info(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
689 'commit %s transactions' % (len(self.transactions)))
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
690
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
691 # keep a handle to all the database files opened
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
692 self.databases = {}
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
693
1904
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
694 try:
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
695 # now, do all the transactions
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
696 reindex = {}
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
697 for method, args in self.transactions:
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
698 reindex[method(*args)] = 1
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
699 finally:
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
700 # make sure we close all the database files
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
701 for db in self.databases.values():
1904
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
702 db.close()
9445caec3ff5 more database closing cleanups, finally mysql has no dangling references
Richard Jones <richard@users.sourceforge.net>
parents: 1840
diff changeset
703 del self.databases
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
704
3960
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3955
diff changeset
705 # clear the transactions list now so the blobfile implementation
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3955
diff changeset
706 # doesn't think there's still pending file commits when it tries
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3955
diff changeset
707 # to access the file data
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3955
diff changeset
708 self.transactions = []
bc412bb2ccd3 Workaround race condition in file storage during transaction commit [SF#883580]
Richard Jones <richard@users.sourceforge.net>
parents: 3955
diff changeset
709
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
710 # reindex the nodes that request it
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
711 for classname, nodeid in [k for k in reindex if k]:
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
712 self.getclass(classname).index(nodeid)
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
713
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
714 # save the indexer state
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
715 self.indexer.save_index()
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
716
1181
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
717 self.clearCache()
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
718
49aebf5a8691 some speedups, some fixes to the benchmarking
Richard Jones <richard@users.sourceforge.net>
parents: 1176
diff changeset
719 def clearCache(self):
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
720 # all transactions committed, back to normal
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
721 self.cache = {}
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
722 self.dirtynodes = {}
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
723 self.newnodes = {}
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
724 self.destroyednodes = {}
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
725 self.transactions = []
4652
dfbc0cfa9811 Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4538
diff changeset
726 # upcall is necessary!
dfbc0cfa9811 Add an interface to register clearCache callbacks in roundupdb.
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4538
diff changeset
727 roundupdb.Database.clearCache(self)
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
728
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
729 def getCachedClassDB(self, classname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
730 """ get the class db, looking in our cache of databases for commit
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
731 """
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
732 # get the database handle
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
733 db_name = 'nodes.%s' % classname
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
734 if db_name not in self.databases:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
735 self.databases[db_name] = self.getclassdb(classname, 'c')
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
736 return self.databases[db_name]
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
737
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
738 def doSaveNode(self, classname, nodeid, node):
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
739 db = self.getCachedClassDB(classname)
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
740
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
741 # now save the marshalled data
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
742 db[nodeid] = marshal.dumps(self.serialise(classname, node))
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
743
825
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
744 # return the classname, nodeid so we reindex this content
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
745 return (classname, nodeid)
0779ea9f1f18 More indexer work:
Richard Jones <richard@users.sourceforge.net>
parents: 818
diff changeset
746
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
747 def getCachedJournalDB(self, classname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
748 """ get the journal db, looking in our cache of databases for commit
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
749 """
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
750 # get the database handle
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
751 db_name = 'journals.%s' % classname
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
752 if db_name not in self.databases:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
753 self.databases[db_name] = self.opendb(db_name, 'c')
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
754 return self.databases[db_name]
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
755
1143
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
756 def doSaveJournal(self, classname, nodeid, action, params, creator,
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
757 creation):
1143
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
758 # serialise the parameters now if necessary
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
759 if isinstance(params, type({})):
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
760 if action in ('set', 'create'):
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
761 params = self.serialise(classname, params)
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
762
952
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
763 # handle supply of the special journalling parameters (usually
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
764 # supplied on importing an existing database)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
765 journaltag = creator
1143
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
766 if creation:
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
767 journaldate = creation.serialise()
Richard Jones <richard@users.sourceforge.net>
parents: 1131
diff changeset
768 else:
964
832d1209aaa2 Preparing to turn back on link/unlink journal events.
Richard Jones <richard@users.sourceforge.net>
parents: 952
diff changeset
769 journaldate = date.Date().serialise()
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
770
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
771 # create the journal entry
952
f615fbd02c18 full database export and import is done
Richard Jones <richard@users.sourceforge.net>
parents: 950
diff changeset
772 entry = (nodeid, journaldate, journaltag, action, params)
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
773
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
774 db = self.getCachedJournalDB(classname)
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
775
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
776 # now insert the journal entry
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
777 if nodeid in db:
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
778 # append to existing
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
779 s = db[nodeid]
650
9b2575610953 Ran it through pychecker, made fixes
Richard Jones <richard@users.sourceforge.net>
parents: 646
diff changeset
780 l = marshal.loads(s)
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
781 l.append(entry)
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
782 else:
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
783 l = [entry]
676
bc46480e2a2b Fixed serialisation problem...
Richard Jones <richard@users.sourceforge.net>
parents: 650
diff changeset
784
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
785 db[nodeid] = marshal.dumps(l)
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
786
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
787 def doSetJournal(self, classname, nodeid, journal):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
788 l = []
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
789 for nodeid, journaldate, journaltag, action, params in journal:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
790 # serialise the parameters now if necessary
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
791 if isinstance(params, type({})):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
792 if action in ('set', 'create'):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
793 params = self.serialise(classname, params)
2512
f5542d92307a fix anydbm journal import [SF#983166]
Richard Jones <richard@users.sourceforge.net>
parents: 2505
diff changeset
794 journaldate = journaldate.serialise()
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
795 l.append((nodeid, journaldate, journaltag, action, params))
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
796 db = self.getCachedJournalDB(classname)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
797 db[nodeid] = marshal.dumps(l)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
798
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
799 def doDestroyNode(self, classname, nodeid):
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
800 # delete from the class database
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
801 db = self.getCachedClassDB(classname)
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
802 if nodeid in db:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
803 del db[nodeid]
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
804
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
805 # delete from the database
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
806 db = self.getCachedJournalDB(classname)
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
807 if nodeid in db:
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
808 del db[nodeid]
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
809
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
810 def rollback(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
811 """ Reverse all actions from the current transaction.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
812 """
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
813 logging.getLogger('roundup.hyperdb.backend').info(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
814 'rollback %s transactions' % (len(self.transactions)))
2514
091711fb2f8c Initial logging integration: replace all debug prints with logging calls...
Richard Jones <richard@users.sourceforge.net>
parents: 2512
diff changeset
815
461
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
816 for method, args in self.transactions:
b579418f7ed1 Implemented file store rollback.
Richard Jones <richard@users.sourceforge.net>
parents: 460
diff changeset
817 # delete temporary files
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
818 if method == self.doStoreFile:
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
819 self.rollbackStoreFile(*args)
430
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
820 self.cache = {}
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
821 self.dirtynodes = {}
350685601f37 Database transactions.
Richard Jones <richard@users.sourceforge.net>
parents: 394
diff changeset
822 self.newnodes = {}
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
823 self.destroyednodes = {}
252
76c6994aa4e8 CGI interfaces now spit up a top-level index of all instances they can serve.
Richard Jones <richard@users.sourceforge.net>
parents: 224
diff changeset
824 self.transactions = []
46
3c5920433866 *sigh* some databases have _foo.so as their underlying implementation.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
825
1131
92e92ae58494 add close() methods where they are missing!
Richard Jones <richard@users.sourceforge.net>
parents: 1127
diff changeset
826 def close(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
827 """ Nothing to do
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
828 """
1333
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
829 if self.lockfile is not None:
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
830 locking.release_lock(self.lockfile)
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
831 self.lockfile.close()
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
832 self.lockfile = None
1131
92e92ae58494 add close() methods where they are missing!
Richard Jones <richard@users.sourceforge.net>
parents: 1127
diff changeset
833
7002
1c56bd744be4 More whiespace fixes
John Rouillard <rouilj@ieee.org>
parents: 7001
diff changeset
834
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
835 class Class(hyperdb.Class):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
836 """The handle to a particular class of nodes in a hyperdatabase."""
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
837
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
838 def enableJournalling(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
839 """Turn journalling on for this class
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
840 """
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
841 self.do_journal = 1
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
842
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
843 def disableJournalling(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
844 """Turn journalling off for this class
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
845 """
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
846 self.do_journal = 0
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
847
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
848 # Editing nodes:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
849
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
850 def create(self, **propvalues):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
851 """Create a new node of this class and return its id.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
852
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
853 The keyword arguments in 'propvalues' map property names to values.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
854
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
855 The values of arguments must be acceptable for the types of their
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
856 corresponding properties or a TypeError is raised.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
857
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
858 If this class has a key property, it must be present and its value
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
859 must not collide with other key strings or a ValueError is raised.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
860
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
861 Any other properties on this class that are missing from the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
862 'propvalues' dictionary are set to None.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
863
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
864 If an id in a link or multilink property does not refer to a valid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
865 node, an IndexError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
866
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
867 These operations trigger detectors and can be vetoed. Attempts
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
868 to modify the "creation" or "activity" properties cause a KeyError.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
869 """
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4342
diff changeset
870 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
871 raise hyperdb.DatabaseError(_('Database open read-only'))
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
872 self.fireAuditors('create', None, propvalues)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
873 newid = self.create_inner(**propvalues)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
874 self.fireReactors('create', newid, None)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
875 return newid
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
876
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
877 def create_inner(self, **propvalues):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
878 """ Called by create, in-between the audit and react calls.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
879 """
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
880 if 'id' in propvalues:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
881 raise KeyError('"id" is reserved')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
882
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
883 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
884 raise hyperdb.DatabaseError(_('Database open read-only'))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
885
7002
1c56bd744be4 More whiespace fixes
John Rouillard <rouilj@ieee.org>
parents: 7001
diff changeset
886 if ('creator' in propvalues or 'actor' in propvalues or
1c56bd744be4 More whiespace fixes
John Rouillard <rouilj@ieee.org>
parents: 7001
diff changeset
887 'creation' in propvalues or 'activity' in propvalues):
5704
aa13a517cc63 Anydbm back end didn't list creator or actor as protected properties.
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
888 raise KeyError('"creator", "actor", "creation" and '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
889 '"activity" are reserved')
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
890
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
891 for p in propvalues:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
892 prop = self.properties[p]
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
893 if prop.computed:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
894 raise KeyError('"%s" is a computed property' % p)
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
895
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
896 # new node's id
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
897 newid = self.db.newid(self.classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
898
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
899 # validate propvalues
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
900 num_re = re.compile(r'^\d+$')
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
901 for key, value in propvalues.items():
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
902 if key == self.key:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
903 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
904 self.lookup(value)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
905 except KeyError:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
906 pass
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
907 else:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
908 raise ValueError('node with key "%s" exists' % value)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
909
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
910 # try to handle this property
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
911 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
912 prop = self.properties[key]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
913 except KeyError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
914 raise KeyError('"%s" has no property "%s"' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
915 self.classname, key))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
916
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
917 if value is not None and isinstance(prop, hyperdb.Link):
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
918 if not isinstance(value, str):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
919 raise ValueError('link value must be String')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
920 link_class = self.properties[key].classname
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
921 # if it isn't a number, it's a key
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
922 if not num_re.match(value):
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
923 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
924 value = self.db.classes[link_class].lookup(value)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
925 except (TypeError, KeyError):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
926 raise IndexError('new property "%s": %s not a %s' % (
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
927 key, value, link_class))
902
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
928 elif not self.db.getclass(link_class).hasnode(value):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
929 raise IndexError('%s has no node %s' % (link_class, value))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
930
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
931 # save off the value
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
932 propvalues[key] = value
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
933
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
934 # register the link with the newly linked node
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
935 if self.do_journal and self.properties[key].do_journal:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
936 self.db.addjournal(link_class, value, 'link',
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
937 (self.classname, newid, key))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
938
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
939 elif isinstance(prop, hyperdb.Multilink):
3872
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
940 if value is None:
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
941 value = []
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
942 if not hasattr(value, '__iter__') or isinstance(value, str):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
943 raise TypeError(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
944 'new property "%s" not an iterable of ids' % key)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
945
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
946 # clean up and validate the list of links
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
947 link_class = self.properties[key].classname
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
948 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
949 for entry in value:
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
950 if not isinstance(entry, str):
7004
d3d099432ba2 Remove uneeded \
John Rouillard <rouilj@ieee.org>
parents: 7003
diff changeset
951 raise ValueError('"%s" multilink value (%r) '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
952 'must contain Strings' % (key, value))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
953 # if it isn't a number, it's a key
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
954 if not num_re.match(entry):
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
955 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
956 entry = self.db.classes[link_class].lookup(entry)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
957 except (TypeError, KeyError):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
958 raise IndexError(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
959 'new property "%s": %s not a %s' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
960 key, entry,
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
961 self.properties[key].classname))
6999
e5a28b8ded72 backout checkin 60b37f632601
John Rouillard <rouilj@ieee.org>
parents: 6998
diff changeset
962 l.append(entry)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
963 value = l
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
964 propvalues[key] = value
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
965
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
966 # handle additions
902
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
967 for nodeid in value:
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
968 if not self.db.getclass(link_class).hasnode(nodeid):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
969 raise IndexError('%s has no node %s' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
970 link_class, nodeid))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
971 # register the link with the newly linked node
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
972 if self.do_journal and self.properties[key].do_journal:
902
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
973 self.db.addjournal(link_class, nodeid, 'link',
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
974 (self.classname, newid, key))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
975
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
976 elif isinstance(prop, hyperdb.String):
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
977 if not isinstance(value, (str, unicode)):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
978 raise TypeError('new property "%s" not a string' % key)
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2736
diff changeset
979 if prop.indexme:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
980 self.db.indexer.add_text(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
981 (self.classname, newid, key), value)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
982
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
983 elif isinstance(prop, hyperdb.Password):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
984 if not isinstance(value, password.Password):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
985 raise TypeError('new property "%s" not a Password' % key)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
986
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
987 elif isinstance(prop, hyperdb.Date):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
988 if value is not None and not isinstance(value, date.Date):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
989 raise TypeError('new property "%s" not a Date' % key)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
990
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
991 elif isinstance(prop, hyperdb.Interval):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
992 if value is not None and not isinstance(value, date.Interval):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
993 raise TypeError('new property "%s" not an Interval' % key)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
994
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
995 elif value is not None and isinstance(prop, hyperdb.Number):
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
996 try:
890
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
997 float(value)
887
e7169d6e6e45 added tests for number type too
Richard Jones <richard@users.sourceforge.net>
parents: 886
diff changeset
998 except ValueError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
999 raise TypeError('new property "%s" not numeric' % key)
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
1000
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1001 elif value is not None and isinstance(prop, hyperdb.Integer):
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1002 try:
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1003 int(value)
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1004 except ValueError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1005 raise TypeError('new property "%s" not an integer' % key)
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1006
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1007 elif value is not None and isinstance(prop, hyperdb.Boolean):
890
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1008 try:
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1009 int(value)
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1010 except ValueError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1011 raise TypeError('new property "%s" not boolean' % key)
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
1012
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1013 # make sure there's data where there needs to be
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
1014 for key, prop in self.properties.items():
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1015 if key in propvalues:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1016 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1017 if key == self.key:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1018 raise ValueError('key property "%s" is required' % key)
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1019 if isinstance(prop, hyperdb.Multilink):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1020 propvalues[key] = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1021
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1022 # done
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1023 self.db.addnode(self.classname, newid, propvalues)
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1024 if self.do_journal:
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1303
diff changeset
1025 self.db.addjournal(self.classname, newid, 'create', {})
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1026
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1027 return newid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1028
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1029 def get(self, nodeid, propname, default=_marker, cache=1):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1030 """Get the value of a property on an existing node of this class.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1031
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1032 'nodeid' must be the id of an existing node of this class or an
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1033 IndexError is raised. 'propname' must be the name of a property
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1034 of this class or a KeyError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1035
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
1036 'cache' exists for backward compatibility, and is not used.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1037
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1038 Attempts to get the "creation" or "activity" properties should
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1039 do the right thing.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1040 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1041 if propname == 'id':
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1042 return nodeid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1043
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1044 # get the node's dict
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
1045 d = self.db.getnode(self.classname, nodeid)
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1046
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1047 # check for one of the special props
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1048 if propname == 'creation':
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1049 if 'creation' in d:
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1050 return d['creation']
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1051 if not self.do_journal:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1052 raise ValueError('Journalling is disabled for this class')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1053 journal = self.db.getjournal(self.classname, nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1054 if journal:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents: 4075
diff changeset
1055 return journal[0][1]
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1056 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1057 # on the strange chance that there's no journal
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1058 return date.Date()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1059 if propname == 'activity':
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1060 if 'activity' in d:
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1061 return d['activity']
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1062 if not self.do_journal:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1063 raise ValueError('Journalling is disabled for this class')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1064 journal = self.db.getjournal(self.classname, nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1065 if journal:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1066 return self.db.getjournal(self.classname, nodeid)[-1][1]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1067 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1068 # on the strange chance that there's no journal
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1069 return date.Date()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1070 if propname == 'creator':
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1071 if 'creator' in d:
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1072 return d['creator']
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1073 if not self.do_journal:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1074 raise ValueError('Journalling is disabled for this class')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1075 journal = self.db.getjournal(self.classname, nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1076 if journal:
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
1077 num_re = re.compile(r'^\d+$')
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1078 value = journal[0][2]
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1079 if num_re.match(value):
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1080 return value
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1081 else:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1082 # old-style "username" journal tag
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1083 try:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1084 return self.db.user.lookup(value)
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1085 except KeyError:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1086 # user's been retired, return admin
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1087 return '1'
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1088 else:
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1089 return self.db.getuid()
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1090 if propname == 'actor':
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1091 if 'actor' in d:
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1092 return d['actor']
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1093 if not self.do_journal:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1094 raise ValueError('Journalling is disabled for this class')
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1095 journal = self.db.getjournal(self.classname, nodeid)
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1096 if journal:
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
1097 num_re = re.compile(r'^\d+$')
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
1098 value = journal[-1][2]
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1099 if num_re.match(value):
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1100 return value
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1101 else:
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1102 # old-style "username" journal tag
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1103 try:
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1104 return self.db.user.lookup(value)
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1105 except KeyError:
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1106 # user's been retired, return admin
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
1107 return '1'
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1108 else:
1800
a3b1b1dcf639 Use getuid(), not figure_curuserid()
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 1794
diff changeset
1109 return self.db.getuid()
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1110
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1111 # get the property (raises KeyErorr if invalid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1112 prop = self.properties[propname]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1113
6179
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
1114 if isinstance(prop, hyperdb.Multilink) and prop.computed:
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
1115 cls = self.db.getclass(prop.rev_classname)
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
1116 ids = cls.find(**{prop.rev_propname: nodeid})
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
1117 return ids
a701c9c81597 Fix rev_multilink properties search/retrieval
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6151
diff changeset
1118
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1119 if propname not in d:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1120 if default is _marker:
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1121 if isinstance(prop, hyperdb.Multilink):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1122 return []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1123 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1124 return None
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1125 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1126 return default
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1127
1014
8816534e6a1a Fixed nasty bug that was preventing changes to multilinks going through.
Richard Jones <richard@users.sourceforge.net>
parents: 1002
diff changeset
1128 # return a dupe of the list so code doesn't get confused
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1129 if isinstance(prop, hyperdb.Multilink):
6351
0db59cc2cd37 Enable testMultilinkOrdering check. Fix back_anydbm to pass.
John Rouillard <rouilj@ieee.org>
parents: 6238
diff changeset
1130 ids = d[propname][:]
0db59cc2cd37 Enable testMultilinkOrdering check. Fix back_anydbm to pass.
John Rouillard <rouilj@ieee.org>
parents: 6238
diff changeset
1131 ids.sort(key=lambda x: int(x))
0db59cc2cd37 Enable testMultilinkOrdering check. Fix back_anydbm to pass.
John Rouillard <rouilj@ieee.org>
parents: 6238
diff changeset
1132 return ids
1014
8816534e6a1a Fixed nasty bug that was preventing changes to multilinks going through.
Richard Jones <richard@users.sourceforge.net>
parents: 1002
diff changeset
1133
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1134 return d[propname]
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1135
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1136 def set(self, nodeid, **propvalues):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1137 """Modify a property on an existing node of this class.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1138
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1139 'nodeid' must be the id of an existing node of this class or an
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1140 IndexError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1141
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1142 Each key in 'propvalues' must be the name of a property of this
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1143 class or a KeyError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1144
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1145 All values in 'propvalues' must be acceptable types for their
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1146 corresponding properties or a TypeError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1147
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1148 If the value of the key property is set, it must not collide with
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1149 other key strings or a ValueError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1150
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1151 If the value of a Link or Multilink property contains an invalid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1152 node id, a ValueError is raised.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1153
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1154 These operations trigger detectors and can be vetoed. Attempts
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1155 to modify the "creation" or "activity" properties cause a KeyError.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1156 """
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4342
diff changeset
1157 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1158 raise hyperdb.DatabaseError(_('Database open read-only'))
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4342
diff changeset
1159
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1160 self.fireAuditors('set', nodeid, propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1161 oldvalues = copy.deepcopy(self.db.getnode(self.classname, nodeid))
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
1162 for name, prop in self.getprops(protected=0).items():
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1163 if name in oldvalues:
2608
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
1164 continue
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1165 if isinstance(prop, hyperdb.Multilink):
2608
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
1166 oldvalues[name] = []
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
1167 else:
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
1168 oldvalues[name] = None
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1169 propvalues = self.set_inner(nodeid, **propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1170 self.fireReactors('set', nodeid, oldvalues)
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1171 return propvalues
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1172
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1173 def set_inner(self, nodeid, **propvalues):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1174 """ Called by set, in-between the audit and react calls.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1175 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1176 if not propvalues:
930
3c344e942055 Use same regex to split search terms as used to index text.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 927
diff changeset
1177 return propvalues
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1178
7002
1c56bd744be4 More whiespace fixes
John Rouillard <rouilj@ieee.org>
parents: 7001
diff changeset
1179 if ('creator' in propvalues or 'actor' in propvalues or
1c56bd744be4 More whiespace fixes
John Rouillard <rouilj@ieee.org>
parents: 7001
diff changeset
1180 'creation' in propvalues or 'activity' in propvalues):
5704
aa13a517cc63 Anydbm back end didn't list creator or actor as protected properties.
John Rouillard <rouilj@ieee.org>
parents: 5544
diff changeset
1181 raise KeyError('"creator", "actor", "creation" and '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1182 '"activity" are reserved')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1183
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1184 if 'id' in propvalues:
5378
35ea9b1efc14 Python 3 preparation: "raise" syntax.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5319
diff changeset
1185 raise KeyError('"id" is reserved')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1186
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1187 for p in propvalues:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1188 prop = self.properties[p]
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1189 if prop.computed:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1190 raise KeyError('"%s" is a computed property' % p)
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1191
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1192 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1193 raise hyperdb.DatabaseError(_('Database open read-only'))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1194
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1195 node = self.db.getnode(self.classname, nodeid)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1196 if self.db.RETIRED_FLAG in node:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1197 raise IndexError
5809
936275dfe1fa Try to fix:
John Rouillard <rouilj@ieee.org>
parents: 5725
diff changeset
1198 num_re = re.compile(r'^\d+$')
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1199
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1200 # if the journal value is to be different, store it in here
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1201 journalvalues = {}
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1202
5112
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1203 # omit quiet properties from history/changelog
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1204 quiet_props = []
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1205
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1206 # list() propvalues 'cos it might be modified by the loop
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1207 for propname, value in list(propvalues.items()):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1208 # check to make sure we're not duplicating an existing key
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1209 if propname == self.key and node[propname] != value:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1210 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1211 self.lookup(value)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1212 except KeyError:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1213 pass
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1214 else:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1215 raise ValueError('node with key "%s" exists' % value)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1216
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1217 # this will raise the KeyError if the property isn't valid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1218 # ... we don't use getprops() here because we only care about
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1219 # the writeable properties.
1174
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1220 try:
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1221 prop = self.properties[propname]
8e318dfaf479 Verify contents of tracker module when the tracker is opened
Richard Jones <richard@users.sourceforge.net>
parents: 1170
diff changeset
1222 except KeyError:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1223 raise KeyError('"%s" has no property named "%s"' % (
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1224 self.classname, propname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1225
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1226 # if the value's the same as the existing value, no sense in
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1227 # doing anything
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1303
diff changeset
1228 current = node.get(propname, None)
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1303
diff changeset
1229 if value == current:
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1230 del propvalues[propname]
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1231 continue
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1303
diff changeset
1232 journalvalues[propname] = current
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1233
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1234 # do stuff based on the prop type
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1235 if isinstance(prop, hyperdb.Link):
927
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1236 link_class = prop.classname
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1237 # if it isn't a number, it's a key
927
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1238 if value is not None and not isinstance(value, type('')):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1239 raise ValueError('property "%s" link value be a string' % (
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1240 propname))
927
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1241 if isinstance(value, type('')) and not num_re.match(value):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1242 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1243 value = self.db.classes[link_class].lookup(value)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1244 except (TypeError, KeyError):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1245 raise IndexError('new property "%s": %s not a %s' % (
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1246 propname, value, prop.classname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1247
927
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1248 if (value is not None and
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1249 not self.db.getclass(link_class).hasnode(value)):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1250 raise IndexError('%s has no node %s' % (link_class,
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1251 value))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1252
927
51519406b73e web forms may now unset Link values (like assignedto)
Richard Jones <richard@users.sourceforge.net>
parents: 924
diff changeset
1253 if self.do_journal and prop.do_journal:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1254 # register the unlink with the old linked node
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1255 if propname in node and node[propname] is not None:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1256 self.db.addjournal(link_class, node[propname],
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1257 'unlink',
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1258 (self.classname, nodeid, propname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1259
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1260 # register the link with the newly linked node
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1261 if value is not None:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1262 self.db.addjournal(link_class, value, 'link',
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1263 (self.classname, nodeid, propname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1264
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1265 elif isinstance(prop, hyperdb.Multilink):
3872
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
1266 if value is None:
34128a809e22 Allow multilinks to take None (treated as an empty list).
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3869
diff changeset
1267 value = []
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
1268 if not hasattr(value, '__iter__') or isinstance(value, str):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1269 raise TypeError('new property "%s" not an iterable of'
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1270 ' ids' % propname)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1271 link_class = self.properties[propname].classname
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1272 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1273 for entry in value:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1274 # if it isn't a number, it's a key
7011
4d2625f10314 flake8 use isinstance rather than comparing types.
John Rouillard <rouilj@ieee.org>
parents: 7010
diff changeset
1275 if not isinstance(entry, str):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1276 raise ValueError('new property "%s" link value '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1277 'must be a string' % propname)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1278 if not num_re.match(entry):
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1279 try:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1280 entry = self.db.classes[link_class].lookup(entry)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1281 except (TypeError, KeyError):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1282 raise IndexError(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1283 'new property "%s": %s not a %s' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1284 propname, entry,
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1285 self.properties[propname].classname))
6999
e5a28b8ded72 backout checkin 60b37f632601
John Rouillard <rouilj@ieee.org>
parents: 6998
diff changeset
1286 l.append(entry)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1287 value = l
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1288 propvalues[propname] = value
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1289
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1290 # figure the journal entry for this property
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1291 add = []
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1292 remove = []
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1293
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1294 # handle removals
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1295 if propname in node:
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1296 l = node[propname]
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1297 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1298 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1299 for id in l[:]:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1300 if id in value:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1301 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1302 # register the unlink with the old linked node
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1303 if self.do_journal and self.properties[propname].do_journal:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1304 self.db.addjournal(link_class, id, 'unlink',
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1305 (self.classname, nodeid, propname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1306 l.remove(id)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1307 remove.append(id)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1308
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1309 # handle additions
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1310 for id in value:
902
b0d3d3535998 Bugger it. Here's the current shape of the new security implementation.
Richard Jones <richard@users.sourceforge.net>
parents: 891
diff changeset
1311 if not self.db.getclass(link_class).hasnode(id):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1312 raise IndexError('%s has no node %s' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1313 link_class, id))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1314 if id in l:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1315 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1316 # register the link with the newly linked node
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1317 if self.do_journal and self.properties[propname].do_journal:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1318 self.db.addjournal(link_class, id, 'link',
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1319 (self.classname, nodeid, propname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1320 l.append(id)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1321 add.append(id)
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1322
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1323 # figure the journal entry
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1324 l = []
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1325 if add:
967
dd35bab19dd9 use more robust date stamp comparisons in pack(), make journal smaller too
Richard Jones <richard@users.sourceforge.net>
parents: 964
diff changeset
1326 l.append(('+', add))
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1327 if remove:
967
dd35bab19dd9 use more robust date stamp comparisons in pack(), make journal smaller too
Richard Jones <richard@users.sourceforge.net>
parents: 964
diff changeset
1328 l.append(('-', remove))
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1329 if l:
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1330 journalvalues[propname] = tuple(l)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1331
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1332 elif isinstance(prop, hyperdb.String):
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1333 if value is not None and not isinstance(value, (str, unicode)):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1334 raise TypeError('new property "%s" not a '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1335 'string' % propname)
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2736
diff changeset
1336 if prop.indexme:
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1337 self.db.indexer.add_text(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1338 (self.classname, nodeid, propname),
2892
2eae5848912d always honor indexme property on Strings (patch [SF#063711])
Richard Jones <richard@users.sourceforge.net>
parents: 2736
diff changeset
1339 value)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1340
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1341 elif isinstance(prop, hyperdb.Password):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1342 if not isinstance(value, password.Password):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1343 raise TypeError('new property "%s" not a '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1344 'Password' % propname)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1345 propvalues[propname] = value
4483
22bc0426e348 Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4480
diff changeset
1346 journalvalues[propname] = \
22bc0426e348 Second patch from issue2550688 -- with some changes:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4480
diff changeset
1347 current and password.JournalPassword(current)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1348
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1349 elif value is not None and isinstance(prop, hyperdb.Date):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1350 if not isinstance(value, date.Date):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1351 raise TypeError('new property "%s" not a '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1352 'Date' % propname)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1353 propvalues[propname] = value
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1354
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1355 elif value is not None and isinstance(prop, hyperdb.Interval):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1356 if not isinstance(value, date.Interval):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1357 raise TypeError('new property "%s" not an '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1358 'Interval' % propname)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1359 propvalues[propname] = value
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1360
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1361 elif value is not None and isinstance(prop, hyperdb.Number):
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
1362 try:
890
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1363 float(value)
887
e7169d6e6e45 added tests for number type too
Richard Jones <richard@users.sourceforge.net>
parents: 886
diff changeset
1364 except ValueError:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1365 raise TypeError('new property "%s" not '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1366 'numeric' % propname)
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
1367
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1368 elif value is not None and isinstance(prop, hyperdb.Integer):
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1369 try:
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1370 int(value)
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1371 except ValueError:
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1372 raise TypeError('new property "%s" not '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1373 'numeric' % propname)
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1374
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1375 elif value is not None and isinstance(prop, hyperdb.Boolean):
890
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1376 try:
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1377 int(value)
a568596dbea7 Unit tests and a few fixes.
Richard Jones <richard@users.sourceforge.net>
parents: 887
diff changeset
1378 except ValueError:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1379 raise TypeError('new property "%s" not '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1380 'boolean' % propname)
880
de3da99a7c02 Add Number and Boolean types to hyperdb.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 869
diff changeset
1381
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1382 node[propname] = value
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1383
5112
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1384 # record quiet properties to omit from history/changelog
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1385 if prop.quiet:
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1386 quiet_props.append(propname)
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1387
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1388 # nothing to do?
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1389 if not propvalues:
930
3c344e942055 Use same regex to split search terms as used to index text.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 927
diff changeset
1390 return propvalues
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1391
3083
b9a819ef7554 actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3052
diff changeset
1392 # update the activity time
b9a819ef7554 actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3052
diff changeset
1393 node['activity'] = date.Date()
b9a819ef7554 actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3052
diff changeset
1394 node['actor'] = self.db.getuid()
b9a819ef7554 actor/activity update moved from Database.setnode() to Class.set_inner()
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 3052
diff changeset
1395
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1396 # do the set, and journal it
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1397 self.db.setnode(self.classname, nodeid, node)
869
6d98bec4e52e fixed the journal bloat from multilink changes
Richard Jones <richard@users.sourceforge.net>
parents: 862
diff changeset
1398
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1399 if self.do_journal:
1304
61ad556cfc8d working toward 0.5.2 release
Richard Jones <richard@users.sourceforge.net>
parents: 1303
diff changeset
1400 self.db.addjournal(self.classname, nodeid, 'set', journalvalues)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1401
5112
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1402 # remove quiet properties from output
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1403 for propname in quiet_props:
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1404 if propname in propvalues:
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1405 del propvalues[propname]
8901cc4ef0e0 - issue1714899: Feature Request: Optional Change Note. Added a new
John Rouillard <rouilj@ieee.org>
parents: 5107
diff changeset
1406
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
1407 return propvalues
930
3c344e942055 Use same regex to split search terms as used to index text.
Gordon B. McMillan <gmcm@users.sourceforge.net>
parents: 927
diff changeset
1408
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1409 def retire(self, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1410 """Retire a node.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1411
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1412 The properties on the node remain available from the get() method,
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1413 and the node's id is never reused.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1414
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1415 Retired nodes are not returned by the find(), list(), or lookup()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1416 methods, and other nodes may reuse the values of their key properties.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1417
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1418 These operations trigger detectors and can be vetoed. Attempts
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1419 to modify the "creation" or "activity" properties cause a KeyError.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1420 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1421 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1422 raise hyperdb.DatabaseError(_('Database open read-only'))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1423
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1424 self.fireAuditors('retire', nodeid, None)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1425
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1426 node = self.db.getnode(self.classname, nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1427 node[self.db.RETIRED_FLAG] = 1
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1428 self.db.setnode(self.classname, nodeid, node)
860
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1429 if self.do_journal:
2df32a73eb45 Implemented a switch to disable journalling for a Class.
Richard Jones <richard@users.sourceforge.net>
parents: 858
diff changeset
1430 self.db.addjournal(self.classname, nodeid, 'retired', None)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1431
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1432 self.fireReactors('retire', nodeid, None)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1433
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1434 def restore(self, nodeid):
5107
33ac07a7ef96 spelling error fixed in comment
John Rouillard <rouilj@ieee.org>
parents: 5100
diff changeset
1435 """Restore a retired node.
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1436
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1437 Make node available for all operations like it was before retirement.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1438 """
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1439 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1440 raise hyperdb.DatabaseError(_('Database open read-only'))
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1441
1523
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1442 node = self.db.getnode(self.classname, nodeid)
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1443 # check if key property was overrided
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1444 key = self.getkey()
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1445 try:
6998
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
1446 # eval for exception side effect
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
1447 id = self.lookup(node[key]) # noqa: F841
1523
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1448 except KeyError:
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1449 pass
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1450 else:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1451 raise KeyError("Key property (%s) of retired node clashes "
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1452 "with existing one (%s)" % (key, node[key]))
1523
63aa7be52d2c checked to make sure that the restored item doesn't clash...
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1519
diff changeset
1453 # Now we can safely restore node
1519
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1454 self.fireAuditors('restore', nodeid, None)
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1455 del node[self.db.RETIRED_FLAG]
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1456 self.db.setnode(self.classname, nodeid, node)
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1457 if self.do_journal:
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1458 self.db.addjournal(self.classname, nodeid, 'restored', None)
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1459
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1460 self.fireReactors('restore', nodeid, None)
6fede2aa6a12 added ability to restore retired nodes
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1505
diff changeset
1461
1476
5a01e90b7dc9 fixed export/import of retired nodes [SF#685273]
Richard Jones <richard@users.sourceforge.net>
parents: 1467
diff changeset
1462 def is_retired(self, nodeid, cldb=None):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1463 """Return true if the node is retired.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1464 """
1476
5a01e90b7dc9 fixed export/import of retired nodes [SF#685273]
Richard Jones <richard@users.sourceforge.net>
parents: 1467
diff changeset
1465 node = self.db.getnode(self.classname, nodeid, cldb)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1466 if self.db.RETIRED_FLAG in node:
940
301a02ea6020 added is_retired query to Class
Richard Jones <richard@users.sourceforge.net>
parents: 930
diff changeset
1467 return 1
301a02ea6020 added is_retired query to Class
Richard Jones <richard@users.sourceforge.net>
parents: 930
diff changeset
1468 return 0
301a02ea6020 added is_retired query to Class
Richard Jones <richard@users.sourceforge.net>
parents: 930
diff changeset
1469
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1470 def destroy(self, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1471 """Destroy a node.
1333
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
1472
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1473 WARNING: this method should never be used except in extremely rare
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1474 situations where there could never be links to the node being
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1475 deleted
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1476
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1477 WARNING: use retire() instead
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1478
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1479 WARNING: the properties of this node will not be available ever again
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1480
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1481 WARNING: really, use retire() instead
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1482
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1483 Well, I think that's enough warnings. This method exists mostly to
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1484 support the session storage of the cgi interface.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1485 """
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1486 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1487 raise hyperdb.DatabaseError(_('Database open read-only'))
891
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1488 self.db.destroynode(self.classname, nodeid)
974a4b94c5e3 Implemented the destroy() method needed by the session database...
Richard Jones <richard@users.sourceforge.net>
parents: 890
diff changeset
1489
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1490 # Locating nodes:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1491 def hasnode(self, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1492 """Determine if the given nodeid actually exists
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1493 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1494 return self.db.hasnode(self.classname, nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1495
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1496 def setkey(self, propname):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1497 """Select a String property of this class to be the key property.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1498
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1499 'propname' must be the name of a String property of this class or
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1500 None, or a TypeError is raised. The values of the key property on
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
1501 all existing nodes must be unique or a ValueError is raised. If the
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
1502 property doesn't exist, KeyError is raised.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1503 """
862
37fb48c3a136 Did some old TODOs
Richard Jones <richard@users.sourceforge.net>
parents: 860
diff changeset
1504 prop = self.getprops()[propname]
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1505 if not isinstance(prop, hyperdb.String):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1506 raise TypeError('key properties must be String')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1507 self.key = propname
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1508
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1509 def getkey(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1510 """Return the name of the key property for this class or None."""
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1511 return self.key
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1512
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1513 # TODO: set up a separate index db file for this? profile?
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1514 def lookup(self, keyvalue):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1515 """Locate a particular node by its key property and return its id.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1516
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1517 If this class has no key property, a TypeError is raised. If the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1518 'keyvalue' matches one of the values for the key property among
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1519 the nodes in this class, the matching node's id is returned;
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1520 otherwise a KeyError is raised.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1521 """
969
ddcb527491ba Consistent quoting
Richard Jones <richard@users.sourceforge.net>
parents: 967
diff changeset
1522 if not self.key:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1523 raise TypeError('No key property set for '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1524 'class %s' % self.classname)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1525 cldb = self.db.getclassdb(self.classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1526 try:
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1527 for nodeid in self.getnodeids(cldb):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1528 node = self.db.getnode(self.classname, nodeid, cldb)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1529 if self.db.RETIRED_FLAG in node:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1530 continue
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1531 if self.key not in node:
2606
17eb5aeada7f fix bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2601
diff changeset
1532 continue
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1533 if node[self.key] == keyvalue:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1534 return nodeid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1535 finally:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1536 cldb.close()
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1537 raise KeyError('No key (%s) value "%s" for "%s"' % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1538 self.key, keyvalue, self.classname))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1539
1103
db787cef1385 handled some XXXs
Richard Jones <richard@users.sourceforge.net>
parents: 1099
diff changeset
1540 # change from spec - allows multiple props to match
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1541 def find(self, **propspec):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1542 """Get the ids of nodes in this class which link to the given nodes.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1543
3239
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3166
diff changeset
1544 'propspec' consists of keyword args propname=nodeid or
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3166
diff changeset
1545 propname={nodeid:1, }
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
1546 'propname' must be the name of a property in this class, or a
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
1547 KeyError is raised. That property must be a Link or
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
1548 Multilink property, or a TypeError is raised.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1549
3239
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3166
diff changeset
1550 Any node in this class whose 'propname' property links to any of
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3166
diff changeset
1551 the nodeids will be returned. Examples::
1222
bc3bc3248dd1 added Class.find() unit test, fixed implementations
Richard Jones <richard@users.sourceforge.net>
parents: 1195
diff changeset
1552
3239
440f0a6a2e3c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 3166
diff changeset
1553 db.issue.find(messages='1')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1554 db.issue.find(messages={'1':1,'3':1}, files={'7':1})
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1555 db.issue.find(messages=('1','3'), files=('7',))
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1556 db.issue.find(messages=['1','3'], files=['7'])
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1557 """
6427
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1558 # shortcut
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1559 if not propspec:
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1560 return []
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1561
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1562 # validate the args
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1563 props = self.getprops()
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
1564 for propname, itemids in propspec.items():
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1565 # check the prop is OK
6427
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1566 prop = props[propname]
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1567 if not isinstance(prop, hyperdb.Link) and not isinstance(prop, hyperdb.Multilink):
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1568 raise TypeError("'%s' not a Link/Multilink "
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1569 "property" % propname)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1570
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1571 # ok, now do the find
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1572 cldb = self.db.getclassdb(self.classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1573 l = []
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1574 rev_multilinks = []
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1575 try:
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1576 for id in self.getnodeids(db=cldb):
1563
e2a8ce4d2317 Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents: 1558
diff changeset
1577 item = self.db.getnode(self.classname, id, db=cldb)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1578 if self.db.RETIRED_FLAG in item:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1579 continue
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
1580 for propname, itemids in propspec.items():
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1581 if not isinstance(itemids, dict):
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1582 if itemids is None or isinstance(itemids, type("")):
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1583 itemids = {itemids: 1}
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1584 else:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1585 itemids = dict.fromkeys(itemids)
1563
e2a8ce4d2317 Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents: 1558
diff changeset
1586
2450
c45ed2413044 fixed lookup of "missing" Link values for new props in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2437
diff changeset
1587 # special case if the item doesn't have this property
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1588 if propname not in item:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1589 if None in itemids:
2450
c45ed2413044 fixed lookup of "missing" Link values for new props in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2437
diff changeset
1590 l.append(id)
c45ed2413044 fixed lookup of "missing" Link values for new props in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2437
diff changeset
1591 break
c45ed2413044 fixed lookup of "missing" Link values for new props in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2437
diff changeset
1592 continue
c45ed2413044 fixed lookup of "missing" Link values for new props in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2437
diff changeset
1593
1563
e2a8ce4d2317 Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents: 1558
diff changeset
1594 # grab the property definition and its value on this item
6427
f08907bfd5a1 Fix find() with anydbm. Add fast return shortcut.
John Rouillard <rouilj@ieee.org>
parents: 6412
diff changeset
1595 prop = props[propname]
1563
e2a8ce4d2317 Class.find() may now find unset Links [SF#700620]
Richard Jones <richard@users.sourceforge.net>
parents: 1558
diff changeset
1596 value = item[propname]
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1597 if isinstance(prop, hyperdb.Link) and value in itemids:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1598 l.append(id)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1599 break
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1600 elif isinstance(prop, hyperdb.Multilink):
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1601 if prop.rev_property:
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
1602 rev_multilinks.append((prop, itemids))
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1603 continue
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1604 hit = 0
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1605 for v in value:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1606 if v in itemids:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1607 l.append(id)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1608 hit = 1
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1609 break
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1610 if hit:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1611 break
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1612 for prop, itemids in rev_multilinks:
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1613 rprop = prop.rev_property
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1614 fun = l.append
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
1615 if isinstance(rprop, hyperdb.Multilink):
6151
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1616 fun = l.extend
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1617 for id in itemids:
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1618 fun(rprop.cls.get(id, rprop.name))
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1619 if rev_multilinks:
ff059afae50a Make 'find' work for rev_multilink properties
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6148
diff changeset
1620 l = list(sorted(set(l)))
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1621 finally:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1622 cldb.close()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1623 return l
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1624
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1625 def stringFind(self, **requirements):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1626 """Locate a particular node by matching a set of its String
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1627 properties in a caseless search.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1628
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1629 If the property is not a String property, a TypeError is raised.
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1630
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1631 The return is a list of the id of all nodes that match.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1632 """
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1633 for propname in requirements:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1634 prop = self.properties[propname]
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
1635 if not isinstance(prop, hyperdb.String):
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
1636 raise TypeError("'%s' not a String property" % propname)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1637 requirements[propname] = requirements[propname].lower()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1638 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1639 cldb = self.db.getclassdb(self.classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1640 try:
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1641 for nodeid in self.getnodeids(cldb):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1642 node = self.db.getnode(self.classname, nodeid, cldb)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1643 if self.db.RETIRED_FLAG in node:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1644 continue
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
1645 for key, value in requirements.items():
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1646 if key not in node:
1296
34b1e6490e65 hardening for stringFind with missing props
Richard Jones <richard@users.sourceforge.net>
parents: 1252
diff changeset
1647 break
905
502a5ae11cc5 Very close now. The cgi and mailgw now use the new security API.
Richard Jones <richard@users.sourceforge.net>
parents: 902
diff changeset
1648 if node[key] is None or node[key].lower() != value:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1649 break
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1650 else:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1651 l.append(nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1652 finally:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1653 cldb.close()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1654 return l
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1655
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1656 def list(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1657 """ Return a list of the ids of the active nodes in this class.
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1658 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1659 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1660 cn = self.classname
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1661 cldb = self.db.getclassdb(cn)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1662 try:
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1663 for nodeid in self.getnodeids(cldb):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1664 node = self.db.getnode(cn, nodeid, cldb)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1665 if self.db.RETIRED_FLAG in node:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1666 continue
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1667 l.append(nodeid)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1668 finally:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1669 cldb.close()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1670 l.sort()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1671 return l
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1672
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1673 def getnodeids(self, db=None, retired=None):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1674 """ Return a list of ALL nodeids
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1675
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1676 Set retired=None to get all nodes. Otherwise it'll get all the
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1677 retired or non-retired nodes, depending on the flag.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
1678 """
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1679 res = []
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1680
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1681 # start off with the new nodes
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1682 if self.classname in self.db.newnodes:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1683 res.extend(self.db.newnodes[self.classname])
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1684
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1685 must_close = False
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1686 if db is None:
1486
f5f60c75a458 added test for error in sqlite backend, and fixed *dbm backend error
Richard Jones <richard@users.sourceforge.net>
parents: 1484
diff changeset
1687 db = self.db.getclassdb(self.classname)
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
1688 must_close = True
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1689 try:
5444
167f0d25ea8e Python 3 preparation: convert dbm keys back from bytes to strings.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5395
diff changeset
1690 res.extend(map(b2s, db.keys()))
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1691
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1692 # remove the uncommitted, destroyed nodes
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1693 if self.classname in self.db.destroyednodes:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1694 for nodeid in self.db.destroyednodes[self.classname]:
5011
d5da643b3d25 Remove key_in() from roundup.anypy.dbm_
John Kristensen <john@jerrykan.com>
parents: 4995
diff changeset
1695 if nodeid in db:
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1696 res.remove(nodeid)
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1697
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1698 # check retired flag
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1699 if retired is False or retired is True:
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1700 l = []
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1701 for nodeid in res:
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1702 node = self.db.getnode(self.classname, nodeid, db)
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1703 is_ret = self.db.RETIRED_FLAG in node
3486
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1704 if retired == is_ret:
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1705 l.append(nodeid)
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1706 res = l
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1707 finally:
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1708 if must_close:
34ada15b9936 all backends implement the retired check in getnodeids [SF#1290560]
Richard Jones <richard@users.sourceforge.net>
parents: 3476
diff changeset
1709 db.close()
1484
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1710 return res
b3f2484babce fixes to import/export
Richard Jones <richard@users.sourceforge.net>
parents: 1476
diff changeset
1711
7006
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1712 num_re = re.compile(r'^\d+$')
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1713
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1714 def _filter(self, search_matches, filterspec, proptree,
7006
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1715 num_re=num_re, retired=False,
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1716 exact_match_spec=_marker):
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1717 """Return a list of the ids of the nodes in this class that
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1718 match the 'filter' spec, sorted by the group spec and then the
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1719 sort spec.
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1720
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1721 "filterspec" is {propname: value(s)}
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1722 same for "exact_match_spec". The latter specifies exact matching
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1723 for String type while String types in "filterspec" are searched
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1724 for as case insensitive substring match.
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1725
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1726 "sort" and "group" are (dir, prop) where dir is '+', '-' or None
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1727 and prop is a prop name or None
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1728
4044
27a9906cd8d1 Fix issue2550505.
Stefan Seefeld <stefan@seefeld.name>
parents: 3979
diff changeset
1729 "search_matches" is a sequence type or None
1170
af104fa52746 Added some words to the installation doc about choosing backends.
Richard Jones <richard@users.sourceforge.net>
parents: 1159
diff changeset
1730
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1731 "retired" specifies if we should find only non-retired nodes
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1732 (default) or only retired node (value True) or all nodes.
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1733
3455
e01bc6d65fa9 fixed documentation of filter()...
Richard Jones <richard@users.sourceforge.net>
parents: 3419
diff changeset
1734 The filter must match all properties specificed. If the property
e01bc6d65fa9 fixed documentation of filter()...
Richard Jones <richard@users.sourceforge.net>
parents: 3419
diff changeset
1735 value to match is a list:
3906
666b70676ec6 destroy blobfiles if they exist
Justus Pendleton <jpend@users.sourceforge.net>
parents: 3897
diff changeset
1736
3455
e01bc6d65fa9 fixed documentation of filter()...
Richard Jones <richard@users.sourceforge.net>
parents: 3419
diff changeset
1737 1. String properties must match all elements in the list, and
e01bc6d65fa9 fixed documentation of filter()...
Richard Jones <richard@users.sourceforge.net>
parents: 3419
diff changeset
1738 2. Other properties must match any of the elements in the list.
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1955
diff changeset
1739 """
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
1740 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
1741 start_t = time.time()
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
1742
7006
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1743 if exact_match_spec is _marker:
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1744 exact_match_spec = {}
11e2b0b1cc71 Replace mutable type and function call in arguments of function declaration.
John Rouillard <rouilj@ieee.org>
parents: 7005
diff changeset
1745
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1746 cn = self.classname
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1747
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1748 # optimise filterspec
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1749 l = []
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1750 props = self.getprops()
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1751 LINK = 'spec:link'
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1752 MULTILINK = 'spec:multilink'
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1753 STRING = 'spec:string'
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1754 DATE = 'spec:date'
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1755 INTERVAL = 'spec:interval'
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1756 OTHER = 'spec:other'
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1757
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1758 for exact, filtertype in enumerate((filterspec, exact_match_spec)):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1759 for k, v in filtertype.items():
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1760 propclass = props[k]
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1761 if isinstance(propclass, hyperdb.Link):
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1762 if not isinstance(v, list):
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1763 v = [v]
6412
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
1764 l.append((LINK, k, v))
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1765 elif isinstance(propclass, hyperdb.Multilink):
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1766 # If it's a reverse multilink, we've already
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1767 # computed the ids of our own class.
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1768 if propclass.rev_property:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1769 l.append((OTHER, 'id', v))
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1770 else:
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1771 # the value -1 is a special "not set" sentinel
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1772 if v in ('-1', ['-1']):
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1773 v = []
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1774 elif not isinstance(v, list):
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1775 v = [v]
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6118
diff changeset
1776 l.append((MULTILINK, k, v))
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1777 elif isinstance(propclass, hyperdb.String) and k != 'id':
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1778 if not isinstance(v, list):
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1779 v = [v]
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1780 for x in v:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1781 if exact:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1782 l.append((STRING, k, x))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1783 else:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1784 # simple glob searching
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1785 x = re.sub(r'([\|\{\}\\\.\+\[\]\(\)])', r'\\\1', x)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1786 x = x.replace('?', '.')
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1787 x = x.replace('*', '.*?')
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1788 l.append((STRING, k, re.compile(x, re.I)))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1789 elif isinstance(propclass, hyperdb.Date):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1790 try:
6118
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1791 ranges = []
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1792 for d in v.split(','):
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1793 if d == '-':
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1794 ranges.append(None)
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1795 continue
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1796 date_rng = propclass.range_from_raw(d, self.db)
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1797 ranges.append(date_rng)
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1798 l.append((DATE, k, ranges))
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1799 except ValueError:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1800 # If range creation fails - ignore that search parameter
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1801 pass
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1802 elif isinstance(propclass, hyperdb.Interval):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1803 try:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1804 intv_rng = date.Range(v, date.Interval)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1805 l.append((INTERVAL, k, intv_rng))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1806 except ValueError:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1807 # If range creation fails - ignore that search parameter
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1808 pass
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
1809
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1810 elif isinstance(propclass, hyperdb.Boolean):
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1811 if isinstance(v, str):
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1812 v = v.split(',')
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1813 if not isinstance(v, list):
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1814 v = [v]
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1815 bv = []
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1816 for val in v:
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1817 if isinstance(val, str):
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
1818 bv.append(propclass.from_raw(val))
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1819 else:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1820 bv.append(val)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1821 l.append((OTHER, k, bv))
2453
0e2a0c2c8142 allow list of values for id, Number and Boolean filtering in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2450
diff changeset
1822
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1823 elif k == 'id':
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1824 if not isinstance(v, list):
4364
0e81742d0e2f - unify number searching across backends
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4357
diff changeset
1825 v = v.split(',')
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1826 l.append((OTHER, k, [str(int(val)) for val in v]))
2453
0e2a0c2c8142 allow list of values for id, Number and Boolean filtering in anydbm backend
Richard Jones <richard@users.sourceforge.net>
parents: 2450
diff changeset
1827
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1828 elif isinstance(propclass, hyperdb.Number):
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1829 if not isinstance(v, list):
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1830 try:
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1831 v = v.split(',')
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1832 except AttributeError:
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1833 v = [v]
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1834 l.append((OTHER, k, [float(val) for val in v]))
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1835
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1836 elif isinstance(propclass, hyperdb.Integer):
7012
921bf9bdeea9 flake8 convert type comparisons to isinstance.
John Rouillard <rouilj@ieee.org>
parents: 7011
diff changeset
1837 if not isinstance(v, list):
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1838 try:
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1839 v = v.split(',')
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1840 except AttributeError:
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1841 v = [v]
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1842 l.append((OTHER, k, [int(val) for val in v]))
5067
e424987d294a Add support for an integer type to join the existing number type.
John Rouillard <rouilj@ieee.org>
parents: 5041
diff changeset
1843
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1844 filterspec = l
4480
1613754d2646 Fix first part of Password handling security issue2550688
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 4466
diff changeset
1845
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1846 # now, find all the nodes that pass filtering
2239
c8a06e10e2c6 much faster anydbm filter(), but it breaks most filtering tests
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
1847 matches = []
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1848 cldb = self.db.getclassdb(cn)
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1849 t = 0
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1850 try:
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1851 # TODO: only full-scan once (use items())
5318
506c7ee9a385 Add a 'retired' parameter to Class.filter
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5248
diff changeset
1852 for nodeid in self.getnodeids(cldb, retired=retired):
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1853 node = self.db.getnode(cn, nodeid, cldb)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1854 # apply filter
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1855 for t, k, v in filterspec:
1303
71be6588904f fixed filtering by id in anydbm
Richard Jones <richard@users.sourceforge.net>
parents: 1296
diff changeset
1856 # handle the id prop
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1857 if k == 'id':
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1858 if nodeid not in v:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1859 break
1303
71be6588904f fixed filtering by id in anydbm
Richard Jones <richard@users.sourceforge.net>
parents: 1296
diff changeset
1860 continue
71be6588904f fixed filtering by id in anydbm
Richard Jones <richard@users.sourceforge.net>
parents: 1296
diff changeset
1861
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1862 # get the node value
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1863 nv = node.get(k, None)
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1864
2617
33fffbf7ae68 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2608
diff changeset
1865 match = 0
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1866
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1867 # now apply the property filter
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1868 if t == LINK:
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1869 # link - if this node's property doesn't appear in the
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1870 # filterspec's nodeid list, skip it
6412
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
1871 expr = Expression(v, is_link=True)
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
1872 if expr.evaluate(nv):
a0c0ee3ed8b1 Tests for Link expressions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6401
diff changeset
1873 match = 1
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1874 elif t == MULTILINK:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1875 # multilink - if any of the nodeids required by the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1876 # filterspec aren't in this node's property, then skip
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1877 # it
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1878 nv = node.get(k, [])
1555
948c7764d46c implemented ability to search for multilink properties with no value (not in mk)
Richard Jones <richard@users.sourceforge.net>
parents: 1523
diff changeset
1879
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1880 # check for matching the absence of multilink values
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1881 if not v:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1882 match = not nv
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1883 else:
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4447
diff changeset
1884 # otherwise, make sure this node has each of the
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1885 # required values
4466
f1fe6fd0aa61 Multilinks can be filtered by combining elements with AND, OR and NOT now.
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4447
diff changeset
1886 expr = Expression(v)
6396
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6352
diff changeset
1887 if expr.evaluate(nv):
75a53956cf13 Multilink expressions with simple "or"
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6352
diff changeset
1888 match = 1
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1889 elif t == STRING:
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1890 if nv is None:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1891 nv = ''
5867
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1892 if is_us(v):
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1893 # Exact match
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1894 match = (nv == v)
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1895 else:
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1896 # RE search
ee2e8f8d6648 Implement exact string search
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5809
diff changeset
1897 match = v.search(nv)
6118
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1898 elif t == DATE:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1899 for x in v:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1900 if x is None or nv is None:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1901 if nv is None and x is None:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1902 match = 1
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1903 break
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1904 continue
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1905 elif x.to_value:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1906 if x.from_value <= nv <= x.to_value:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1907 match = 1
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1908 break
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1909 else:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1910 if x.from_value <= nv:
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1911 match = 1
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1912 break
e6073c2291c6 Better Date filtering
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6002
diff changeset
1913 elif t == INTERVAL:
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1914 if nv is None:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1915 match = v is None
1499
8ee69708da0c added support for searching on ranges of dates
Andrey Lebedev <kedder@users.sourceforge.net>
parents: 1496
diff changeset
1916 else:
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1917 if v.to_value:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1918 if v.from_value <= nv and v.to_value >= nv:
2617
33fffbf7ae68 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2608
diff changeset
1919 match = 1
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1920 else:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1921 if v.from_value <= nv:
2617
33fffbf7ae68 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2608
diff changeset
1922 match = 1
924
aa10112dd7d1 cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 916
diff changeset
1923 elif t == OTHER:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1924 # straight value comparison for the other types
2486
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1925 match = nv in v
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1926 if not match:
d7488e349a03 merge from maint-0-7
Richard Jones <richard@users.sourceforge.net>
parents: 2453
diff changeset
1927 break
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
1928 else:
2239
c8a06e10e2c6 much faster anydbm filter(), but it breaks most filtering tests
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
1929 matches.append([nodeid, node])
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1930
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1931 # filter based on full text search
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1932 if search_matches is not None:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1933 k = []
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1934 for v in matches:
4044
27a9906cd8d1 Fix issue2550505.
Stefan Seefeld <stefan@seefeld.name>
parents: 3979
diff changeset
1935 if v[0] in search_matches:
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1936 k.append(v)
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1937 matches = k
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1938
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1939 # add sorting information to the proptree
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1940 JPROPS = {'actor': 1, 'activity': 1, 'creator': 1, 'creation': 1}
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1941 children = []
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1942 if proptree:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1943 children = proptree.sortable_children()
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1944 for pt in children:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1945 dir = pt.sort_direction
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1946 prop = pt.name
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1947 assert (dir and prop)
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1948 propclass = props[prop]
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1949 pt.sort_ids = []
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1950 is_pointer = isinstance(propclass, (hyperdb.Link,
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1951 hyperdb.Multilink))
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1952 if not is_pointer:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1953 pt.sort_result = []
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1954 try:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1955 # cache the opened link class db, if needed.
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1956 lcldb = None
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1957 # cache the linked class items too
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1958 lcache = {}
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1959
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1960 for entry in matches:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1961 itemid = entry[-2]
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1962 item = entry[-1]
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1963 # handle the properties that might be "faked"
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1964 # also, handle possible missing properties
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1965 try:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1966 v = item[prop]
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1967 except KeyError:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1968 if prop in JPROPS:
2437
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1969 # force lookup of the special journal prop
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1970 v = self.get(itemid, prop)
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1971 else:
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1972 # the node doesn't have a value for this
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1973 # property
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1974 v = None
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1975 if isinstance(propclass, hyperdb.Multilink):
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1976 v = []
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1977 if prop == 'id':
7001
a41d88b560f8 Remove whitespace before ( and [
John Rouillard <rouilj@ieee.org>
parents: 7000
diff changeset
1978 v = int(itemid)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1979 pt.sort_ids.append(v)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1980 if not is_pointer:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1981 pt.sort_result.append(v)
2437
f37f3617b9e9 force lookup of journal props in anydbm filtering
Richard Jones <richard@users.sourceforge.net>
parents: 2418
diff changeset
1982 continue
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1983
2285
c7f780c24a87 fixed anydbm sorting with None values [SF#952853]
Richard Jones <richard@users.sourceforge.net>
parents: 2252
diff changeset
1984 # missing (None) values are always sorted first
c7f780c24a87 fixed anydbm sorting with None values [SF#952853]
Richard Jones <richard@users.sourceforge.net>
parents: 2252
diff changeset
1985 if v is None:
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1986 pt.sort_ids.append(v)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1987 if not is_pointer:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1988 pt.sort_result.append(v)
2285
c7f780c24a87 fixed anydbm sorting with None values [SF#952853]
Richard Jones <richard@users.sourceforge.net>
parents: 2252
diff changeset
1989 continue
c7f780c24a87 fixed anydbm sorting with None values [SF#952853]
Richard Jones <richard@users.sourceforge.net>
parents: 2252
diff changeset
1990
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1991 if isinstance(propclass, hyperdb.Link):
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1992 lcn = propclass.classname
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1993 link = self.db.classes[lcn]
3476
1142dafe0d7f added setorderprop() and setlabelprop() to Class (lots of patches)
Richard Jones <richard@users.sourceforge.net>
parents: 3455
diff changeset
1994 key = link.orderprop()
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
1995 child = pt.propdict[key]
7003
230852cb909f Final whitespace cleanups.
John Rouillard <rouilj@ieee.org>
parents: 7002
diff changeset
1996 if key != 'id':
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
1997 if v not in lcache:
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1998 # open the link class db if it's not already
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
1999 if lcldb is None:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2000 lcldb = self.db.getclassdb(lcn)
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2001 lcache[v] = self.db.getnode(lcn, v, lcldb)
3682
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2002 r = lcache[v][key]
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2003 child.propdict[key].sort_ids.append(r)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2004 else:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2005 child.propdict[key].sort_ids.append(v)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2006 pt.sort_ids.append(v)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2007 if not is_pointer:
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2008 r = propclass.sort_repr(pt.parent.cls, v, pt.name)
193f316dbbe9 More transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3634
diff changeset
2009 pt.sort_result.append(r)
2252
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2010 finally:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2011 # if we opened the link class db, close it now
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2012 if lcldb is not None:
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2013 lcldb.close()
b63e455462ee some small performance boost for *dbm backends... bsddb3 bites though
Richard Jones <richard@users.sourceforge.net>
parents: 2249
diff changeset
2014 del lcache
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2015 finally:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2016 cldb.close()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2017
2239
c8a06e10e2c6 much faster anydbm filter(), but it breaks most filtering tests
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
2018 # pull the id out of the individual entries
c8a06e10e2c6 much faster anydbm filter(), but it breaks most filtering tests
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
2019 matches = [entry[-2] for entry in matches]
2237
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
2020 if __debug__:
f624fc20f8fe added capturing of stats
Richard Jones <richard@users.sourceforge.net>
parents: 2191
diff changeset
2021 self.db.stats['filtering'] += (time.time() - start_t)
2239
c8a06e10e2c6 much faster anydbm filter(), but it breaks most filtering tests
Richard Jones <richard@users.sourceforge.net>
parents: 2237
diff changeset
2022 return matches
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2023
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2024 def count(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2025 """Get the number of nodes in this class.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2026
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2027 If the returned integer is 'numnodes', the ids of all the nodes
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2028 in this class run from 1 to numnodes, and numnodes+1 will be the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2029 id of the next node to be created in this class.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2030 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2031 return self.db.countnodes(self.classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2032
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2033 # Manipulating properties:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2034
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2035 def getprops(self, protected=1):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2036 """Return a dictionary mapping property names to property objects.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2037 If the "protected" flag is true, we include protected properties -
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2038 those which may not be modified.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2039
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2040 In addition to the actual properties on the node, these
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2041 methods provide the "creation" and "activity" properties. If the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2042 "protected" flag is true, we include protected properties - those
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2043 which may not be modified.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2044 """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2045 d = self.properties.copy()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2046 if protected:
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
2047 d['id'] = hyperdb.String()
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2048 d['creation'] = hyperdb.Date()
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2049 d['activity'] = hyperdb.Date()
1176
bd3b57859c37 On second thought, that last checkin was dumb.
Richard Jones <richard@users.sourceforge.net>
parents: 1174
diff changeset
2050 d['creator'] = hyperdb.Link('user')
2077
3e0961d6d44d Added the "actor" property.
Richard Jones <richard@users.sourceforge.net>
parents: 2076
diff changeset
2051 d['actor'] = hyperdb.Link('user')
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2052 return d
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2053
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2054 def addprop(self, **properties):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2055 """Add properties to this class.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2056
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2057 The keyword arguments in 'properties' must map names to property
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2058 objects, or a TypeError is raised. None of the keys in 'properties'
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2059 may collide with the names of existing properties, or a ValueError
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2060 is raised before any properties have been added.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2061 """
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2062 for key in properties:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2063 if key in self.properties:
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2064 raise ValueError(key)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2065 self.properties.update(properties)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2066
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2067 def index(self, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2068 """ Add (or refresh) the node to search indexes """
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2069 # find all the String properties that have indexme
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
2070 for prop, propclass in self.getprops().items():
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2071 if isinstance(propclass, hyperdb.String) and propclass.indexme:
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2072 # index them under (classname, nodeid, property)
7393
9e612a39547a Make anydbm indexer handle missing items the same as rdbms backend
John Rouillard <rouilj@ieee.org>
parents: 7211
diff changeset
2073 self.db.indexer.add_text((self.classname, nodeid, prop),
9e612a39547a Make anydbm indexer handle missing items the same as rdbms backend
John Rouillard <rouilj@ieee.org>
parents: 7211
diff changeset
2074 str(self.get(nodeid, prop)))
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2075
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2076 #
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2077 # import / export support
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2078 #
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2079 def export_list(self, propnames, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2080 """ Export a node - generate a list of CSV-able data in the order
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2081 specified by propnames for the given node.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2082 """
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2083 properties = self.getprops()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2084 l = []
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2085 for prop in propnames:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2086 proptype = properties[prop]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2087 value = self.get(nodeid, prop)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2088 # "marshal" data where needed
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2089 if value is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2090 pass
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2091 elif isinstance(proptype, hyperdb.Date):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2092 value = value.get_tuple()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2093 elif isinstance(proptype, hyperdb.Interval):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2094 value = value.get_tuple()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2095 elif isinstance(proptype, hyperdb.Password):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2096 value = str(value)
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2097 l.append(repr_export(value))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2098
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2099 # append retired flag
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2100 l.append(repr_export(self.is_retired(nodeid)))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2101
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2102 return l
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2103
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2104 def import_list(self, propnames, proplist):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2105 """ Import a node - all information including "id" is present and
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2106 should not be sanity checked. Triggers are not triggered. The
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2107 journal should be initialised using the "creator" and "created"
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2108 information.
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2109
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2110 Return the nodeid of the node imported.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2111 """
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2112 if self.db.journaltag is None:
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2113 raise hyperdb.DatabaseError(_('Database open read-only'))
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2114 properties = self.getprops()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2115
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2116 # make the new node's property map
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2117 d = {}
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2118 newid = None
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2119 for i in range(len(propnames)):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2120 # Figure the property for this column
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2121 propname = propnames[i]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2122
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2123 # Use eval_import to reverse the repr_export() used to
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2124 # output the CSV
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2125 value = eval_import(proplist[i])
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2126
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2127 # "unmarshal" where necessary
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2128 if propname == 'id':
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2129 newid = value
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2130 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2131 elif propname == 'is retired':
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2132 # is the item retired?
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2133 if int(value):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2134 d[self.db.RETIRED_FLAG] = 1
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2135 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2136 elif value is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2137 d[propname] = None
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2138 continue
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2139
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2140 prop = properties[propname]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2141 if isinstance(prop, hyperdb.Date):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2142 value = date.Date(value)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2143 elif isinstance(prop, hyperdb.Interval):
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2144 value = date.Interval(value)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2145 elif isinstance(prop, hyperdb.Password):
7211
506c86823abb Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents: 7038
diff changeset
2146 value = password.Password(encrypted=value,
506c86823abb Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents: 7038
diff changeset
2147 config=self.db.config)
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2148 d[propname] = value
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2149
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2150 # get a new id if necessary
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2151 if newid is None:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2152 newid = self.db.newid(self.classname)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2153
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2154 # add the node and journal
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2155 self.db.addnode(self.classname, newid, d)
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2156 return newid
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2157
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2158 def export_journals(self):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2159 """Export a class's journal - generate a list of lists of
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2160 CSV-able data:
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2161
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2162 nodeid, date, user, action, params
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2163
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2164 No heading here - the columns are fixed.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2165 """
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2166 properties = self.getprops()
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2167 r = []
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2168 for nodeid in self.getnodeids():
6998
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
2169 for nodeid, date_, user, action, params in self.history(
6997
60b37f632601 Flake8 fix whitespace/indent issues.
John Rouillard <rouilj@ieee.org>
parents: 6820
diff changeset
2170 nodeid, enforceperm=False, skipquiet=False):
6998
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
2171 date_ = date_.get_tuple()
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2172 if action == 'set':
3003
855e9f99c96c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 2966
diff changeset
2173 export_data = {}
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
2174 for propname, value in params.items():
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2175 if propname not in properties:
3003
855e9f99c96c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 2966
diff changeset
2176 # property no longer in the schema
855e9f99c96c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 2966
diff changeset
2177 continue
855e9f99c96c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 2966
diff changeset
2178
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2179 prop = properties[propname]
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2180 # make sure the params are eval()'able
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2181 if value is None:
3826
bf2e9535da00 Journal and database testing.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3687
diff changeset
2182 pass
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
2183 elif isinstance(prop, hyperdb.Date):
3166
3204488d20b5 hack to fix some anydbm export problems [SF#1081454]
Richard Jones <richard@users.sourceforge.net>
parents: 3155
diff changeset
2184 # this is a hack - some dates are stored as strings
3204488d20b5 hack to fix some anydbm export problems [SF#1081454]
Richard Jones <richard@users.sourceforge.net>
parents: 3155
diff changeset
2185 if not isinstance(value, type('')):
3204488d20b5 hack to fix some anydbm export problems [SF#1081454]
Richard Jones <richard@users.sourceforge.net>
parents: 3155
diff changeset
2186 value = value.get_tuple()
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
2187 elif isinstance(prop, hyperdb.Interval):
3390
a2e4b8ab5b51 include hack from [SF#1238571]
Richard Jones <richard@users.sourceforge.net>
parents: 3383
diff changeset
2188 # hack too - some intervals are stored as strings
a2e4b8ab5b51 include hack from [SF#1238571]
Richard Jones <richard@users.sourceforge.net>
parents: 3383
diff changeset
2189 if not isinstance(value, type('')):
a2e4b8ab5b51 include hack from [SF#1238571]
Richard Jones <richard@users.sourceforge.net>
parents: 3383
diff changeset
2190 value = value.get_tuple()
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
2191 elif isinstance(prop, hyperdb.Password):
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2192 value = str(value)
3003
855e9f99c96c merge from maint-0-8
Richard Jones <richard@users.sourceforge.net>
parents: 2966
diff changeset
2193 export_data[propname] = value
3052
230f7394cce9 fix to fix
Richard Jones <richard@users.sourceforge.net>
parents: 3003
diff changeset
2194 params = export_data
6998
e14b013ac83b flake8: fix imports; fix shadowed import; silence unused var warning
John Rouillard <rouilj@ieee.org>
parents: 6997
diff changeset
2195 r.append([repr_export(nodeid), repr_export(date_),
5525
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2196 repr_export(user), repr_export(action),
bb7865241f8a Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975).
Joseph Myers <jsm@polyomino.org.uk>
parents: 5492
diff changeset
2197 repr_export(params)])
2175
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2198 return r
723098a10677 Export and import now include journals (incompatible with export < 0.7)
Richard Jones <richard@users.sourceforge.net>
parents: 2089
diff changeset
2199
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
2200
2597
c86b2179085b fix journal export of files to remove content from CSV files
Richard Jones <richard@users.sourceforge.net>
parents: 2514
diff changeset
2201 class FileClass(hyperdb.FileClass, Class):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2202 """This class defines a large chunk of data. To support this, it has a
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2203 mandatory String property "content" which is typically saved off
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2204 externally to the hyperdb.
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2205
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2206 The default MIME type of this data is defined by the
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2207 "default_mime_type" class attribute, which may be overridden by each
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2208 node if the class defines a "type" String property.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2209 """
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2210 def __init__(self, db, classname, **properties):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2211 """The newly-created class automatically includes the "content"
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2212 and "type" properties.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2213 """
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2214 if 'content' not in properties:
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2215 properties['content'] = hyperdb.String(indexme='yes')
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2216 if 'type' not in properties:
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2217 properties['type'] = hyperdb.String()
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2218 Class.__init__(self, db, classname, **properties)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2219
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2220 def create(self, **propvalues):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2221 """ Snarf the "content" propvalue and store in a file
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2222 """
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
2223 # we need to fire the auditors now, or the content property won't
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
2224 # be in propvalues for the auditors to play with
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
2225 self.fireAuditors('create', None, propvalues)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
2226
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
2227 # now remove the content property so it's not stored in the db
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2228 content = propvalues['content']
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2229 del propvalues['content']
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
2230
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2231 # make sure we have a MIME type
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2232 mime_type = propvalues.get('type', self.default_mime_type)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2233
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
2234 # do the database create
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2235 newid = self.create_inner(**propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2236
3979
954971d612ee Fire reactors after file storage is all done (patch [SF#2001243])
Richard Jones <richard@users.sourceforge.net>
parents: 3960
diff changeset
2237 # store off the content as a file
5492
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
2238 self.db.storefile(self.classname, newid, None, bs2b(content))
3979
954971d612ee Fire reactors after file storage is all done (patch [SF#2001243])
Richard Jones <richard@users.sourceforge.net>
parents: 3960
diff changeset
2239
1431
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
2240 # fire reactors
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
2241 self.fireReactors('create', newid, None)
c70068162e64 Altered Class.create() and FileClass.create() methods...
Richard Jones <richard@users.sourceforge.net>
parents: 1424
diff changeset
2242
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2243 return newid
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2244
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2245 def get(self, nodeid, propname, default=_marker, cache=1):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2246 """ Trap the content propname and get it from the file
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
2247
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
2248 'cache' exists for backwards compatibility, and is not used.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2249 """
1333
80d27b7d6db5 implemented whole-database locking
Richard Jones <richard@users.sourceforge.net>
parents: 1327
diff changeset
2250 poss_msg = 'Possibly an access right configuration problem.'
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2251 if propname == 'content':
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2252 try:
5492
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
2253 return b2s(self.db.getfile(self.classname, nodeid, None))
5248
198b6e810c67 Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents: 5232
diff changeset
2254 except IOError as strerror:
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
2255 # XXX by catching this we don't see an error in the log.
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
2256 return 'ERROR reading file: %s%s\n%s\n%s' % (
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2257 self.classname, nodeid, poss_msg, strerror)
5725
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5704
diff changeset
2258 except UnicodeDecodeError as e:
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5704
diff changeset
2259 # if content is not text (e.g. jpeg file) we get
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5704
diff changeset
2260 # unicode error trying to convert to string in python 3.
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5704
diff changeset
2261 # trap it and supply an error message. Include md5sum
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5704
diff changeset
2262 # of content as this string is included in the etag
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5704
diff changeset
2263 # calculation of the object.
6923225fd781 Handle UnicodeDecodeError in file class when file contents are not
John Rouillard <rouilj@ieee.org>
parents: 5704
diff changeset
2264 return ('%s%s is not text, retrieve using '
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
2265 'binary_content property. mdsum: %s') % (
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
2266 self.classname, nodeid,
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
2267 md5(self.db.getfile(
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
2268 self.classname, nodeid, None)).hexdigest()) # nosec - bandit md5 use ok
5492
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
2269 elif propname == 'binary_content':
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
2270 return self.db.getfile(self.classname, nodeid, None)
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
2271
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2272 if default is not _marker:
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
2273 return Class.get(self, nodeid, propname, default)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2274 else:
1780
d2801a2b0a77 Initial implementation (half-baked) at new Tracker instance.
Richard Jones <richard@users.sourceforge.net>
parents: 1751
diff changeset
2275 return Class.get(self, nodeid, propname)
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2276
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2277 def set(self, itemid, **propvalues):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2278 """ Snarf the "content" propvalue and update it in a file
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2279 """
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2280 self.fireAuditors('set', itemid, propvalues)
2608
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
2281
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
2282 # create the oldvalues dict - fill in any missing values
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2283 oldvalues = copy.deepcopy(self.db.getnode(self.classname, itemid))
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
2284 for name, prop in self.getprops(protected=0).items():
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2285 if name in oldvalues:
2608
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
2286 continue
2906
a8808157f892 fix some bugs introduced in refactoring of blobfiles filename()
Richard Jones <richard@users.sourceforge.net>
parents: 2899
diff changeset
2287 if isinstance(prop, hyperdb.Multilink):
2608
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
2288 oldvalues[name] = []
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
2289 else:
151ee7f0ca7d fix another bug exposed by earlier change to defaulting of *dbm property values
Richard Jones <richard@users.sourceforge.net>
parents: 2606
diff changeset
2290 oldvalues[name] = None
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2291
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2292 # now remove the content property so it's not stored in the db
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2293 content = None
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2294 if 'content' in propvalues:
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2295 content = propvalues['content']
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2296 del propvalues['content']
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2297
2640
b01eca163779 The "type" parameter is supposed to be optional
Richard Jones <richard@users.sourceforge.net>
parents: 2637
diff changeset
2298 # do the database update
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2299 propvalues = self.set_inner(itemid, **propvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2300
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2301 # do content?
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2302 if content:
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2303 # store and possibly index
5492
6b0c542642be blobfiles now always stores/returns bytes
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5467
diff changeset
2304 self.db.storefile(self.classname, itemid, None, bs2b(content))
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2305 if self.properties['content'].indexme:
5544
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
2306 index_content = content
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
2307 if bytes != str and isinstance(content, bytes):
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
2308 index_content = content.decode('utf-8', errors='ignore')
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2309 mime_type = self.get(itemid, 'type', self.default_mime_type)
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2310 self.db.indexer.add_text((self.classname, itemid, 'content'),
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
2311 index_content, mime_type)
2640
b01eca163779 The "type" parameter is supposed to be optional
Richard Jones <richard@users.sourceforge.net>
parents: 2637
diff changeset
2312 propvalues['content'] = content
b01eca163779 The "type" parameter is supposed to be optional
Richard Jones <richard@users.sourceforge.net>
parents: 2637
diff changeset
2313
2089
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2314 # fire reactors
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2315 self.fireReactors('set', itemid, oldvalues)
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2316 return propvalues
93f03c6714d8 A few big changes in this commit:
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
2317
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2318 def index(self, nodeid):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2319 """ Add (or refresh) the node to search indexes.
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2320
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2321 Use the content-type property for the content property.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2322 """
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2323 # find all the String properties that have indexme
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
2324 for prop, propclass in self.getprops().items():
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2325 if prop == 'content' and propclass.indexme:
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2326 mime_type = self.get(nodeid, 'type', self.default_mime_type)
5544
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
2327 index_content = self.get(nodeid, 'binary_content')
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
2328 if bytes != str and isinstance(index_content, bytes):
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
2329 index_content = index_content.decode('utf-8',
1a0498c1ed90 Avoid errors indexing binary uploads with Python 3.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5525
diff changeset
2330 errors='ignore')
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2331 self.db.indexer.add_text((self.classname, nodeid, 'content'),
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
2332 index_content, mime_type)
3601
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2333 elif isinstance(propclass, hyperdb.String) and propclass.indexme:
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2334 # index them under (classname, nodeid, property)
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2335 try:
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2336 value = str(self.get(nodeid, prop))
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2337 except IndexError:
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2338 # node has been destroyed
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2339 continue
7b25567f0f54 indexing may be turned off for FileClass "content" now
Richard Jones <richard@users.sourceforge.net>
parents: 3589
diff changeset
2340 self.db.indexer.add_text((self.classname, nodeid, prop), value)
2505
bdd112cf61ba rdbms backend full text search failure after import [SF#980314]
Richard Jones <richard@users.sourceforge.net>
parents: 2496
diff changeset
2341
7000
30cc3d6f1a3d Flake8 fixes: spaces around modulo, need extra blank lines, underindent
John Rouillard <rouilj@ieee.org>
parents: 6999
diff changeset
2342
1103
db787cef1385 handled some XXXs
Richard Jones <richard@users.sourceforge.net>
parents: 1099
diff changeset
2343 # deviation from spec - was called ItemClass
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2344 class IssueClass(Class, roundupdb.IssueClass):
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2345 # Overridden methods:
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2346 def __init__(self, db, classname, **properties):
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2347 """The newly-created class automatically includes the "messages",
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2348 "files", "nosy", and "superseder" properties. If the 'properties'
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2349 dictionary attempts to specify any of these properties or a
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2350 "creation" or "activity" property, a ValueError is raised.
4075
b87d022a2f40 Uniformly use """...""" instead of '''...''' for comments.
Stefan Seefeld <stefan@seefeld.name>
parents: 4044
diff changeset
2351 """
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2352 if 'title' not in properties:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2353 properties['title'] = hyperdb.String(indexme='yes')
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2354 if 'messages' not in properties:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2355 properties['messages'] = hyperdb.Multilink("msg")
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2356 if 'files' not in properties:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2357 properties['files'] = hyperdb.Multilink("file")
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2358 if 'nosy' not in properties:
1002
1798d2fa9fec Hack hack...
Richard Jones <richard@users.sourceforge.net>
parents: 990
diff changeset
2359 # note: journalling is turned off as it really just wastes
1798d2fa9fec Hack hack...
Richard Jones <richard@users.sourceforge.net>
parents: 990
diff changeset
2360 # space. this behaviour may be overridden in an instance
1798d2fa9fec Hack hack...
Richard Jones <richard@users.sourceforge.net>
parents: 990
diff changeset
2361 properties['nosy'] = hyperdb.Multilink("user", do_journal="no")
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4348
diff changeset
2362 if 'superseder' not in properties:
858
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2363 properties['superseder'] = hyperdb.Multilink(classname)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2364 Class.__init__(self, db, classname, **properties)
2dd862af72ee all storage-specific code (ie. backend) is now implemented by the backends
Richard Jones <richard@users.sourceforge.net>
parents: 843
diff changeset
2365
2699
2f5bf63a4b2c trim trailing spaces; add vim modeline
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2650
diff changeset
2366 # vim: set et sts=4 sw=4 :
Roundup Issue Tracker: http://roundup-tracker.org/