annotate roundup/test/memorydb.py @ 6565:2c2dbfc332ba

Try to handle multiple connections better. The session database is a hot spot. When multiple requests (e.g. 20) come in at the same time session database contention can get great. The original code didn't retry session database access when the open failed. This resulted in errors at the client. The second pass delayed 0.01 seconds and retried. It was better but we still had multiple second stalls. I think the first request got in, everybody else backed up and then retried at the same time. Again they stepped on each other. With logging I would see many counters go all the way to low single digits or to -1 indicating falure. This pass uses randomint to generate delays from 0-.125 seconds in 5ms increments. This performs better in testing. I rarely saw a counter less than 13 (2 failed retries). Current logging starts after 6 failures and counts down until success or failure.
author John Rouillard <rouilj@ieee.org>
date Thu, 16 Dec 2021 20:02:00 -0500
parents a7072cf4ad7e
children 408fd477761f
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1 '''Implement an in-memory hyperdb for testing purposes.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2 '''
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
4 import shutil
5077
88fc7202f87d Remove 'import *' statement from test/memorydb.py
John Kristensen <john@jerrykan.com>
parents: 4795
diff changeset
5 import os
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
6
5077
88fc7202f87d Remove 'import *' statement from test/memorydb.py
John Kristensen <john@jerrykan.com>
parents: 4795
diff changeset
7 from roundup import date
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
8 from roundup import hyperdb
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
9 from roundup import roundupdb
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
10 from roundup import security
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
11 from roundup import password
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
12 from roundup import configuration
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
13 from roundup.backends import back_anydbm
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
14 from roundup.backends import indexer_dbm
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
15 from roundup.backends import sessions_dbm
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
16 from roundup.backends import indexer_common
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
17 from roundup.support import ensureParentsExist
5462
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
18 from roundup.anypy.strings import s2b
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
19 from roundup.test.tx_Source_detector import init as tx_Source_init
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
20
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
21 default_prefix = '../../share/roundup/templates/classic'
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
22
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
23 def new_config(debug=False, prefix=default_prefix):
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
24 if not prefix.startswith('/'):
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
25 prefix = os.path.join (os.path.dirname(__file__), prefix)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
26 config = configuration.CoreConfig()
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
27 config.detectors = configuration.UserConfig(
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
28 os.path.join(prefix, "detectors/config.ini"))
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
29 config.ext = configuration.UserConfig(
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
30 os.path.join(prefix, "extensions/config.ini"))
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
31 config.DATABASE = "db"
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
32 #config.logging = MockNull()
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
33 # these TRACKER_WEB and MAIL_DOMAIN values are used in mailgw tests
4358
887a153cedec The email for the first message on an issue...
Richard Jones <richard@users.sourceforge.net>
parents: 4350
diff changeset
34 if debug:
887a153cedec The email for the first message on an issue...
Richard Jones <richard@users.sourceforge.net>
parents: 4350
diff changeset
35 config.LOGGING_LEVEL = "DEBUG"
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
36 config.MAIL_DOMAIN = "your.tracker.email.domain.example"
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
37 config.TRACKER_WEB = "http://tracker.example/cgi-bin/roundup.cgi/bugs/"
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
38 return config
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
39
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
40 def create(journaltag, create=True, debug=False, prefix=default_prefix):
6365
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
41 # "Nuke" in-memory db
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
42 db_nuke('')
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
43
4358
887a153cedec The email for the first message on an issue...
Richard Jones <richard@users.sourceforge.net>
parents: 4350
diff changeset
44 db = Database(new_config(debug), journaltag)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
45
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
46 # load standard schema
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
47 if not prefix.startswith('/'):
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
48 prefix = os.path.join (os.path.dirname(__file__), prefix)
6492
4351c95faace Fix more opens that aren't closed.
John Rouillard <rouilj@ieee.org>
parents: 6491
diff changeset
49
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
50 schema = os.path.join(prefix, 'schema.py')
5077
88fc7202f87d Remove 'import *' statement from test/memorydb.py
John Kristensen <john@jerrykan.com>
parents: 4795
diff changeset
51 vars = hyperdb.__dict__
88fc7202f87d Remove 'import *' statement from test/memorydb.py
John Kristensen <john@jerrykan.com>
parents: 4795
diff changeset
52 vars['Class'] = Class
88fc7202f87d Remove 'import *' statement from test/memorydb.py
John Kristensen <john@jerrykan.com>
parents: 4795
diff changeset
53 vars['FileClass'] = FileClass
88fc7202f87d Remove 'import *' statement from test/memorydb.py
John Kristensen <john@jerrykan.com>
parents: 4795
diff changeset
54 vars['IssueClass'] = IssueClass
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
55 vars['db'] = db
6492
4351c95faace Fix more opens that aren't closed.
John Rouillard <rouilj@ieee.org>
parents: 6491
diff changeset
56 fd = open(schema)
4351c95faace Fix more opens that aren't closed.
John Rouillard <rouilj@ieee.org>
parents: 6491
diff changeset
57 exec(compile(fd.read(), schema, 'exec'), vars)
4351c95faace Fix more opens that aren't closed.
John Rouillard <rouilj@ieee.org>
parents: 6491
diff changeset
58 fd.close()
4351c95faace Fix more opens that aren't closed.
John Rouillard <rouilj@ieee.org>
parents: 6491
diff changeset
59
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
60 initial_data = os.path.join(prefix, 'initial_data.py')
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
61 vars = dict(db=db, admin_email='admin@test.com',
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
62 adminpw=password.Password('sekrit'))
6492
4351c95faace Fix more opens that aren't closed.
John Rouillard <rouilj@ieee.org>
parents: 6491
diff changeset
63 fd = open(initial_data)
4351c95faace Fix more opens that aren't closed.
John Rouillard <rouilj@ieee.org>
parents: 6491
diff changeset
64 exec(compile(fd.read(), initial_data, 'exec'), vars)
6493
a7072cf4ad7e Actually call close().
John Rouillard <rouilj@ieee.org>
parents: 6492
diff changeset
65 fd.close()
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
66
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
67 # load standard detectors
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
68 dirname = os.path.join(prefix, 'detectors')
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
69 for fn in os.listdir(dirname):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
70 if not fn.endswith('.py'): continue
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
71 vars = {}
6491
087cae2fbcea Handle more ResourceWarning issues.
John Rouillard <rouilj@ieee.org>
parents: 6365
diff changeset
72 with open(os.path.join(dirname, fn)) as fd:
087cae2fbcea Handle more ResourceWarning issues.
John Rouillard <rouilj@ieee.org>
parents: 6365
diff changeset
73 exec(compile(fd.read(),
087cae2fbcea Handle more ResourceWarning issues.
John Rouillard <rouilj@ieee.org>
parents: 6365
diff changeset
74 os.path.join(dirname, fn), 'exec'), vars)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
75 vars['init'](db)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
76
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
77 tx_Source_init(db)
4781
6e9b9743de89 Implementation for:
John Rouillard <rouilj@ieee.org>
parents: 4570
diff changeset
78
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
79 '''
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
80 status = Class(db, "status", name=String())
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
81 status.setkey("name")
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
82 priority = Class(db, "priority", name=String(), order=String())
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
83 priority.setkey("name")
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
84 keyword = Class(db, "keyword", name=String(), order=String())
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
85 keyword.setkey("name")
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
86 user = Class(db, "user", username=String(), password=Password(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
87 assignable=Boolean(), age=Number(), roles=String(), address=String(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
88 supervisor=Link('user'),realname=String(),alternate_addresses=String())
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
89 user.setkey("username")
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
90 file = FileClass(db, "file", name=String(), type=String(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
91 comment=String(indexme="yes"), fooz=Password())
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
92 file_nidx = FileClass(db, "file_nidx", content=String(indexme='no'))
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
93 issue = IssueClass(db, "issue", title=String(indexme="yes"),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
94 status=Link("status"), nosy=Multilink("user"), deadline=Date(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
95 foo=Interval(), files=Multilink("file"), assignedto=Link('user'),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
96 priority=Link('priority'), spam=Multilink('msg'),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
97 feedback=Link('msg'))
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
98 stuff = Class(db, "stuff", stuff=String())
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
99 session = Class(db, 'session', title=String())
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
100 msg = FileClass(db, "msg", date=Date(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
101 author=Link("user", do_journal='no'),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
102 files=Multilink('file'), inreplyto=String(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
103 messageid=String(), summary=String(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
104 content=String(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
105 recipients=Multilink("user", do_journal='no')
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
106 )
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
107 '''
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
108 if create:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
109 db.user.create(username="fred", roles='User',
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
110 password=password.Password('sekrit'), address='fred@example.com')
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
111
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
112 db.security.addPermissionToRole('User', 'Email Access')
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
113 '''
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
114 db.security.addPermission(name='Register', klass='user')
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
115 db.security.addPermissionToRole('User', 'Web Access')
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
116 db.security.addPermissionToRole('Anonymous', 'Email Access')
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
117 db.security.addPermissionToRole('Anonymous', 'Register', 'user')
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
118 for cl in 'issue', 'file', 'msg', 'keyword':
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
119 db.security.addPermissionToRole('User', 'View', cl)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
120 db.security.addPermissionToRole('User', 'Edit', cl)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
121 db.security.addPermissionToRole('User', 'Create', cl)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
122 for cl in 'priority', 'status':
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
123 db.security.addPermissionToRole('User', 'View', cl)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
124 '''
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
125 return db
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
126
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
127 class cldb(dict):
5462
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
128 def __init__(self, **values):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
129 super(cldb, self).__init__()
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
130 for key, value in values.items():
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
131 super(cldb, self).__setitem__(s2b(key), value)
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
132 def __getitem__(self, key):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
133 return super(cldb, self).__getitem__(s2b(key))
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
134 def __setitem__(self, key, value):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
135 return super(cldb, self).__setitem__(s2b(key), value)
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
136 def __delitem__(self, key):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
137 return super(cldb, self).__delitem__(s2b(key))
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
138 def __contains__(self, key):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
139 return super(cldb, self).__contains__(s2b(key))
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
140 def close(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
141 pass
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
142
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
143 class BasicDatabase(dict):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
144 ''' Provide a nice encapsulation of an anydbm store.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
145
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
146 Keys are id strings, values are automatically marshalled data.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
147 '''
5462
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
148 def __init__(self, **values):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
149 super(BasicDatabase, self).__init__()
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
150 for k, v in values.items():
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
151 super(BasicDatabase, self).__setitem__(s2b(k), v)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
152 def __getitem__(self, key):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
153 if key not in self:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
154 d = self[key] = {}
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
155 return d
5462
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
156 return super(BasicDatabase, self).__getitem__(s2b(key))
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
157 def __setitem__(self, key, value):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
158 return super(BasicDatabase, self).__setitem__(s2b(key), value)
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
159 def __delitem__(self, key):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
160 return super(BasicDatabase, self).__delitem__(s2b(key))
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
161 def __contains__(self, key):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
162 return super(BasicDatabase, self).__contains__(s2b(key))
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
163 def exists(self, infoid):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
164 return infoid in self
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
165 def get(self, infoid, value, default=None):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
166 return self[infoid].get(value, default)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
167 def getall(self, infoid):
4389
355cd8fdeb32 memorydb API consistency fix
Richard Jones <richard@users.sourceforge.net>
parents: 4358
diff changeset
168 if infoid not in self:
355cd8fdeb32 memorydb API consistency fix
Richard Jones <richard@users.sourceforge.net>
parents: 4358
diff changeset
169 raise KeyError(infoid)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
170 return self[infoid]
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
171 def set(self, infoid, **newvalues):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
172 self[infoid].update(newvalues)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
173 def list(self):
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5385
diff changeset
174 return list(self.keys())
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
175 def destroy(self, infoid):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
176 del self[infoid]
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
177 def commit(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
178 pass
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
179 def close(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
180 pass
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
181 def updateTimestamp(self, sessid):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
182 pass
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
183 def clean(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
184 pass
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
185
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
186 class Sessions(BasicDatabase, sessions_dbm.Sessions):
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
187 name = 'sessions'
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
188
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
189 class OneTimeKeys(BasicDatabase, sessions_dbm.Sessions):
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
190 name = 'otks'
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
191
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
192 class Indexer(indexer_dbm.Indexer):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
193 def __init__(self, db):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
194 indexer_common.Indexer.__init__(self, db)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
195 self.reindex = 0
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
196 self.quiet = 9
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
197 self.changed = 0
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
198
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
199 def load_index(self, reload=0, wordlist=None):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
200 # Unless reload is indicated, do not load twice
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
201 if self.index_loaded() and not reload:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
202 return 0
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
203 self.words = {}
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
204 self.files = {'_TOP':(0,None)}
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
205 self.fileids = {}
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
206 self.changed = 0
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
207
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
208 def save_index(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
209 pass
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
210 def force_reindex(self):
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
211 # TODO I'm concerned that force_reindex may not be tested by
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
212 # testForcedReindexing if the functionality can just be removed
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
213 pass
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
214
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
215 class Database(back_anydbm.Database):
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
216 """A database for storing records containing flexible data types.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
217
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
218 Transaction stuff TODO:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
219
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
220 - check the timestamp of the class file and nuke the cache if it's
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
221 modified. Do some sort of conflict checking on the dirty stuff.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
222 - perhaps detect write collisions (related to above)?
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
223 """
5317
c0cda00af479 Fix tests for memorydb and anydbm backends
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5077
diff changeset
224
c0cda00af479 Fix tests for memorydb and anydbm backends
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5077
diff changeset
225 dbtype = "memorydb"
c0cda00af479 Fix tests for memorydb and anydbm backends
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5077
diff changeset
226
6365
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
227 # Make it a little more persistent across re-open
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
228 memdb = {}
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
229
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
230 def __init__(self, config, journaltag=None):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
231 self.config, self.journaltag = config, journaltag
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
232 self.classes = {}
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
233 self.files = {}
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
234 self.tx_files = {}
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
235 self.security = security.Security(self)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
236 self.stats = {'cache_hits': 0, 'cache_misses': 0, 'get_items': 0,
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
237 'filtering': 0}
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
238 self.sessions = Sessions()
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
239 self.otks = OneTimeKeys()
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
240 self.indexer = Indexer(self)
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
241
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
242 # anydbm bits
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
243 self.cache = {} # cache of nodes loaded or created
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
244 self.dirtynodes = {} # keep track of the dirty nodes by class
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
245 self.newnodes = {} # keep track of the new nodes by class
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
246 self.destroyednodes = {}# keep track of the destroyed nodes by class
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
247 self.transactions = []
4781
6e9b9743de89 Implementation for:
John Rouillard <rouilj@ieee.org>
parents: 4570
diff changeset
248 self.tx_Source = None
6365
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
249 # persistence across re-open
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
250 self.items = self.__class__.memdb.get('items', {})
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
251 self.ids = self.__class__.memdb.get('ids', {})
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
252 self.journals = self.__class__.memdb.get('journals', {})
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
253
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
254 def filename(self, classname, nodeid, property=None, create=0):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
255 shutil.copyfile(__file__, __file__+'.dummy')
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
256 return __file__+'.dummy'
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
257
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4346
diff changeset
258 def filesize(self, classname, nodeid, property=None, create=0):
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
259 return len(self.getfile(classname, nodeid, property))
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4346
diff changeset
260
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
261 def post_init(self):
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5972
diff changeset
262 super(Database, self).post_init()
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
263
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
264 def refresh_database(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
265 pass
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
266
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
267 def getSessionManager(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
268 return self.sessions
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
269
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
270 def getOTKManager(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
271 return self.otks
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
272
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
273 def reindex(self, classname=None, show_progress=False):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
274 pass
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
275
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
276 def __repr__(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
277 return '<memorydb instance at %x>'%id(self)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
278
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
279 def storefile(self, classname, nodeid, property, content):
5465
c4f9a152258d keep test code in sync with real implementation
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5462
diff changeset
280 if isinstance(content, str):
c4f9a152258d keep test code in sync with real implementation
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5462
diff changeset
281 content = s2b(content)
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
282 self.tx_files[classname, nodeid, property] = content
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
283 self.transactions.append((self.doStoreFile, (classname, nodeid,
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
284 property)))
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
285
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
286 def getfile(self, classname, nodeid, property):
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
287 if (classname, nodeid, property) in self.tx_files:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
288 return self.tx_files[classname, nodeid, property]
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
289 return self.files[classname, nodeid, property]
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
290
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
291 def doStoreFile(self, classname, nodeid, property, **databases):
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
292 self.files[classname, nodeid, property] = self.tx_files[classname, nodeid, property]
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
293 return (classname, nodeid)
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
294
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
295 def rollbackStoreFile(self, classname, nodeid, property, **databases):
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
296 del self.tx_files[classname, nodeid, property]
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
297
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
298 def numfiles(self):
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
299 return len(self.files) + len(self.tx_files)
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
300
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
301 def close(self):
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
302 self.clearCache()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
303 self.tx_files = {}
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
304 # kill the schema too
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
305 self.classes = {}
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
306 # just keep the .items
6365
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
307 # persistence across re-open
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
308 self.__class__.memdb['items'] = self.items
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
309 self.__class__.memdb['ids'] = self.ids
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
310 self.__class__.memdb['journals'] = self.journals
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
311
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
312 #
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
313 # Classes
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
314 #
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
315 def __getattr__(self, classname):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
316 """A convenient way of calling self.getclass(classname)."""
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
317 if classname in self.classes:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
318 return self.classes[classname]
5378
35ea9b1efc14 Python 3 preparation: "raise" syntax.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5317
diff changeset
319 raise AttributeError(classname)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
320
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
321 def addclass(self, cl):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
322 cn = cl.classname
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
323 if cn in self.classes:
6243
1e53e3ad9e5c Remove translation markings. Not needed.
John Rouillard <rouilj@ieee.org>
parents: 6241
diff changeset
324 raise ValueError('Class "%s" already defined.'%cn)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
325 self.classes[cn] = cl
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
326 if cn not in self.items:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
327 self.items[cn] = cldb()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
328 self.ids[cn] = 0
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
329
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
330 # add default Edit and View permissions
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
331 self.security.addPermission(name="Create", klass=cn,
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
332 description="User is allowed to create "+cn)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
333 self.security.addPermission(name="Edit", klass=cn,
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
334 description="User is allowed to edit "+cn)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
335 self.security.addPermission(name="View", klass=cn,
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
336 description="User is allowed to access "+cn)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
337
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
338 def getclasses(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
339 """Return a list of the names of all existing classes."""
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5385
diff changeset
340 return sorted(self.classes.keys())
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
341
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
342 def getclass(self, classname):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
343 """Get the Class object representing a particular class.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
344
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
345 If 'classname' is not a valid class name, a KeyError is raised.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
346 """
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
347 try:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
348 return self.classes[classname]
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
349 except KeyError:
5378
35ea9b1efc14 Python 3 preparation: "raise" syntax.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5317
diff changeset
350 raise KeyError('There is no class called "%s"'%classname)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
351
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
352 #
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
353 # Class DBs
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
354 #
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
355 def clear(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
356 self.items = {}
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
357
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
358 def getclassdb(self, classname, mode='r'):
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
359 """ grab a connection to the class db that will be used for
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
360 multiple actions
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
361 """
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
362 return self.items[classname]
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
363
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
364 def getCachedJournalDB(self, classname):
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
365 return self.journals.setdefault(classname, {})
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
366
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
367 #
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
368 # Node IDs
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
369 #
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
370 def newid(self, classname):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
371 self.ids[classname] += 1
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
372 return str(self.ids[classname])
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4346
diff changeset
373 def setid(self, classname, id):
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
374 self.ids[classname] = int(id)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
375
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
376 #
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
377 # Journal
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
378 #
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
379 def doSaveJournal(self, classname, nodeid, action, params, creator,
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
380 creation):
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
381 if creator is None:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
382 creator = self.getuid()
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
383 if creation is None:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
384 creation = date.Date()
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
385 self.journals.setdefault(classname, {}).setdefault(nodeid,
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
386 []).append((nodeid, creation, creator, action, params))
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
387
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
388 def doSetJournal(self, classname, nodeid, journal):
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
389 self.journals.setdefault(classname, {})[nodeid] = journal
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
390
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
391 def getjournal(self, classname, nodeid):
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
392 # our journal result
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
393 res = []
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
394
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
395 # add any journal entries for transactions not committed to the
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
396 # database
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
397 for method, args in self.transactions:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
398 if method != self.doSaveJournal:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
399 continue
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
400 (cache_classname, cache_nodeid, cache_action, cache_params,
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
401 cache_creator, cache_creation) = args
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
402 if cache_classname == classname and cache_nodeid == nodeid:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
403 if not cache_creator:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
404 cache_creator = self.getuid()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
405 if not cache_creation:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
406 cache_creation = date.Date()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
407 res.append((cache_nodeid, cache_creation, cache_creator,
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
408 cache_action, cache_params))
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
409 try:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
410 res += self.journals.get(classname, {})[nodeid]
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
411 except KeyError:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
412 if res: return res
5378
35ea9b1efc14 Python 3 preparation: "raise" syntax.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5317
diff changeset
413 raise IndexError(nodeid)
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
414 return res
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
415
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
416 def pack(self, pack_before):
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
417 """ Delete all journal entries except "create" before 'pack_before'.
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
418 """
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
419 pack_before = pack_before.serialise()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
420 for classname in self.journals:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
421 db = self.journals[classname]
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
422 for key in db:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
423 # get the journal for this db entry
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
424 l = []
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
425 last_set_entry = None
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
426 for entry in db[key]:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
427 # unpack the entry
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
428 (nodeid, date_stamp, self.journaltag, action,
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
429 params) = entry
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
430 date_stamp = date_stamp.serialise()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
431 # if the entry is after the pack date, _or_ the initial
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
432 # create entry, then it stays
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
433 if date_stamp > pack_before or action == 'create':
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
434 l.append(entry)
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
435 db[key] = l
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
436
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
437 class Class(back_anydbm.Class):
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
438 pass
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
439
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
440 class FileClass(back_anydbm.FileClass):
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
441 def __init__(self, db, classname, **properties):
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
442 if 'content' not in properties:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
443 properties['content'] = hyperdb.String(indexme='yes')
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
444 if 'type' not in properties:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
445 properties['type'] = hyperdb.String()
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
446 back_anydbm.Class.__init__(self, db, classname, **properties)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
447
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
448 def export_files(self, dirname, nodeid):
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
449 dest = self.exportFilename(dirname, nodeid)
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
450 ensureParentsExist(dest)
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
451 f = open(dest, 'wb')
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
452 f.write(self.db.files[self.classname, nodeid, None])
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
453 f.close()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
454
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
455 def import_files(self, dirname, nodeid):
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
456 source = self.exportFilename(dirname, nodeid)
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
457 f = open(source, 'rb')
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
458 self.db.files[self.classname, nodeid, None] = f.read()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
459 f.close()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
460 mime_type = None
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
461 props = self.getprops()
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
462 if 'type' in props:
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
463 mime_type = self.get(nodeid, 'type')
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
464 if not mime_type:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
465 mime_type = self.default_mime_type
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
466 if props['content'].indexme:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
467 self.db.indexer.add_text((self.classname, nodeid, 'content'),
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
468 self.get(nodeid, 'content'), mime_type)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
469
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
470 # deviation from spec - was called ItemClass
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
471 class IssueClass(Class, roundupdb.IssueClass):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
472 # Overridden methods:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
473 def __init__(self, db, classname, **properties):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
474 """The newly-created class automatically includes the "messages",
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
475 "files", "nosy", and "superseder" properties. If the 'properties'
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
476 dictionary attempts to specify any of these properties or a
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
477 "creation" or "activity" property, a ValueError is raised.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
478 """
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
479 if 'title' not in properties:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
480 properties['title'] = hyperdb.String(indexme='yes')
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
481 if 'messages' not in properties:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
482 properties['messages'] = hyperdb.Multilink("msg")
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
483 if 'files' not in properties:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
484 properties['files'] = hyperdb.Multilink("file")
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
485 if 'nosy' not in properties:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
486 # note: journalling is turned off as it really just wastes
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
487 # space. this behaviour may be overridden in an instance
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
488 properties['nosy'] = hyperdb.Multilink("user", do_journal="no")
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
489 if 'superseder' not in properties:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
490 properties['superseder'] = hyperdb.Multilink(classname)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
491 Class.__init__(self, db, classname, **properties)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
492
6365
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
493 # Methods to check for existence and nuke the db
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
494 # We don't support multiple named databases
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
495
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
496 def db_exists(name):
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
497 return bool(Database.memdb)
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
498
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
499 def db_nuke(name):
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
500 Database.memdb = {}
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
501
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
502 # vim: set et sts=4 sw=4 :

Roundup Issue Tracker: http://roundup-tracker.org/