Mercurial > p > roundup > code

annotate roundup/test/memorydb.py @ 7683:b04e222501b8

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
fix: rest - set self.start from client.start Make elasped time include time since client was initialized. So elapsed is as close as we can get to an overall request service time. May need to add rest_elapsed or some other subsystem based timers as we try to track a possible performance regression in 2.3.0.
author John Rouillard <rouilj@ieee.org>
date Wed, 25 Oct 2023 13:12:18 -0400
parents 506c86823abb
children 617d85ce4ac3
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1 '''Implement an in-memory hyperdb for testing purposes.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2 '''
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
4 import shutil
5077
88fc7202f87d Remove 'import *' statement from test/memorydb.py
John Kristensen <john@jerrykan.com>
parents: 4795
diff changeset
5 import os
6808
375d40a9e730 Add tests to BasicDatabase and merge implementations
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
6 import time
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
7
5077
88fc7202f87d Remove 'import *' statement from test/memorydb.py
John Kristensen <john@jerrykan.com>
parents: 4795
diff changeset
8 from roundup import date
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
9 from roundup import hyperdb
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
10 from roundup import roundupdb
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
11 from roundup import security
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
12 from roundup import password
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
13 from roundup import configuration
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
14 from roundup.backends import back_anydbm
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
15 from roundup.backends import indexer_dbm
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
16 from roundup.backends import sessions_dbm
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
17 from roundup.backends import indexer_common
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
18 from roundup.support import ensureParentsExist
5462
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
19 from roundup.anypy.strings import s2b
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
20 from roundup.test.tx_Source_detector import init as tx_Source_init
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
21
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
22 default_prefix = '../../share/roundup/templates/classic'
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
23
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
24
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
25 def new_config(debug=False, prefix=default_prefix):
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
26 if not prefix.startswith('/'):
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
27 prefix = os.path.join(os.path.dirname(__file__), prefix)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
28 config = configuration.CoreConfig()
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
29 config.detectors = configuration.UserConfig(
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
30 os.path.join(prefix, "detectors/config.ini"))
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
31 config.ext = configuration.UserConfig(
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
32 os.path.join(prefix, "extensions/config.ini"))
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
33 config.DATABASE = "db"
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
34 # config.logging = MockNull()
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
35 # these TRACKER_WEB and MAIL_DOMAIN values are used in mailgw tests
4358
887a153cedec The email for the first message on an issue...
Richard Jones <richard@users.sourceforge.net>
parents: 4350
diff changeset
36 if debug:
887a153cedec The email for the first message on an issue...
Richard Jones <richard@users.sourceforge.net>
parents: 4350
diff changeset
37 config.LOGGING_LEVEL = "DEBUG"
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
38 config.MAIL_DOMAIN = "your.tracker.email.domain.example"
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
39 config.TRACKER_WEB = "http://tracker.example/cgi-bin/roundup.cgi/bugs/"
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
40 return config
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
41
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
42
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
43 def create(journaltag, create=True, debug=False, prefix=default_prefix):
6365
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
44 # "Nuke" in-memory db
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
45 db_nuke('')
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
46
4358
887a153cedec The email for the first message on an issue...
Richard Jones <richard@users.sourceforge.net>
parents: 4350
diff changeset
47 db = Database(new_config(debug), journaltag)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
48
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
49 # load standard schema
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
50 if not prefix.startswith('/'):
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
51 prefix = os.path.join(os.path.dirname(__file__), prefix)
6492
4351c95faace Fix more opens that aren't closed.
John Rouillard <rouilj@ieee.org>
parents: 6491
diff changeset
52
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
53 schema = os.path.join(prefix, 'schema.py')
5077
88fc7202f87d Remove 'import *' statement from test/memorydb.py
John Kristensen <john@jerrykan.com>
parents: 4795
diff changeset
54 vars = hyperdb.__dict__
88fc7202f87d Remove 'import *' statement from test/memorydb.py
John Kristensen <john@jerrykan.com>
parents: 4795
diff changeset
55 vars['Class'] = Class
88fc7202f87d Remove 'import *' statement from test/memorydb.py
John Kristensen <john@jerrykan.com>
parents: 4795
diff changeset
56 vars['FileClass'] = FileClass
88fc7202f87d Remove 'import *' statement from test/memorydb.py
John Kristensen <john@jerrykan.com>
parents: 4795
diff changeset
57 vars['IssueClass'] = IssueClass
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
58 vars['db'] = db
6492
4351c95faace Fix more opens that aren't closed.
John Rouillard <rouilj@ieee.org>
parents: 6491
diff changeset
59 fd = open(schema)
4351c95faace Fix more opens that aren't closed.
John Rouillard <rouilj@ieee.org>
parents: 6491
diff changeset
60 exec(compile(fd.read(), schema, 'exec'), vars)
4351c95faace Fix more opens that aren't closed.
John Rouillard <rouilj@ieee.org>
parents: 6491
diff changeset
61 fd.close()
4351c95faace Fix more opens that aren't closed.
John Rouillard <rouilj@ieee.org>
parents: 6491
diff changeset
62
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
63 initial_data = os.path.join(prefix, 'initial_data.py')
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
64 vars = dict(db=db, admin_email='admin@test.com',
7211
506c86823abb Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents: 6937
diff changeset
65 adminpw=password.Password('sekrit', config=db.config))
6492
4351c95faace Fix more opens that aren't closed.
John Rouillard <rouilj@ieee.org>
parents: 6491
diff changeset
66 fd = open(initial_data)
4351c95faace Fix more opens that aren't closed.
John Rouillard <rouilj@ieee.org>
parents: 6491
diff changeset
67 exec(compile(fd.read(), initial_data, 'exec'), vars)
6493
a7072cf4ad7e Actually call close().
John Rouillard <rouilj@ieee.org>
parents: 6492
diff changeset
68 fd.close()
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
69
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
70 # load standard detectors
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
71 dirname = os.path.join(prefix, 'detectors')
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
72 for fn in os.listdir(dirname):
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
73 if not fn.endswith('.py'): continue # noqa: E701
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
74 vars = {}
6491
087cae2fbcea Handle more ResourceWarning issues.
John Rouillard <rouilj@ieee.org>
parents: 6365
diff changeset
75 with open(os.path.join(dirname, fn)) as fd:
087cae2fbcea Handle more ResourceWarning issues.
John Rouillard <rouilj@ieee.org>
parents: 6365
diff changeset
76 exec(compile(fd.read(),
087cae2fbcea Handle more ResourceWarning issues.
John Rouillard <rouilj@ieee.org>
parents: 6365
diff changeset
77 os.path.join(dirname, fn), 'exec'), vars)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
78 vars['init'](db)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
79
6361
58817c3bf471 Fix roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6360
diff changeset
80 tx_Source_init(db)
4781
6e9b9743de89 Implementation for:
John Rouillard <rouilj@ieee.org>
parents: 4570
diff changeset
81
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
82 '''
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
83 status = Class(db, "status", name=String())
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
84 status.setkey("name")
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
85 priority = Class(db, "priority", name=String(), order=String())
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
86 priority.setkey("name")
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
87 keyword = Class(db, "keyword", name=String(), order=String())
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
88 keyword.setkey("name")
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
89 user = Class(db, "user", username=String(), password=Password(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
90 assignable=Boolean(), age=Number(), roles=String(), address=String(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
91 supervisor=Link('user'),realname=String(),alternate_addresses=String())
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
92 user.setkey("username")
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
93 file = FileClass(db, "file", name=String(), type=String(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
94 comment=String(indexme="yes"), fooz=Password())
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
95 file_nidx = FileClass(db, "file_nidx", content=String(indexme='no'))
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
96 issue = IssueClass(db, "issue", title=String(indexme="yes"),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
97 status=Link("status"), nosy=Multilink("user"), deadline=Date(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
98 foo=Interval(), files=Multilink("file"), assignedto=Link('user'),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
99 priority=Link('priority'), spam=Multilink('msg'),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
100 feedback=Link('msg'))
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
101 stuff = Class(db, "stuff", stuff=String())
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
102 session = Class(db, 'session', title=String())
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
103 msg = FileClass(db, "msg", date=Date(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
104 author=Link("user", do_journal='no'),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
105 files=Multilink('file'), inreplyto=String(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
106 messageid=String(), summary=String(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
107 content=String(),
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
108 recipients=Multilink("user", do_journal='no')
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
109 )
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
110 '''
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
111 if create:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
112 db.user.create(username="fred", roles='User',
7211
506c86823abb Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents: 6937
diff changeset
113 password=password.Password('sekrit',
506c86823abb Add config argument to more password.Password invocations.
John Rouillard <rouilj@ieee.org>
parents: 6937
diff changeset
114 config=db.config),
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
115 address='fred@example.com')
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
116
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
117 db.security.addPermissionToRole('User', 'Email Access')
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
118 '''
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
119 db.security.addPermission(name='Register', klass='user')
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
120 db.security.addPermissionToRole('User', 'Web Access')
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
121 db.security.addPermissionToRole('Anonymous', 'Email Access')
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
122 db.security.addPermissionToRole('Anonymous', 'Register', 'user')
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
123 for cl in 'issue', 'file', 'msg', 'keyword':
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
124 db.security.addPermissionToRole('User', 'View', cl)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
125 db.security.addPermissionToRole('User', 'Edit', cl)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
126 db.security.addPermissionToRole('User', 'Create', cl)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
127 for cl in 'priority', 'status':
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
128 db.security.addPermissionToRole('User', 'View', cl)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
129 '''
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
130 return db
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
131
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
132
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
133 class cldb(dict):
5462
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
134 def __init__(self, **values):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
135 super(cldb, self).__init__()
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
136 for key, value in values.items():
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
137 super(cldb, self).__setitem__(s2b(key), value)
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
138
5462
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
139 def __getitem__(self, key):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
140 return super(cldb, self).__getitem__(s2b(key))
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
141
5462
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
142 def __setitem__(self, key, value):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
143 return super(cldb, self).__setitem__(s2b(key), value)
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
144
5462
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
145 def __delitem__(self, key):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
146 return super(cldb, self).__delitem__(s2b(key))
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
147
5462
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
148 def __contains__(self, key):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
149 return super(cldb, self).__contains__(s2b(key))
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
150
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
151 def close(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
152 pass
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
153
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
154
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
155 class BasicDatabase(dict):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
156 ''' Provide a nice encapsulation of an anydbm store.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
157
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
158 Keys are id strings, values are automatically marshalled data.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
159 '''
5462
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
160 def __init__(self, **values):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
161 super(BasicDatabase, self).__init__()
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
162 for k, v in values.items():
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
163 super(BasicDatabase, self).__setitem__(s2b(k), v)
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
164
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
165 def __getitem__(self, key):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
166 if key not in self:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
167 d = self[key] = {}
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
168 return d
5462
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
169 return super(BasicDatabase, self).__getitem__(s2b(key))
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
170
5462
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
171 def __setitem__(self, key, value):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
172 return super(BasicDatabase, self).__setitem__(s2b(key), value)
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
173
5462
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
174 def __delitem__(self, key):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
175 return super(BasicDatabase, self).__delitem__(s2b(key))
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
176
5462
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
177 def __contains__(self, key):
d0fd8bb64c3f fixed memorydb string encoding
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5395
diff changeset
178 return super(BasicDatabase, self).__contains__(s2b(key))
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
179
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
180 def exists(self, infoid):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
181 return infoid in self
6816
a4652d809ec5 implement default argument return in BasicDatabase.get()
John Rouillard <rouilj@ieee.org>
parents: 6808
diff changeset
182 _marker = []
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
183
6816
a4652d809ec5 implement default argument return in BasicDatabase.get()
John Rouillard <rouilj@ieee.org>
parents: 6808
diff changeset
184 def get(self, infoid, value, default=_marker):
6808
375d40a9e730 Add tests to BasicDatabase and merge implementations
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
185 if infoid not in self:
6816
a4652d809ec5 implement default argument return in BasicDatabase.get()
John Rouillard <rouilj@ieee.org>
parents: 6808
diff changeset
186 if default is self._marker:
a4652d809ec5 implement default argument return in BasicDatabase.get()
John Rouillard <rouilj@ieee.org>
parents: 6808
diff changeset
187 raise KeyError
a4652d809ec5 implement default argument return in BasicDatabase.get()
John Rouillard <rouilj@ieee.org>
parents: 6808
diff changeset
188 else:
a4652d809ec5 implement default argument return in BasicDatabase.get()
John Rouillard <rouilj@ieee.org>
parents: 6808
diff changeset
189 return default
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
190 return self[infoid].get(value, default)
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
191
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
192 def getall(self, infoid):
4389
355cd8fdeb32 memorydb API consistency fix
Richard Jones <richard@users.sourceforge.net>
parents: 4358
diff changeset
193 if infoid not in self:
355cd8fdeb32 memorydb API consistency fix
Richard Jones <richard@users.sourceforge.net>
parents: 4358
diff changeset
194 raise KeyError(infoid)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
195 return self[infoid]
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
196
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
197 def set(self, infoid, **newvalues):
6808
375d40a9e730 Add tests to BasicDatabase and merge implementations
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
198 if '__timestamp' in newvalues:
375d40a9e730 Add tests to BasicDatabase and merge implementations
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
199 try:
375d40a9e730 Add tests to BasicDatabase and merge implementations
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
200 float(newvalues['__timestamp'])
375d40a9e730 Add tests to BasicDatabase and merge implementations
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
201 except ValueError:
375d40a9e730 Add tests to BasicDatabase and merge implementations
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
202 if infoid in self:
375d40a9e730 Add tests to BasicDatabase and merge implementations
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
203 del(newvalues['__timestamp'])
375d40a9e730 Add tests to BasicDatabase and merge implementations
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
204 else:
375d40a9e730 Add tests to BasicDatabase and merge implementations
John Rouillard <rouilj@ieee.org>
parents: 6658
diff changeset
205 newvalues['__timestamp'] = time.time()
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
206 self[infoid].update(newvalues)
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
207
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
208 def list(self):
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5385
diff changeset
209 return list(self.keys())
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
210
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
211 def destroy(self, infoid):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
212 del self[infoid]
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
213
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
214 def commit(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
215 pass
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
216
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
217 def close(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
218 pass
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
219
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
220 def updateTimestamp(self, sessid):
6828
2ce855803633 Implement UpdateTimestamp for memorydb and test.
John Rouillard <rouilj@ieee.org>
parents: 6822
diff changeset
221 sess = self.get(sessid, '__timestamp', None)
2ce855803633 Implement UpdateTimestamp for memorydb and test.
John Rouillard <rouilj@ieee.org>
parents: 6822
diff changeset
222 now = time.time()
2ce855803633 Implement UpdateTimestamp for memorydb and test.
John Rouillard <rouilj@ieee.org>
parents: 6822
diff changeset
223 if sess is None or now > sess + 60:
2ce855803633 Implement UpdateTimestamp for memorydb and test.
John Rouillard <rouilj@ieee.org>
parents: 6822
diff changeset
224 self.set(sessid, __timestamp=now)
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
225
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
226 def clean(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
227 pass
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
228
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
229
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
230 class Sessions(BasicDatabase, sessions_dbm.Sessions):
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
231 name = 'sessions'
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
232
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
233
6822
5053ee6c846b memorydb fixes for otks tests.
John Rouillard <rouilj@ieee.org>
parents: 6816
diff changeset
234 class OneTimeKeys(BasicDatabase, sessions_dbm.OneTimeKeys):
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
235 name = 'otks'
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
236
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
237
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
238 class Indexer(indexer_dbm.Indexer):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
239 def __init__(self, db):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
240 indexer_common.Indexer.__init__(self, db)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
241 self.reindex = 0
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
242 self.quiet = 9
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
243 self.changed = 0
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
244
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
245 def load_index(self, reload=0, wordlist=None):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
246 # Unless reload is indicated, do not load twice
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
247 if self.index_loaded() and not reload:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
248 return 0
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
249 self.words = {}
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
250 self.files = {'_TOP': (0, None)}
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
251 self.fileids = {}
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
252 self.changed = 0
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
253
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
254 def save_index(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
255 pass
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
256
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
257 def force_reindex(self):
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
258 # TODO I'm concerned that force_reindex may not be tested by
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
259 # testForcedReindexing if the functionality can just be removed
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
260 pass
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
261
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
262
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
263 class Database(back_anydbm.Database):
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
264 """A database for storing records containing flexible data types.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
265
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
266 Transaction stuff TODO:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
267
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
268 - check the timestamp of the class file and nuke the cache if it's
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
269 modified. Do some sort of conflict checking on the dirty stuff.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
270 - perhaps detect write collisions (related to above)?
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
271 """
5317
c0cda00af479 Fix tests for memorydb and anydbm backends
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5077
diff changeset
272
c0cda00af479 Fix tests for memorydb and anydbm backends
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5077
diff changeset
273 dbtype = "memorydb"
c0cda00af479 Fix tests for memorydb and anydbm backends
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5077
diff changeset
274
6365
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
275 # Make it a little more persistent across re-open
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
276 memdb = {}
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
277
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
278 def __init__(self, config, journaltag=None):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
279 self.config, self.journaltag = config, journaltag
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
280 self.classes = {}
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
281 self.files = {}
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
282 self.tx_files = {}
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
283 self.security = security.Security(self)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
284 self.stats = {'cache_hits': 0, 'cache_misses': 0, 'get_items': 0,
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
285 'filtering': 0}
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
286 self.sessions = Sessions()
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
287 self.otks = OneTimeKeys()
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
288 self.indexer = Indexer(self)
6658
408fd477761f Add i18n object to roundupdb.Database
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6493
diff changeset
289 roundupdb.Database.__init__(self)
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
290
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
291 # anydbm bits
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
292 self.cache = {} # cache of nodes loaded or created
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
293 self.dirtynodes = {} # keep track of the dirty nodes by class
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
294 self.newnodes = {} # keep track of the new nodes by class
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
295 self.destroyednodes = {} # keep track of the destroyed nodes by class
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
296 self.transactions = []
4781
6e9b9743de89 Implementation for:
John Rouillard <rouilj@ieee.org>
parents: 4570
diff changeset
297 self.tx_Source = None
6365
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
298 # persistence across re-open
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
299 self.items = self.__class__.memdb.get('items', {})
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
300 self.ids = self.__class__.memdb.get('ids', {})
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
301 self.journals = self.__class__.memdb.get('journals', {})
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
302
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
303 def filename(self, classname, nodeid, property=None, create=0):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
304 shutil.copyfile(__file__, __file__+'.dummy')
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
305 return __file__+'.dummy'
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
306
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4346
diff changeset
307 def filesize(self, classname, nodeid, property=None, create=0):
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
308 return len(self.getfile(classname, nodeid, property))
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4346
diff changeset
309
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
310 def post_init(self):
6148
8497bf3f23a1 Allow to define reverse Multilinks
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5972
diff changeset
311 super(Database, self).post_init()
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
312
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
313 def refresh_database(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
314 pass
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
315
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
316 def getSessionManager(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
317 return self.sessions
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
318
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
319 def getOTKManager(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
320 return self.otks
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
321
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
322 def reindex(self, classname=None, show_progress=False):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
323 pass
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
324
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
325 def __repr__(self):
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
326 return '<memorydb instance at %x>' % id(self)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
327
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
328 def storefile(self, classname, nodeid, property, content):
5465
c4f9a152258d keep test code in sync with real implementation
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5462
diff changeset
329 if isinstance(content, str):
c4f9a152258d keep test code in sync with real implementation
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5462
diff changeset
330 content = s2b(content)
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
331 self.tx_files[classname, nodeid, property] = content
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
332 self.transactions.append((self.doStoreFile, (classname, nodeid,
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
333 property)))
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
334
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
335 def getfile(self, classname, nodeid, property):
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
336 if (classname, nodeid, property) in self.tx_files:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
337 return self.tx_files[classname, nodeid, property]
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
338 return self.files[classname, nodeid, property]
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
339
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
340 def doStoreFile(self, classname, nodeid, property, **databases):
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
341 self.files[classname, nodeid, property] = self.tx_files[classname, nodeid, property]
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
342 return (classname, nodeid)
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
343
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
344 def rollbackStoreFile(self, classname, nodeid, property, **databases):
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
345 del self.tx_files[classname, nodeid, property]
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
346
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
347 def numfiles(self):
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
348 return len(self.files) + len(self.tx_files)
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
350 def close(self):
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
351 self.clearCache()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
352 self.tx_files = {}
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
353 # kill the schema too
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
354 self.classes = {}
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
355 # just keep the .items
6365
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
356 # persistence across re-open
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
357 self.__class__.memdb['items'] = self.items
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
358 self.__class__.memdb['ids'] = self.ids
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
359 self.__class__.memdb['journals'] = self.journals
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
360
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
361 #
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
362 # Classes
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
363 #
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
364 def __getattr__(self, classname):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
365 """A convenient way of calling self.getclass(classname)."""
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
366 if classname in self.classes:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
367 return self.classes[classname]
5378
35ea9b1efc14 Python 3 preparation: "raise" syntax.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5317
diff changeset
368 raise AttributeError(classname)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
369
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
370 def addclass(self, cl):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
371 cn = cl.classname
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
372 if cn in self.classes:
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
373 raise ValueError('Class "%s" already defined.' % cn)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
374 self.classes[cn] = cl
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
375 if cn not in self.items:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
376 self.items[cn] = cldb()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
377 self.ids[cn] = 0
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
378
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
379 # add default Edit and View permissions
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
380 self.security.addPermission(name="Create", klass=cn,
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
381 description="User is allowed to create " +
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
382 cn)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
383 self.security.addPermission(name="Edit", klass=cn,
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
384 description="User is allowed to edit " +
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
385 cn)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
386 self.security.addPermission(name="View", klass=cn,
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
387 description="User is allowed to access " +
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
388 cn)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
389
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
390 def getclasses(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
391 """Return a list of the names of all existing classes."""
5395
23b8e6067f7c Python 3 preparation: update calls to dict methods.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5385
diff changeset
392 return sorted(self.classes.keys())
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
393
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
394 def getclass(self, classname):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
395 """Get the Class object representing a particular class.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
396
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
397 If 'classname' is not a valid class name, a KeyError is raised.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
398 """
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
399 try:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
400 return self.classes[classname]
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
401 except KeyError:
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
402 raise KeyError('There is no class called "%s"' % classname)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
403
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
404 #
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
405 # Class DBs
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
406 #
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
407 def clear(self):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
408 self.items = {}
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
409
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
410 def getclassdb(self, classname, mode='r'):
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
411 """ grab a connection to the class db that will be used for
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
412 multiple actions
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
413 """
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
414 return self.items[classname]
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
415
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
416 def getCachedJournalDB(self, classname):
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
417 return self.journals.setdefault(classname, {})
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
418
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
419 #
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
420 # Node IDs
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
421 #
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
422 def newid(self, classname):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
423 self.ids[classname] += 1
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
424 return str(self.ids[classname])
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
425
4347
0e33bf5571dc make some more memorydb tests pass
Richard Jones <richard@users.sourceforge.net>
parents: 4346
diff changeset
426 def setid(self, classname, id):
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
427 self.ids[classname] = int(id)
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
428
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
429 #
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
430 # Journal
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
431 #
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
432 def doSaveJournal(self, classname, nodeid, action, params, creator,
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
433 creation):
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
434 if creator is None:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
435 creator = self.getuid()
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
436 if creation is None:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
437 creation = date.Date()
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
438 self.journals.setdefault(classname, {}).setdefault(
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
439 nodeid, []).append((nodeid, creation, creator, action, params))
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
440
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
441 def doSetJournal(self, classname, nodeid, journal):
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
442 self.journals.setdefault(classname, {})[nodeid] = journal
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
443
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
444 def getjournal(self, classname, nodeid):
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
445 # our journal result
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
446 res = []
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
447
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
448 # add any journal entries for transactions not committed to the
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
449 # database
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
450 for method, args in self.transactions:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
451 if method != self.doSaveJournal:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
452 continue
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
453 (cache_classname, cache_nodeid, cache_action, cache_params,
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
454 cache_creator, cache_creation) = args
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
455 if cache_classname == classname and cache_nodeid == nodeid:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
456 if not cache_creator:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
457 cache_creator = self.getuid()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
458 if not cache_creation:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
459 cache_creation = date.Date()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
460 res.append((cache_nodeid, cache_creation, cache_creator,
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
461 cache_action, cache_params))
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
462 try:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
463 res += self.journals.get(classname, {})[nodeid]
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
464 except KeyError:
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
465 if res: return res # noqa: E701
5378
35ea9b1efc14 Python 3 preparation: "raise" syntax.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5317
diff changeset
466 raise IndexError(nodeid)
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
467 return res
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
468
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
469 def pack(self, pack_before):
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
470 """ Delete all journal entries except "create" before 'pack_before'.
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
471 """
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
472 pack_before = pack_before.serialise()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
473 for classname in self.journals:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
474 db = self.journals[classname]
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
475 for key in db:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
476 # get the journal for this db entry
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
477 kept_journals = []
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
478 for entry in db[key]:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
479 # unpack the entry
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
480 (nodeid, date_stamp, self.journaltag, action,
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
481 params) = entry
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
482 date_stamp = date_stamp.serialise()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
483 # if the entry is after the pack date, _or_ the initial
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
484 # create entry, then it stays
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
485 if date_stamp > pack_before or action == 'create':
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
486 kept_journals.append(entry)
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
487 db[key] = kept_journals
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
488
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
489
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
490 class Class(back_anydbm.Class):
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
491 pass
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
492
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
493
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
494 class FileClass(back_anydbm.FileClass):
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
495 def __init__(self, db, classname, **properties):
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
496 if 'content' not in properties:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
497 properties['content'] = hyperdb.String(indexme='yes')
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
498 if 'type' not in properties:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
499 properties['type'] = hyperdb.String()
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
500 back_anydbm.Class.__init__(self, db, classname, **properties)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
501
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
502 def export_files(self, dirname, nodeid):
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
503 dest = self.exportFilename(dirname, nodeid)
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
504 ensureParentsExist(dest)
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
505 f = open(dest, 'wb')
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
506 f.write(self.db.files[self.classname, nodeid, None])
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
507 f.close()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
508
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
509 def import_files(self, dirname, nodeid):
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
510 source = self.exportFilename(dirname, nodeid)
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
511 f = open(source, 'rb')
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
512 self.db.files[self.classname, nodeid, None] = f.read()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
513 f.close()
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
514 mime_type = None
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
515 props = self.getprops()
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
516 if 'type' in props:
4349
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
517 mime_type = self.get(nodeid, 'type')
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
518 if not mime_type:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
519 mime_type = self.default_mime_type
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
520 if props['content'].indexme:
f0faef4dd023 Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents: 4347
diff changeset
521 self.db.indexer.add_text((self.classname, nodeid, 'content'),
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
522 self.get(nodeid, 'content'), mime_type)
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
523
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
524
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
525 # deviation from spec - was called ItemClass
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
526 class IssueClass(Class, roundupdb.IssueClass):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
527 # Overridden methods:
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
528 def __init__(self, db, classname, **properties):
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
529 """The newly-created class automatically includes the "messages",
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
530 "files", "nosy", and "superseder" properties. If the 'properties'
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
531 dictionary attempts to specify any of these properties or a
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
532 "creation" or "activity" property, a ValueError is raised.
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
533 """
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
534 if 'title' not in properties:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
535 properties['title'] = hyperdb.String(indexme='yes')
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
536 if 'messages' not in properties:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
537 properties['messages'] = hyperdb.Multilink("msg")
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
538 if 'files' not in properties:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
539 properties['files'] = hyperdb.Multilink("file")
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
540 if 'nosy' not in properties:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
541 # note: journalling is turned off as it really just wastes
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
542 # space. this behaviour may be overridden in an instance
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
543 properties['nosy'] = hyperdb.Multilink("user", do_journal="no")
5381
0942fe89e82e Python 3 preparation: change "x.has_key(y)" to "y in x".
Joseph Myers <jsm@polyomino.org.uk>
parents: 5378
diff changeset
544 if 'superseder' not in properties:
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
545 properties['superseder'] = hyperdb.Multilink(classname)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
546 Class.__init__(self, db, classname, **properties)
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
547
6365
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
548 # Methods to check for existence and nuke the db
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
549 # We don't support multiple named databases
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
550
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
551
6365
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
552 def db_exists(name):
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
553 return bool(Database.memdb)
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
554
6937
b1130680d4ed flake8 fixes
John Rouillard <rouilj@ieee.org>
parents: 6828
diff changeset
555
6365
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
556 def db_nuke(name):
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
557 Database.memdb = {}
7f00fc5958ca Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents: 6361
diff changeset
558
4342
94c992852f12 add in-memory hyperdb implementation to speed up testing
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
559 # vim: set et sts=4 sw=4 :
Roundup Issue Tracker: http://roundup-tracker.org/