Mercurial > p > roundup > code
annotate test/test_memorydb.py @ 6565:2c2dbfc332ba
Try to handle multiple connections better.
The session database is a hot spot. When multiple requests (e.g. 20)
come in at the same time session database contention can get great.
The original code didn't retry session database access when the open
failed. This resulted in errors at the client.
The second pass delayed 0.01 seconds and retried. It was better but we
still had multiple second stalls. I think the first request got in,
everybody else backed up and then retried at the same time. Again they
stepped on each other. With logging I would see many counters go all
the way to low single digits or to -1 indicating falure.
This pass uses randomint to generate delays from 0-.125 seconds in 5ms
increments. This performs better in testing. I rarely saw a counter
less than 13 (2 failed retries). Current logging starts after 6
failures and counts down until success or failure.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 16 Dec 2021 20:02:00 -0500 |
| parents | 7f00fc5958ca |
| children | 044dcf3608a2 |
| rev | line source |
|---|---|
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
1 import unittest, os, shutil, time |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
2 |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
3 from roundup import hyperdb |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
4 |
|
5388
d26921b851c3
Python 3 preparation: make relative imports explicit.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5319
diff
changeset
|
5 from .db_test_base import DBTest, ROTest, SchemaTest, config, setupSchema |
|
6360
a77a7d04ed23
Move memorydb from test to roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5388
diff
changeset
|
6 from roundup.test import memorydb |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
7 |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
8 class memorydbOpener: |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
9 module = memorydb |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
10 |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
11 def nuke_database(self): |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
12 # really kill it |
|
6365
7f00fc5958ca
Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6360
diff
changeset
|
13 memorydb.db_nuke('') |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
14 self.db = None |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
15 |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
16 db = None |
|
6365
7f00fc5958ca
Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6360
diff
changeset
|
17 def open_database(self, user='admin'): |
|
7f00fc5958ca
Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6360
diff
changeset
|
18 if self.db: |
|
7f00fc5958ca
Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6360
diff
changeset
|
19 self.db.close() |
|
7f00fc5958ca
Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6360
diff
changeset
|
20 self.db = self.module.Database(config, user) |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
21 return self.db |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
22 |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
23 def setUp(self): |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
24 self.open_database() |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
25 setupSchema(self.db, 1, self.module) |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
26 |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
27 def tearDown(self): |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
28 if self.db is not None: |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
29 self.db.close() |
|
6365
7f00fc5958ca
Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6360
diff
changeset
|
30 self.db = None |
|
7f00fc5958ca
Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6360
diff
changeset
|
31 self.nuke_database() |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
32 |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
33 # nuke and re-create db for restore |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
34 def nukeAndCreate(self): |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
35 self.db.close() |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
36 self.nuke_database() |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
37 self.db = self.module.Database(config, 'admin') |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
38 setupSchema(self.db, 0, self.module) |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
39 |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4570
diff
changeset
|
40 |
|
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4570
diff
changeset
|
41 class memorydbDBTest(memorydbOpener, DBTest, unittest.TestCase): |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
42 pass |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
43 |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4570
diff
changeset
|
44 |
|
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4570
diff
changeset
|
45 class memorydbROTest(memorydbOpener, ROTest, unittest.TestCase): |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
46 def setUp(self): |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
47 self.db = self.module.Database(config) |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
48 setupSchema(self.db, 0, self.module) |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
49 |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4570
diff
changeset
|
50 |
|
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4570
diff
changeset
|
51 class memorydbSchemaTest(memorydbOpener, SchemaTest, unittest.TestCase): |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
52 pass |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
53 |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4570
diff
changeset
|
54 |
|
5388
d26921b851c3
Python 3 preparation: make relative imports explicit.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5319
diff
changeset
|
55 from .session_common import SessionTest |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5037
diff
changeset
|
56 class memorydbSessionTest(memorydbOpener, SessionTest, unittest.TestCase): |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
57 def setUp(self): |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
58 self.db = self.module.Database(config, 'admin') |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
59 setupSchema(self.db, 1, self.module) |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
60 self.sessions = self.db.sessions |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
61 |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
62 # vim: set filetype=python ts=4 sw=4 et si |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
63 |
