Mercurial > p > roundup > code
annotate test/test_memorydb.py @ 7668:5b41018617f2
fix: out of memory error when importing under postgresql
If you try importing more than 20k items under postgresql you can run
out of memory:
psycopg2.errors.OutOfMemory: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
Tuning memory may help, it's unknown at this point.
This checkin forces a commit to the postgres database after 10,000
rows have been added. This clears out the savepoints for each row and
starts a new transaction.
back_postgresql.py:
Implement commit mechanism in checkpoint_data(). Add two class level
attributes for tracking the number of savepoints and the limit when
the commit should happen.
roundup_admin.py:
implement pragma and dynamically create the config item
RDBMS_SAVEPOINT_LIMIT used by checkpoint_data.
Also fixed formatting of descriptions when using pragma list in
verbose mode.
admin_guide.txt, upgrading.txt:
Document change and use of pragma savepoint_limit in roundup-admin
for changing the default of 10,000.
test/db_test_base.py:
add some more asserts. In existing testAdminImportExport, set the
savepoint limit to 5 to test setting method and so that the commit
code will be run by existing tests. This provides coverage, but
does not actually test that the commit is done every 5 savepoints
8-(. The verification of every 5 savepoints was done manually
using a pdb breakpoint just before the commit.
acknowledgements.txt:
Added 2.4.0 section mentioning Norbert as he has done a ton of
testing with much larger datasets than I can test with.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 19 Oct 2023 16:11:25 -0400 |
| parents | 2ce855803633 |
| children |
| rev | line source |
|---|---|
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
1 import unittest, os, shutil, time |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
2 |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
3 from roundup import hyperdb |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
4 |
|
5388
d26921b851c3
Python 3 preparation: make relative imports explicit.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5319
diff
changeset
|
5 from .db_test_base import DBTest, ROTest, SchemaTest, config, setupSchema |
|
6360
a77a7d04ed23
Move memorydb from test to roundup/test
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5388
diff
changeset
|
6 from roundup.test import memorydb |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
7 |
| 6802 | 8 from roundup.anypy import strings |
| 9 | |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
10 class memorydbOpener: |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
11 module = memorydb |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
12 |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
13 def nuke_database(self): |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
14 # really kill it |
|
6365
7f00fc5958ca
Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6360
diff
changeset
|
15 memorydb.db_nuke('') |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
16 self.db = None |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
17 |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
18 db = None |
|
6365
7f00fc5958ca
Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6360
diff
changeset
|
19 def open_database(self, user='admin'): |
|
7f00fc5958ca
Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6360
diff
changeset
|
20 if self.db: |
|
7f00fc5958ca
Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6360
diff
changeset
|
21 self.db.close() |
|
7f00fc5958ca
Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6360
diff
changeset
|
22 self.db = self.module.Database(config, user) |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
23 return self.db |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
24 |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
25 def setUp(self): |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
26 self.open_database() |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
27 setupSchema(self.db, 1, self.module) |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
28 |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
29 def tearDown(self): |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
30 if self.db is not None: |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
31 self.db.close() |
|
6365
7f00fc5958ca
Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6360
diff
changeset
|
32 self.db = None |
|
7f00fc5958ca
Make memorydb persistent across re-open
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6360
diff
changeset
|
33 self.nuke_database() |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
34 |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
35 # nuke and re-create db for restore |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
36 def nukeAndCreate(self): |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
37 self.db.close() |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
38 self.nuke_database() |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
39 self.db = self.module.Database(config, 'admin') |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
40 setupSchema(self.db, 0, self.module) |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
41 |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4570
diff
changeset
|
42 |
|
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4570
diff
changeset
|
43 class memorydbDBTest(memorydbOpener, DBTest, unittest.TestCase): |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
44 pass |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
45 |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4570
diff
changeset
|
46 |
|
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4570
diff
changeset
|
47 class memorydbROTest(memorydbOpener, ROTest, unittest.TestCase): |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
48 def setUp(self): |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
49 self.db = self.module.Database(config) |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
50 setupSchema(self.db, 0, self.module) |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
51 |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4570
diff
changeset
|
52 |
|
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4570
diff
changeset
|
53 class memorydbSchemaTest(memorydbOpener, SchemaTest, unittest.TestCase): |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
54 pass |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
55 |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4570
diff
changeset
|
56 |
|
5388
d26921b851c3
Python 3 preparation: make relative imports explicit.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5319
diff
changeset
|
57 from .session_common import SessionTest |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5037
diff
changeset
|
58 class memorydbSessionTest(memorydbOpener, SessionTest, unittest.TestCase): |
| 6802 | 59 s2b = lambda x,y: strings.s2b(y) |
| 60 | |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
61 def setUp(self): |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
62 self.db = self.module.Database(config, 'admin') |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
63 setupSchema(self.db, 1, self.module) |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
64 self.sessions = self.db.sessions |
|
6828
2ce855803633
Implement UpdateTimestamp for memorydb and test.
John Rouillard <rouilj@ieee.org>
parents:
6827
diff
changeset
|
65 self.db.Session = self.sessions |
|
6822
5053ee6c846b
memorydb fixes for otks tests.
John Rouillard <rouilj@ieee.org>
parents:
6810
diff
changeset
|
66 self.otks = self.db.otks |
|
6828
2ce855803633
Implement UpdateTimestamp for memorydb and test.
John Rouillard <rouilj@ieee.org>
parents:
6827
diff
changeset
|
67 self.db.Otk = self.otks |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
68 |
|
6828
2ce855803633
Implement UpdateTimestamp for memorydb and test.
John Rouillard <rouilj@ieee.org>
parents:
6827
diff
changeset
|
69 def get_ts(self): |
|
2ce855803633
Implement UpdateTimestamp for memorydb and test.
John Rouillard <rouilj@ieee.org>
parents:
6827
diff
changeset
|
70 return (self.sessions.get('random_session', '__timestamp'),) |
|
2ce855803633
Implement UpdateTimestamp for memorydb and test.
John Rouillard <rouilj@ieee.org>
parents:
6827
diff
changeset
|
71 |
|
2ce855803633
Implement UpdateTimestamp for memorydb and test.
John Rouillard <rouilj@ieee.org>
parents:
6827
diff
changeset
|
72 def testDbType(self): |
|
2ce855803633
Implement UpdateTimestamp for memorydb and test.
John Rouillard <rouilj@ieee.org>
parents:
6827
diff
changeset
|
73 self.assertIn("memorydb", repr(self.db)) |
|
2ce855803633
Implement UpdateTimestamp for memorydb and test.
John Rouillard <rouilj@ieee.org>
parents:
6827
diff
changeset
|
74 self.assertIn("{}", repr(self.db.Session)) |
|
6806
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6802
diff
changeset
|
75 |
|
4349
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
76 # vim: set filetype=python ts=4 sw=4 et si |
|
f0faef4dd023
Make memorydb pass all tests;
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
77 |
