Mercurial > p > roundup > code
annotate test/test_anydbm.py @ 7668:5b41018617f2
fix: out of memory error when importing under postgresql
If you try importing more than 20k items under postgresql you can run
out of memory:
psycopg2.errors.OutOfMemory: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
Tuning memory may help, it's unknown at this point.
This checkin forces a commit to the postgres database after 10,000
rows have been added. This clears out the savepoints for each row and
starts a new transaction.
back_postgresql.py:
Implement commit mechanism in checkpoint_data(). Add two class level
attributes for tracking the number of savepoints and the limit when
the commit should happen.
roundup_admin.py:
implement pragma and dynamically create the config item
RDBMS_SAVEPOINT_LIMIT used by checkpoint_data.
Also fixed formatting of descriptions when using pragma list in
verbose mode.
admin_guide.txt, upgrading.txt:
Document change and use of pragma savepoint_limit in roundup-admin
for changing the default of 10,000.
test/db_test_base.py:
add some more asserts. In existing testAdminImportExport, set the
savepoint limit to 5 to test setting method and so that the commit
code will be run by existing tests. This provides coverage, but
does not actually test that the commit is done every 5 savepoints
8-(. The verification of every 5 savepoints was done manually
using a pdb breakpoint just before the commit.
acknowledgements.txt:
Added 2.4.0 section mentioning Norbert as he has done a ton of
testing with much larger datasets than I can test with.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 19 Oct 2023 16:11:25 -0400 |
| parents | 3f60a71b0812 |
| children |
| rev | line source |
|---|---|
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
1 # |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/) |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
3 # This module is free software, and you may redistribute it and/or modify |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
4 # under the same terms as Python, so long as this copyright message and |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
5 # disclaimer are retained in their original form. |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
6 # |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
10 # POSSIBILITY OF SUCH DAMAGE. |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
11 # |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS" |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
17 |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
18 import unittest, os, shutil, time |
|
2856
adec352e2ce0
don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
19 from roundup.backends import get_backend |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
20 |
|
5388
d26921b851c3
Python 3 preparation: make relative imports explicit.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5319
diff
changeset
|
21 from .db_test_base import DBTest, ROTest, SchemaTest, ClassicInitTest, config |
|
d26921b851c3
Python 3 preparation: make relative imports explicit.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5319
diff
changeset
|
22 from .db_test_base import HTMLItemTest, SpecialActionTest |
|
5601
fcbeff272828
Integrate REST tests into db framework
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5388
diff
changeset
|
23 from .rest_common import TestCase as RestTestCase |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
24 |
| 6802 | 25 from roundup.anypy import strings |
| 26 | |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
27 class anydbmOpener: |
|
2856
adec352e2ce0
don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents:
2082
diff
changeset
|
28 module = get_backend('anydbm') |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
29 |
|
1920
f9316d2cd5ba
Fixed retirement of items in rdbms imports [SF#841355]
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
30 def nuke_database(self): |
|
f9316d2cd5ba
Fixed retirement of items in rdbms imports [SF#841355]
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
31 shutil.rmtree(config.DATABASE) |
|
f9316d2cd5ba
Fixed retirement of items in rdbms imports [SF#841355]
Richard Jones <richard@users.sourceforge.net>
parents:
1873
diff
changeset
|
32 |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4878
diff
changeset
|
33 |
|
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4878
diff
changeset
|
34 class anydbmDBTest(anydbmOpener, DBTest, unittest.TestCase): |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
35 pass |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
36 |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4878
diff
changeset
|
37 |
|
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4878
diff
changeset
|
38 class anydbmROTest(anydbmOpener, ROTest, unittest.TestCase): |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
39 pass |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
40 |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4878
diff
changeset
|
41 |
|
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4878
diff
changeset
|
42 class anydbmSchemaTest(anydbmOpener, SchemaTest, unittest.TestCase): |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
43 pass |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
44 |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4878
diff
changeset
|
45 |
|
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4878
diff
changeset
|
46 class anydbmClassicInitTest(ClassicInitTest, unittest.TestCase): |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
47 backend = 'anydbm' |
|
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
48 |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4878
diff
changeset
|
49 |
|
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4878
diff
changeset
|
50 class anydbmHTMLItemTest(HTMLItemTest, unittest.TestCase): |
|
4878
f6e76a03b502
HTML* classes for cgi are now all new-style
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4570
diff
changeset
|
51 backend = 'anydbm' |
|
f6e76a03b502
HTML* classes for cgi are now all new-style
Ralf Schlatterbeck <rsc@runtux.com>
parents:
4570
diff
changeset
|
52 |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
4878
diff
changeset
|
53 |
|
5388
d26921b851c3
Python 3 preparation: make relative imports explicit.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5319
diff
changeset
|
54 from .session_common import SessionTest |
|
5319
62de601bdf6f
Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5310
diff
changeset
|
55 class anydbmSessionTest(anydbmOpener, SessionTest, unittest.TestCase): |
| 6802 | 56 s2b = lambda x,y: strings.s2b(y) |
|
2082
c091cacdc505
Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents:
1920
diff
changeset
|
57 |
|
6814
3f60a71b0812
Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
58 def get_ts(self): |
|
3f60a71b0812
Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
59 return (self.sessions.get('random_session', '__timestamp'),) |
|
6806
bdd28b244839
- issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents:
6802
diff
changeset
|
60 |
|
6814
3f60a71b0812
Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
61 def testDbType(self): |
|
3f60a71b0812
Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
62 self.assertIn("back_anydbm", repr(self.db)) |
|
3f60a71b0812
Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
63 self.assertIn("roundup.backends.sessions_dbm.Sessions", repr(self.db.Session)) |
|
3f60a71b0812
Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents:
6806
diff
changeset
|
64 |
|
5310
efb34cbdba7c
Add (currently failing) test for atomic actions
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5037
diff
changeset
|
65 class anydbmSpecialActionTestCase(anydbmOpener, SpecialActionTest, |
|
efb34cbdba7c
Add (currently failing) test for atomic actions
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5037
diff
changeset
|
66 unittest.TestCase): |
|
efb34cbdba7c
Add (currently failing) test for atomic actions
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5037
diff
changeset
|
67 backend = 'anydbm' |
|
efb34cbdba7c
Add (currently failing) test for atomic actions
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5037
diff
changeset
|
68 |
|
5601
fcbeff272828
Integrate REST tests into db framework
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5388
diff
changeset
|
69 class anydbmRestTest (RestTestCase, unittest.TestCase): |
|
fcbeff272828
Integrate REST tests into db framework
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5388
diff
changeset
|
70 backend = 'anydbm' |
|
fcbeff272828
Integrate REST tests into db framework
Ralf Schlatterbeck <rsc@runtux.com>
parents:
5388
diff
changeset
|
71 |
|
1873
f63aa57386b0
Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
72 # vim: set filetype=python ts=4 sw=4 et si |
