annotate test/test_anydbm.py @ 7668:5b41018617f2

fix: out of memory error when importing under postgresql If you try importing more than 20k items under postgresql you can run out of memory: psycopg2.errors.OutOfMemory: out of shared memory HINT: You might need to increase max_locks_per_transaction. Tuning memory may help, it's unknown at this point. This checkin forces a commit to the postgres database after 10,000 rows have been added. This clears out the savepoints for each row and starts a new transaction. back_postgresql.py: Implement commit mechanism in checkpoint_data(). Add two class level attributes for tracking the number of savepoints and the limit when the commit should happen. roundup_admin.py: implement pragma and dynamically create the config item RDBMS_SAVEPOINT_LIMIT used by checkpoint_data. Also fixed formatting of descriptions when using pragma list in verbose mode. admin_guide.txt, upgrading.txt: Document change and use of pragma savepoint_limit in roundup-admin for changing the default of 10,000. test/db_test_base.py: add some more asserts. In existing testAdminImportExport, set the savepoint limit to 5 to test setting method and so that the commit code will be run by existing tests. This provides coverage, but does not actually test that the commit is done every 5 savepoints 8-(. The verification of every 5 savepoints was done manually using a pdb breakpoint just before the commit. acknowledgements.txt: Added 2.4.0 section mentioning Norbert as he has done a ton of testing with much larger datasets than I can test with.
author John Rouillard <rouilj@ieee.org>
date Thu, 19 Oct 2023 16:11:25 -0400
parents 3f60a71b0812
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1 #
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/)
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3 # This module is free software, and you may redistribute it and/or modify
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
4 # under the same terms as Python, so long as this copyright message and
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
5 # disclaimer are retained in their original form.
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
6 #
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
10 # POSSIBILITY OF SUCH DAMAGE.
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
11 #
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS"
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
17
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
18 import unittest, os, shutil, time
2856
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
19 from roundup.backends import get_backend
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
20
5388
d26921b851c3 Python 3 preparation: make relative imports explicit.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5319
diff changeset
21 from .db_test_base import DBTest, ROTest, SchemaTest, ClassicInitTest, config
d26921b851c3 Python 3 preparation: make relative imports explicit.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5319
diff changeset
22 from .db_test_base import HTMLItemTest, SpecialActionTest
5601
fcbeff272828 Integrate REST tests into db framework
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5388
diff changeset
23 from .rest_common import TestCase as RestTestCase
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
24
6802
044dcf3608a2 update session db tests
John Rouillard <rouilj@ieee.org>
parents: 5601
diff changeset
25 from roundup.anypy import strings
044dcf3608a2 update session db tests
John Rouillard <rouilj@ieee.org>
parents: 5601
diff changeset
26
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
27 class anydbmOpener:
2856
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2082
diff changeset
28 module = get_backend('anydbm')
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
29
1920
f9316d2cd5ba Fixed retirement of items in rdbms imports [SF#841355]
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
30 def nuke_database(self):
f9316d2cd5ba Fixed retirement of items in rdbms imports [SF#841355]
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
31 shutil.rmtree(config.DATABASE)
f9316d2cd5ba Fixed retirement of items in rdbms imports [SF#841355]
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
32
5033
63c79c0992ae Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents: 4878
diff changeset
33
63c79c0992ae Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents: 4878
diff changeset
34 class anydbmDBTest(anydbmOpener, DBTest, unittest.TestCase):
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
35 pass
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
36
5033
63c79c0992ae Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents: 4878
diff changeset
37
63c79c0992ae Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents: 4878
diff changeset
38 class anydbmROTest(anydbmOpener, ROTest, unittest.TestCase):
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
39 pass
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
40
5033
63c79c0992ae Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents: 4878
diff changeset
41
63c79c0992ae Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents: 4878
diff changeset
42 class anydbmSchemaTest(anydbmOpener, SchemaTest, unittest.TestCase):
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
43 pass
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
44
5033
63c79c0992ae Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents: 4878
diff changeset
45
63c79c0992ae Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents: 4878
diff changeset
46 class anydbmClassicInitTest(ClassicInitTest, unittest.TestCase):
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
47 backend = 'anydbm'
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
48
5033
63c79c0992ae Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents: 4878
diff changeset
49
63c79c0992ae Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents: 4878
diff changeset
50 class anydbmHTMLItemTest(HTMLItemTest, unittest.TestCase):
4878
f6e76a03b502 HTML* classes for cgi are now all new-style
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4570
diff changeset
51 backend = 'anydbm'
f6e76a03b502 HTML* classes for cgi are now all new-style
Ralf Schlatterbeck <rsc@runtux.com>
parents: 4570
diff changeset
52
5033
63c79c0992ae Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents: 4878
diff changeset
53
5388
d26921b851c3 Python 3 preparation: make relative imports explicit.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5319
diff changeset
54 from .session_common import SessionTest
5319
62de601bdf6f Fix commits although a Reject exception is raised
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5310
diff changeset
55 class anydbmSessionTest(anydbmOpener, SessionTest, unittest.TestCase):
6802
044dcf3608a2 update session db tests
John Rouillard <rouilj@ieee.org>
parents: 5601
diff changeset
56 s2b = lambda x,y: strings.s2b(y)
2082
c091cacdc505 Finished implementation of session and one-time-key stores for RDBMS backends.
Richard Jones <richard@users.sourceforge.net>
parents: 1920
diff changeset
57
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
58 def get_ts(self):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
59 return (self.sessions.get('random_session', '__timestamp'),)
6806
bdd28b244839 - issue2551223 - fix timestamp truncation in mysql and postgresql
John Rouillard <rouilj@ieee.org>
parents: 6802
diff changeset
60
6814
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
61 def testDbType(self):
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
62 self.assertIn("back_anydbm", repr(self.db))
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
63 self.assertIn("roundup.backends.sessions_dbm.Sessions", repr(self.db.Session))
3f60a71b0812 Summary: Support selecion session/otk data store. Add redis as data store.
John Rouillard <rouilj@ieee.org>
parents: 6806
diff changeset
64
5310
efb34cbdba7c Add (currently failing) test for atomic actions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5037
diff changeset
65 class anydbmSpecialActionTestCase(anydbmOpener, SpecialActionTest,
efb34cbdba7c Add (currently failing) test for atomic actions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5037
diff changeset
66 unittest.TestCase):
efb34cbdba7c Add (currently failing) test for atomic actions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5037
diff changeset
67 backend = 'anydbm'
efb34cbdba7c Add (currently failing) test for atomic actions
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5037
diff changeset
68
5601
fcbeff272828 Integrate REST tests into db framework
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5388
diff changeset
69 class anydbmRestTest (RestTestCase, unittest.TestCase):
fcbeff272828 Integrate REST tests into db framework
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5388
diff changeset
70 backend = 'anydbm'
fcbeff272828 Integrate REST tests into db framework
Ralf Schlatterbeck <rsc@runtux.com>
parents: 5388
diff changeset
71
1873
f63aa57386b0 Backend improvements.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
72 # vim: set filetype=python ts=4 sw=4 et si

Roundup Issue Tracker: http://roundup-tracker.org/