Mercurial > p > roundup > code
view test/test_schema.py @ 7668:5b41018617f2
fix: out of memory error when importing under postgresql
If you try importing more than 20k items under postgresql you can run
out of memory:
psycopg2.errors.OutOfMemory: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
Tuning memory may help, it's unknown at this point.
This checkin forces a commit to the postgres database after 10,000
rows have been added. This clears out the savepoints for each row and
starts a new transaction.
back_postgresql.py:
Implement commit mechanism in checkpoint_data(). Add two class level
attributes for tracking the number of savepoints and the limit when
the commit should happen.
roundup_admin.py:
implement pragma and dynamically create the config item
RDBMS_SAVEPOINT_LIMIT used by checkpoint_data.
Also fixed formatting of descriptions when using pragma list in
verbose mode.
admin_guide.txt, upgrading.txt:
Document change and use of pragma savepoint_limit in roundup-admin
for changing the default of 10,000.
test/db_test_base.py:
add some more asserts. In existing testAdminImportExport, set the
savepoint limit to 5 to test setting method and so that the commit
code will be run by existing tests. This provides coverage, but
does not actually test that the commit is done every 5 savepoints
8-(. The verification of every 5 savepoints was done manually
using a pdb breakpoint just before the commit.
acknowledgements.txt:
Added 2.4.0 section mentioning Norbert as he has done a ton of
testing with much larger datasets than I can test with.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 19 Oct 2023 16:11:25 -0400 |
| parents | f8893e1cde0d |
| children |
line wrap: on
line source
# # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/) # This module is free software, and you may redistribute it and/or modify # under the same terms as Python, so long as this copyright message and # disclaimer are retained in their original form. # # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE # POSSIBILITY OF SUCH DAMAGE. # # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS" # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. import unittest, os, shutil from roundup import configuration from roundup.backends import back_anydbm from roundup.hyperdb import String, Password, Link, Multilink, Date, \ Interval config = configuration.CoreConfig() config.DATABASE = "_test_dir" class SchemaTestCase(unittest.TestCase): def setUp(self): # remove previous test, ignore errors if os.path.exists(config.DATABASE): shutil.rmtree(config.DATABASE) os.makedirs(config.DATABASE + '/files') self.db = back_anydbm.Database(config, 'admin') self.db.post_init() self.db.clear() def tearDown(self): self.db.close() shutil.rmtree(config.DATABASE) def testA_Status(self): status = back_anydbm.Class(self.db, "status", name=String()) self.assertTrue(status, 'no class object generated') status.setkey("name") val = status.create(name="unread") self.assertEqual(val, '1', 'expecting "1"') val = status.create(name="in-progress") self.assertEqual(val, '2', 'expecting "2"') val = status.create(name="testing") self.assertEqual(val, '3', 'expecting "3"') val = status.create(name="resolved") self.assertEqual(val, '4', 'expecting "4"') val = status.count() self.assertEqual(val, 4, 'expecting 4') val = status.list() self.assertEqual(val, ['1', '2', '3', '4'], 'blah') val = status.lookup("in-progress") self.assertEqual(val, '2', 'expecting "2"') status.retire('3') val = status.list() self.assertEqual(val, ['1', '2', '4'], 'blah') def testB_Issue(self): issue = back_anydbm.Class(self.db, "issue", title=String(), status=Link("status")) self.assertTrue(issue, 'no class object returned') def testC_User(self): user = back_anydbm.Class(self.db, "user", username=String(), password=Password()) self.assertTrue(user, 'no class object returned') user.setkey("username") # vim: set filetype=python sts=4 sw=4 et si :
