Mercurial > p > roundup > code
view test/test_memorydb.py @ 7668:5b41018617f2
fix: out of memory error when importing under postgresql
If you try importing more than 20k items under postgresql you can run
out of memory:
psycopg2.errors.OutOfMemory: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
Tuning memory may help, it's unknown at this point.
This checkin forces a commit to the postgres database after 10,000
rows have been added. This clears out the savepoints for each row and
starts a new transaction.
back_postgresql.py:
Implement commit mechanism in checkpoint_data(). Add two class level
attributes for tracking the number of savepoints and the limit when
the commit should happen.
roundup_admin.py:
implement pragma and dynamically create the config item
RDBMS_SAVEPOINT_LIMIT used by checkpoint_data.
Also fixed formatting of descriptions when using pragma list in
verbose mode.
admin_guide.txt, upgrading.txt:
Document change and use of pragma savepoint_limit in roundup-admin
for changing the default of 10,000.
test/db_test_base.py:
add some more asserts. In existing testAdminImportExport, set the
savepoint limit to 5 to test setting method and so that the commit
code will be run by existing tests. This provides coverage, but
does not actually test that the commit is done every 5 savepoints
8-(. The verification of every 5 savepoints was done manually
using a pdb breakpoint just before the commit.
acknowledgements.txt:
Added 2.4.0 section mentioning Norbert as he has done a ton of
testing with much larger datasets than I can test with.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 19 Oct 2023 16:11:25 -0400 |
| parents | 2ce855803633 |
| children |
line wrap: on
line source
import unittest, os, shutil, time from roundup import hyperdb from .db_test_base import DBTest, ROTest, SchemaTest, config, setupSchema from roundup.test import memorydb from roundup.anypy import strings class memorydbOpener: module = memorydb def nuke_database(self): # really kill it memorydb.db_nuke('') self.db = None db = None def open_database(self, user='admin'): if self.db: self.db.close() self.db = self.module.Database(config, user) return self.db def setUp(self): self.open_database() setupSchema(self.db, 1, self.module) def tearDown(self): if self.db is not None: self.db.close() self.db = None self.nuke_database() # nuke and re-create db for restore def nukeAndCreate(self): self.db.close() self.nuke_database() self.db = self.module.Database(config, 'admin') setupSchema(self.db, 0, self.module) class memorydbDBTest(memorydbOpener, DBTest, unittest.TestCase): pass class memorydbROTest(memorydbOpener, ROTest, unittest.TestCase): def setUp(self): self.db = self.module.Database(config) setupSchema(self.db, 0, self.module) class memorydbSchemaTest(memorydbOpener, SchemaTest, unittest.TestCase): pass from .session_common import SessionTest class memorydbSessionTest(memorydbOpener, SessionTest, unittest.TestCase): s2b = lambda x,y: strings.s2b(y) def setUp(self): self.db = self.module.Database(config, 'admin') setupSchema(self.db, 1, self.module) self.sessions = self.db.sessions self.db.Session = self.sessions self.otks = self.db.otks self.db.Otk = self.otks def get_ts(self): return (self.sessions.get('random_session', '__timestamp'),) def testDbType(self): self.assertIn("memorydb", repr(self.db)) self.assertIn("{}", repr(self.db.Session)) # vim: set filetype=python ts=4 sw=4 et si
