Mercurial > p > roundup > code
view test/test_demo.py @ 7668:5b41018617f2
fix: out of memory error when importing under postgresql
If you try importing more than 20k items under postgresql you can run
out of memory:
psycopg2.errors.OutOfMemory: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
Tuning memory may help, it's unknown at this point.
This checkin forces a commit to the postgres database after 10,000
rows have been added. This clears out the savepoints for each row and
starts a new transaction.
back_postgresql.py:
Implement commit mechanism in checkpoint_data(). Add two class level
attributes for tracking the number of savepoints and the limit when
the commit should happen.
roundup_admin.py:
implement pragma and dynamically create the config item
RDBMS_SAVEPOINT_LIMIT used by checkpoint_data.
Also fixed formatting of descriptions when using pragma list in
verbose mode.
admin_guide.txt, upgrading.txt:
Document change and use of pragma savepoint_limit in roundup-admin
for changing the default of 10,000.
test/db_test_base.py:
add some more asserts. In existing testAdminImportExport, set the
savepoint limit to 5 to test setting method and so that the commit
code will be run by existing tests. This provides coverage, but
does not actually test that the commit is done every 5 savepoints
8-(. The verification of every 5 savepoints was done manually
using a pdb breakpoint just before the commit.
acknowledgements.txt:
Added 2.4.0 section mentioning Norbert as he has done a ton of
testing with much larger datasets than I can test with.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 19 Oct 2023 16:11:25 -0400 |
| parents | 859c57bc8d91 |
| children | 5b1163dca9e1 |
line wrap: on
line source
import pytest import unittest import os, sys, shutil from roundup.demo import install_demo, run_demo import roundup.scripts.roundup_server # https://stackoverflow.com/questions/4219717/how-to-assert-output-with-nosetest-unittest-in-python # lightly modified from contextlib import contextmanager _py3 = sys.version_info[0] > 2 if _py3: from io import StringIO # py3 else: from StringIO import StringIO # py2 @contextmanager def captured_output(): new_out, new_err = StringIO(), StringIO() old_out, old_err = sys.stdout, sys.stderr try: sys.stdout, sys.stderr = new_out, new_err yield sys.stdout, sys.stderr finally: sys.stdout, sys.stderr = old_out, old_err try: import jinja2 skip_jinja2 = lambda func, *args, **kwargs: func except ImportError: from .pytest_patcher import mark_class skip_jinja2 = mark_class(pytest.mark.skip( reason='Skipping Jinja2 tests: jinja2 library not available')) class TestDemo(unittest.TestCase): def setUp(self): self.home = os.path.abspath('_test_demo') def tearDown(self): try: shutil.rmtree(self.home) except FileNotFoundError: pass def run_install_demo(self, template, db="anydbm"): with captured_output() as (out, err): install_demo(self.home, db, template) output = out.getvalue().strip() print(output) # verify that db was set properly by reading config with open(self.home + "/config.ini", "r") as f: config_lines = f.read().replace("\r\n", "\n") try: # handle text files with \r\n line endings config_lines.index("\r", 0, 100) config_lines = config_lines.replace("\r\n", "\n") except ValueError: pass self.assertIn("backend = %s\n"%db, config_lines) # dummy up the return of get_server so the serve_forever method # raises keyboard interrupt exiting the server so the test exits. gs = roundup.scripts.roundup_server.ServerConfig.get_server def raise_KeyboardInterrupt(): raise KeyboardInterrupt def test_get_server(self): httpd = gs(self) httpd.serve_forever = raise_KeyboardInterrupt return httpd roundup.scripts.roundup_server.ServerConfig.get_server = test_get_server # Run under context manager to capture output of startup text. with captured_output() as (out, err): run_demo(self.home) output = out.getvalue().strip() print(output) # if the server installed and started this will be the # last line in the output. self.assertIn("Keyboard Interrupt: exiting", output.split('\n')) def testDemoClassic(self): self.run_install_demo("classic") def testDemoMinimal(self): self.run_install_demo('../templates/minimal', db="sqlite") @skip_jinja2 def testDemoJinja(self): self.run_install_demo('jinja2', db="anydbm") # verify that template was set to jinja2 by reading config with open(self.home + "/config.ini", "r") as f: config_lines = f.read() try: # handle text files with \r\n line endings config_lines.index("\r", 0, 100) config_lines = config_lines.replace("\r\n", "\n") except ValueError: pass self.assertIn("template_engine = jinja2\n", config_lines)
