Mercurial > p > roundup > code
annotate test/test_jinja2.py @ 7668:5b41018617f2
fix: out of memory error when importing under postgresql
If you try importing more than 20k items under postgresql you can run
out of memory:
psycopg2.errors.OutOfMemory: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
Tuning memory may help, it's unknown at this point.
This checkin forces a commit to the postgres database after 10,000
rows have been added. This clears out the savepoints for each row and
starts a new transaction.
back_postgresql.py:
Implement commit mechanism in checkpoint_data(). Add two class level
attributes for tracking the number of savepoints and the limit when
the commit should happen.
roundup_admin.py:
implement pragma and dynamically create the config item
RDBMS_SAVEPOINT_LIMIT used by checkpoint_data.
Also fixed formatting of descriptions when using pragma list in
verbose mode.
admin_guide.txt, upgrading.txt:
Document change and use of pragma savepoint_limit in roundup-admin
for changing the default of 10,000.
test/db_test_base.py:
add some more asserts. In existing testAdminImportExport, set the
savepoint limit to 5 to test setting method and so that the commit
code will be run by existing tests. This provides coverage, but
does not actually test that the commit is done every 5 savepoints
8-(. The verification of every 5 savepoints was done manually
using a pdb breakpoint just before the commit.
acknowledgements.txt:
Added 2.4.0 section mentioning Norbert as he has done a ton of
testing with much larger datasets than I can test with.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 19 Oct 2023 16:11:25 -0400 |
| parents | d26921b851c3 |
| children |
| rev | line source |
|---|---|
|
5248
198b6e810c67
Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents:
5037
diff
changeset
|
1 #-*- encoding: utf-8 -*- |
|
4964
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
2 """ Testing the jinja2 templating engine of roundup-tracker. |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
3 |
|
5009
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
4 Copyright: 2016 Intevation GmbH. |
|
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
5 Author: Bernhard E. Reiter <bernhard@intevation.de> |
|
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
6 |
|
4964
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
7 This module is Free Software under the Roundup licensing of 1.5, |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
8 see the COPYING.txt file coming with Roundup. |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
9 |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
10 Just a test file template for now. |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
11 """ |
|
5009
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
12 import shutil # only, needed for tearDown. TODO: Remove when refactored. |
|
4964
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
13 import unittest |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
14 |
|
5388
d26921b851c3
Python 3 preparation: make relative imports explicit.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5248
diff
changeset
|
15 from . import db_test_base |
|
4964
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
16 |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
17 TESTSUITE_IDENTIFIER='jinja2' |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
18 |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
19 class TestCase_Zero(unittest.TestCase): |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
20 def test_zero(self): |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
21 self.assertEqual(True, True) |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
22 |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
5009
diff
changeset
|
23 |
|
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
5009
diff
changeset
|
24 class Jinja2Test(object): |
|
5009
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
25 """Sets up and tears down an instance with database contents. |
|
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
26 |
|
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
27 Setup and teardown modelled after the use of db_test_base |
|
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
28 by several modules like test_xmlrpc and test_userauditor. |
|
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
29 |
|
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
30 TODO: Should probably be moved to a base case in db_test_base.py. |
|
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
31 """ |
|
4964
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
32 |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
33 backend = None # can be used to create tests per backend, see test_xmlrpc |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
34 |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
35 def setUp(self): |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
36 self.dirname = '_test_' + TESTSUITE_IDENTIFIER |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
37 self.instance = db_test_base.setupTracker(self.dirname, self.backend) |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
38 self.db = self.instance.open('admin') |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
39 |
|
5009
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
40 def tearDown(self): |
|
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
41 self.db.close() |
|
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
42 try: |
|
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
43 shutil.rmtree(self.dirname) |
|
5248
198b6e810c67
Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents:
5037
diff
changeset
|
44 except OSError as error: |
|
5009
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
45 if error.errno not in (errno.ENOENT, errno.ESRCH): raise |
|
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
46 |
|
4964
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
47 def test_zero(self): |
|
5009
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
48 """Do nothing just make sure that setup and teardown works.""" |
|
4964
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
49 pass |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
50 |
|
5009
3766e0ca8e7a
test_jinja2: stub improved, now with proper teardown.
Bernhard Reiter <bernhard@intevation.de>
parents:
4964
diff
changeset
|
51 |
|
5037
364c54991861
Remove unneeded TestSuite code from tests
John Kristensen <john@jerrykan.com>
parents:
5033
diff
changeset
|
52 # only using one database backend for now, not sure if doing all |
|
364c54991861
Remove unneeded TestSuite code from tests
John Kristensen <john@jerrykan.com>
parents:
5033
diff
changeset
|
53 # backends will keep the test focussed enough to be useful for the used |
|
364c54991861
Remove unneeded TestSuite code from tests
John Kristensen <john@jerrykan.com>
parents:
5033
diff
changeset
|
54 # computing time. Would be okay to change in the future. |
|
5033
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
5009
diff
changeset
|
55 class anydbmJinja2Test(Jinja2Test, unittest.TestCase): |
|
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
5009
diff
changeset
|
56 backend = 'anydbm' |
|
63c79c0992ae
Update tests to work with py.test
John Kristensen <john@jerrykan.com>
parents:
5009
diff
changeset
|
57 |
|
4964
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
58 # vim: ts=4 et sts=4 sw=4 ai : |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
59 |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
diff
changeset
|
60 |
