Mercurial > p > roundup > code
annotate test/README.txt @ 7668:5b41018617f2
fix: out of memory error when importing under postgresql
If you try importing more than 20k items under postgresql you can run
out of memory:
psycopg2.errors.OutOfMemory: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
Tuning memory may help, it's unknown at this point.
This checkin forces a commit to the postgres database after 10,000
rows have been added. This clears out the savepoints for each row and
starts a new transaction.
back_postgresql.py:
Implement commit mechanism in checkpoint_data(). Add two class level
attributes for tracking the number of savepoints and the limit when
the commit should happen.
roundup_admin.py:
implement pragma and dynamically create the config item
RDBMS_SAVEPOINT_LIMIT used by checkpoint_data.
Also fixed formatting of descriptions when using pragma list in
verbose mode.
admin_guide.txt, upgrading.txt:
Document change and use of pragma savepoint_limit in roundup-admin
for changing the default of 10,000.
test/db_test_base.py:
add some more asserts. In existing testAdminImportExport, set the
savepoint limit to 5 to test setting method and so that the commit
code will be run by existing tests. This provides coverage, but
does not actually test that the commit is done every 5 savepoints
8-(. The verification of every 5 savepoints was done manually
using a pdb breakpoint just before the commit.
acknowledgements.txt:
Added 2.4.0 section mentioning Norbert as he has done a ton of
testing with much larger datasets than I can test with.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 19 Oct 2023 16:11:25 -0400 |
| parents | a86b0c02940d |
| children | 132d450bdc00 |
| rev | line source |
|---|---|
|
4964
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
1 A number of tests uses the infrastructure of |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
2 db_test_base.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
3 |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
4 grep "from db_test_base" -l *.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
5 benchmark.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
6 session_common.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
7 test_anydbm.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
8 test_indexer.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
9 test_memorydb.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
10 test_mysql.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
11 test_postgresql.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
12 test_security.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
13 test_sqlite.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
14 test_userauditor.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
15 |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
16 grep "import db_test_base" -l *.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
17 test_cgi.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
18 test_jinja2.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
19 test_mailgw.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
20 test_xmlrpc.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
21 |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
22 grep "import memory\|from memory" -l *.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
23 test_mailgw.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
24 test_memorydb.py |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
25 |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
26 |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
27 The remaining lines are an 2001 description from Richard, |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
28 which probably is outdated: |
|
2c3cc4ccd024
Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents:
4570
diff
changeset
|
29 |
|
127
0791d13baea7
Added vim command to all source so that we don't get no steenkin' tabs :)
Richard Jones <richard@users.sourceforge.net>
parents:
95
diff
changeset
|
30 Structure of the tests: |
|
95
cd4f0415d0dc
rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
31 |
|
cd4f0415d0dc
rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
32 1 Test date classes |
|
cd4f0415d0dc
rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
33 1.1 Date |
|
cd4f0415d0dc
rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
34 1.2 Interval |
|
cd4f0415d0dc
rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
35 2 Set up schema |
|
cd4f0415d0dc
rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
36 3 Open with specific backend |
|
cd4f0415d0dc
rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
37 3.1 anydbm |
|
cd4f0415d0dc
rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
38 4 Create database base set (stati, priority, etc) |
|
cd4f0415d0dc
rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
39 5 Perform some actions |
|
cd4f0415d0dc
rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
40 6 Perform mail import |
|
cd4f0415d0dc
rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
41 6.1 text/plain |
|
cd4f0415d0dc
rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
42 6.2 multipart/mixed (with one text/plain) |
|
cd4f0415d0dc
rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
43 6.3 text/html |
|
cd4f0415d0dc
rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
44 6.4 multipart/alternative (with one text/plain) |
|
cd4f0415d0dc
rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff
changeset
|
45 6.5 multipart/alternative (with no text/plain) |
