annotate test/README.txt @ 7668:5b41018617f2

fix: out of memory error when importing under postgresql If you try importing more than 20k items under postgresql you can run out of memory: psycopg2.errors.OutOfMemory: out of shared memory HINT: You might need to increase max_locks_per_transaction. Tuning memory may help, it's unknown at this point. This checkin forces a commit to the postgres database after 10,000 rows have been added. This clears out the savepoints for each row and starts a new transaction. back_postgresql.py: Implement commit mechanism in checkpoint_data(). Add two class level attributes for tracking the number of savepoints and the limit when the commit should happen. roundup_admin.py: implement pragma and dynamically create the config item RDBMS_SAVEPOINT_LIMIT used by checkpoint_data. Also fixed formatting of descriptions when using pragma list in verbose mode. admin_guide.txt, upgrading.txt: Document change and use of pragma savepoint_limit in roundup-admin for changing the default of 10,000. test/db_test_base.py: add some more asserts. In existing testAdminImportExport, set the savepoint limit to 5 to test setting method and so that the commit code will be run by existing tests. This provides coverage, but does not actually test that the commit is done every 5 savepoints 8-(. The verification of every 5 savepoints was done manually using a pdb breakpoint just before the commit. acknowledgements.txt: Added 2.4.0 section mentioning Norbert as he has done a ton of testing with much larger datasets than I can test with.
author John Rouillard <rouilj@ieee.org>
date Thu, 19 Oct 2023 16:11:25 -0400
parents a86b0c02940d
children 132d450bdc00
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
4964
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
1 A number of tests uses the infrastructure of
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
2 db_test_base.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
3
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
4 grep "from db_test_base" -l *.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
5 benchmark.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
6 session_common.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
7 test_anydbm.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
8 test_indexer.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
9 test_memorydb.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
10 test_mysql.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
11 test_postgresql.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
12 test_security.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
13 test_sqlite.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
14 test_userauditor.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
15
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
16 grep "import db_test_base" -l *.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
17 test_cgi.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
18 test_jinja2.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
19 test_mailgw.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
20 test_xmlrpc.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
21
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
22 grep "import memory\|from memory" -l *.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
23 test_mailgw.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
24 test_memorydb.py
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
25
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
26
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
27 The remaining lines are an 2001 description from Richard,
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
28 which probably is outdated:
2c3cc4ccd024 Automatic tests: added some notes to the readme and a test_jinja2 stub.
Bernhard Reiter <bernhard@intevation.de>
parents: 4570
diff changeset
29
127
0791d13baea7 Added vim command to all source so that we don't get no steenkin' tabs :)
Richard Jones <richard@users.sourceforge.net>
parents: 95
diff changeset
30 Structure of the tests:
95
cd4f0415d0dc rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
31
cd4f0415d0dc rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
32 1 Test date classes
cd4f0415d0dc rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
33 1.1 Date
cd4f0415d0dc rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
34 1.2 Interval
cd4f0415d0dc rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
35 2 Set up schema
cd4f0415d0dc rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
36 3 Open with specific backend
cd4f0415d0dc rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
37 3.1 anydbm
cd4f0415d0dc rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
38 4 Create database base set (stati, priority, etc)
cd4f0415d0dc rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
39 5 Perform some actions
cd4f0415d0dc rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
40 6 Perform mail import
cd4f0415d0dc rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
41 6.1 text/plain
cd4f0415d0dc rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
42 6.2 multipart/mixed (with one text/plain)
cd4f0415d0dc rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
43 6.3 text/html
cd4f0415d0dc rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
44 6.4 multipart/alternative (with one text/plain)
cd4f0415d0dc rename for consistency
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
45 6.5 multipart/alternative (with no text/plain)

Roundup Issue Tracker: http://roundup-tracker.org/