comparison doc/upgrading.txt @ 7668:5b41018617f2

fix: out of memory error when importing under postgresql If you try importing more than 20k items under postgresql you can run out of memory: psycopg2.errors.OutOfMemory: out of shared memory HINT: You might need to increase max_locks_per_transaction. Tuning memory may help, it's unknown at this point. This checkin forces a commit to the postgres database after 10,000 rows have been added. This clears out the savepoints for each row and starts a new transaction. back_postgresql.py: Implement commit mechanism in checkpoint_data(). Add two class level attributes for tracking the number of savepoints and the limit when the commit should happen. roundup_admin.py: implement pragma and dynamically create the config item RDBMS_SAVEPOINT_LIMIT used by checkpoint_data. Also fixed formatting of descriptions when using pragma list in verbose mode. admin_guide.txt, upgrading.txt: Document change and use of pragma savepoint_limit in roundup-admin for changing the default of 10,000. test/db_test_base.py: add some more asserts. In existing testAdminImportExport, set the savepoint limit to 5 to test setting method and so that the commit code will be run by existing tests. This provides coverage, but does not actually test that the commit is done every 5 savepoints 8-(. The verification of every 5 savepoints was done manually using a pdb breakpoint just before the commit. acknowledgements.txt: Added 2.4.0 section mentioning Norbert as he has done a ton of testing with much larger datasets than I can test with.
author John Rouillard <rouilj@ieee.org>
date Thu, 19 Oct 2023 16:11:25 -0400
parents 978285986b2c
children a27f30709d46
comparison
equal deleted inserted replaced
7667:08e4399c3ae4 7668:5b41018617f2
138 from roundup.anypy.cgi_ import FieldStorage, MiniFieldStorage 138 from roundup.anypy.cgi_ import FieldStorage, MiniFieldStorage
139 139
140 It is unlikey that you will care unless you have done some expert 140 It is unlikey that you will care unless you have done some expert
141 level Roundup customization. If you have, use one of the imports above 141 level Roundup customization. If you have, use one of the imports above
142 if you plan on running on Python 3.13 (expected in 2024) or newer. 142 if you plan on running on Python 3.13 (expected in 2024) or newer.
143
144 Fixing PostgreSQL Out of Memory Errors when Importing Tracker (info)
145 --------------------------------------------------------------------
146
147 Importing a tracker into PostgreSQL can run out of memory with the
148 error::
149
150 psycopg2.errors.OutOfMemory: out of shared memory
151 HINT: You might need to increase max_locks_per_transaction.
152
153 before changing your PostgreSQL configuration, try changing the pragma
154 ``savepoint_limit`` to a lower value. By default it is set to
155 ``10000``. In some cases this may be too high. See the `administration
156 guide`_ for further details.
143 157
144 .. index:: Upgrading; 2.2.0 to 2.3.0 158 .. index:: Upgrading; 2.2.0 to 2.3.0
145 159
146 Migrating from 2.2.0 to 2.3.0 160 Migrating from 2.2.0 to 2.3.0
147 ============================= 161 =============================

Roundup Issue Tracker: http://roundup-tracker.org/