Mercurial > p > roundup > code
comparison doc/upgrading.txt @ 7668:5b41018617f2
fix: out of memory error when importing under postgresql
If you try importing more than 20k items under postgresql you can run
out of memory:
psycopg2.errors.OutOfMemory: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
Tuning memory may help, it's unknown at this point.
This checkin forces a commit to the postgres database after 10,000
rows have been added. This clears out the savepoints for each row and
starts a new transaction.
back_postgresql.py:
Implement commit mechanism in checkpoint_data(). Add two class level
attributes for tracking the number of savepoints and the limit when
the commit should happen.
roundup_admin.py:
implement pragma and dynamically create the config item
RDBMS_SAVEPOINT_LIMIT used by checkpoint_data.
Also fixed formatting of descriptions when using pragma list in
verbose mode.
admin_guide.txt, upgrading.txt:
Document change and use of pragma savepoint_limit in roundup-admin
for changing the default of 10,000.
test/db_test_base.py:
add some more asserts. In existing testAdminImportExport, set the
savepoint limit to 5 to test setting method and so that the commit
code will be run by existing tests. This provides coverage, but
does not actually test that the commit is done every 5 savepoints
8-(. The verification of every 5 savepoints was done manually
using a pdb breakpoint just before the commit.
acknowledgements.txt:
Added 2.4.0 section mentioning Norbert as he has done a ton of
testing with much larger datasets than I can test with.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 19 Oct 2023 16:11:25 -0400 |
| parents | 978285986b2c |
| children | a27f30709d46 |
comparison
equal
deleted
inserted
replaced
| 7667:08e4399c3ae4 | 7668:5b41018617f2 |
|---|---|
| 138 from roundup.anypy.cgi_ import FieldStorage, MiniFieldStorage | 138 from roundup.anypy.cgi_ import FieldStorage, MiniFieldStorage |
| 139 | 139 |
| 140 It is unlikey that you will care unless you have done some expert | 140 It is unlikey that you will care unless you have done some expert |
| 141 level Roundup customization. If you have, use one of the imports above | 141 level Roundup customization. If you have, use one of the imports above |
| 142 if you plan on running on Python 3.13 (expected in 2024) or newer. | 142 if you plan on running on Python 3.13 (expected in 2024) or newer. |
| 143 | |
| 144 Fixing PostgreSQL Out of Memory Errors when Importing Tracker (info) | |
| 145 -------------------------------------------------------------------- | |
| 146 | |
| 147 Importing a tracker into PostgreSQL can run out of memory with the | |
| 148 error:: | |
| 149 | |
| 150 psycopg2.errors.OutOfMemory: out of shared memory | |
| 151 HINT: You might need to increase max_locks_per_transaction. | |
| 152 | |
| 153 before changing your PostgreSQL configuration, try changing the pragma | |
| 154 ``savepoint_limit`` to a lower value. By default it is set to | |
| 155 ``10000``. In some cases this may be too high. See the `administration | |
| 156 guide`_ for further details. | |
| 143 | 157 |
| 144 .. index:: Upgrading; 2.2.0 to 2.3.0 | 158 .. index:: Upgrading; 2.2.0 to 2.3.0 |
| 145 | 159 |
| 146 Migrating from 2.2.0 to 2.3.0 | 160 Migrating from 2.2.0 to 2.3.0 |
| 147 ============================= | 161 ============================= |
