Mercurial > p > roundup > code
comparison doc/admin_guide.txt @ 7668:5b41018617f2
fix: out of memory error when importing under postgresql
If you try importing more than 20k items under postgresql you can run
out of memory:
psycopg2.errors.OutOfMemory: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
Tuning memory may help, it's unknown at this point.
This checkin forces a commit to the postgres database after 10,000
rows have been added. This clears out the savepoints for each row and
starts a new transaction.
back_postgresql.py:
Implement commit mechanism in checkpoint_data(). Add two class level
attributes for tracking the number of savepoints and the limit when
the commit should happen.
roundup_admin.py:
implement pragma and dynamically create the config item
RDBMS_SAVEPOINT_LIMIT used by checkpoint_data.
Also fixed formatting of descriptions when using pragma list in
verbose mode.
admin_guide.txt, upgrading.txt:
Document change and use of pragma savepoint_limit in roundup-admin
for changing the default of 10,000.
test/db_test_base.py:
add some more asserts. In existing testAdminImportExport, set the
savepoint limit to 5 to test setting method and so that the commit
code will be run by existing tests. This provides coverage, but
does not actually test that the commit is done every 5 savepoints
8-(. The verification of every 5 savepoints was done manually
using a pdb breakpoint just before the commit.
acknowledgements.txt:
Added 2.4.0 section mentioning Norbert as he has done a ton of
testing with much larger datasets than I can test with.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 19 Oct 2023 16:11:25 -0400 |
| parents | 08e4399c3ae4 |
| children | d7d68da9878f |
comparison
equal
deleted
inserted
replaced
| 7667:08e4399c3ae4 | 7668:5b41018617f2 |
|---|---|
| 960 backend. | 960 backend. |
| 961 8. Move the old tracker home out of the way (rename to "tracker.old") and | 961 8. Move the old tracker home out of the way (rename to "tracker.old") and |
| 962 move the new tracker home into its place. | 962 move the new tracker home into its place. |
| 963 9. Restart web and email frontends. | 963 9. Restart web and email frontends. |
| 964 | 964 |
| 965 If you are importing into PostgreSQL, it autocommits the data every | |
| 966 10000 objects/rows by default. This can slow down importing, but it | |
| 967 prevents an out of memory error caused by using a savepoint for each | |
| 968 object. You can control the commit frequency by using:: | |
| 969 | |
| 970 pragma savepoint_limit=20000 | |
| 971 | |
| 972 to set a higher or lower number in roundup-admin. In this example a | |
| 973 commit will be done every 20,000 objects/rows. The pragma can also be | |
| 974 set on the roundup-admin command line as described below. | |
| 965 | 975 |
| 966 Moving a Tracker | 976 Moving a Tracker |
| 967 ---------------- | 977 ---------------- |
| 968 | 978 |
| 969 If you're moving the tracker to a similar machine, you should: | 979 If you're moving the tracker to a similar machine, you should: |
