Mercurial > p > roundup > code
view doc/postgresql.txt @ 7668:5b41018617f2
fix: out of memory error when importing under postgresql
If you try importing more than 20k items under postgresql you can run
out of memory:
psycopg2.errors.OutOfMemory: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
Tuning memory may help, it's unknown at this point.
This checkin forces a commit to the postgres database after 10,000
rows have been added. This clears out the savepoints for each row and
starts a new transaction.
back_postgresql.py:
Implement commit mechanism in checkpoint_data(). Add two class level
attributes for tracking the number of savepoints and the limit when
the commit should happen.
roundup_admin.py:
implement pragma and dynamically create the config item
RDBMS_SAVEPOINT_LIMIT used by checkpoint_data.
Also fixed formatting of descriptions when using pragma list in
verbose mode.
admin_guide.txt, upgrading.txt:
Document change and use of pragma savepoint_limit in roundup-admin
for changing the default of 10,000.
test/db_test_base.py:
add some more asserts. In existing testAdminImportExport, set the
savepoint limit to 5 to test setting method and so that the commit
code will be run by existing tests. This provides coverage, but
does not actually test that the commit is done every 5 savepoints
8-(. The verification of every 5 savepoints was done manually
using a pdb breakpoint just before the commit.
acknowledgements.txt:
Added 2.4.0 section mentioning Norbert as he has done a ton of
testing with much larger datasets than I can test with.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 19 Oct 2023 16:11:25 -0400 |
| parents | b21ebabeb2ef |
| children | 3071db43bfb6 |
line wrap: on
line source
.. index:: postgres; deployment notes ========================== PostgreSQL/psycopg Backend ========================== This are notes about PostqreSQL backend based on the psycopg adapter for Roundup issue tracker. Prerequisites ============= To use PostgreSQL as backend for storing roundup data, you should additionally install: 1. PostgreSQL_ 8.x or higher 2. The psycopg_ python interface to PostgreSQL_ Note that you currently need at least version 2.8 of psycopg_ -- the postgresql backend will work with 2.7 but if you're using the native postgresql indexer you need at least 2.8. Also if you want to run the tests, these also need 2.8. If your distribution has an older version we suggest that you install into a python virtual environment. .. _psycopg: https://www.psycopg.org/ .. _PostgreSQL: https://www.postgresql.org/ Running the PostgreSQL unit tests ================================= The user that you're running the tests as will need to be able to access the postgresql database on the local machine and create and drop databases. See the config values in 'test/db_test_base.py' about which database connection, name and user will be used. At this time the following command will setup the user:: sudo -u postgres psql -c "CREATE ROLE rounduptest WITH CREATEDB LOGIN PASSWORD 'rounduptest';" -U postgres Note ``rounduptest`` is a well known account, so you should remove/disable the account after testing and set up a suitable production account. You need to remove any database owned by ``rounduptest`` first. So something like this should work:: sudo -u postgres psql -c "DROP DATABASE rounduptest;" -U postgres sudo -u postgres psql -c "DROP ROLE rounduptest;" -U postgres If the ``rounduptest`` database is left in a broken state (e.g. because of a crash during testing) dropping the database and restarting the tests should fix it. Credit ====== The postgresql backend was originally submitted by Federico Di Gregorio <fog@initd.org>