Mercurial > p > roundup > code
view doc/mysql.txt @ 7668:5b41018617f2
fix: out of memory error when importing under postgresql
If you try importing more than 20k items under postgresql you can run
out of memory:
psycopg2.errors.OutOfMemory: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
Tuning memory may help, it's unknown at this point.
This checkin forces a commit to the postgres database after 10,000
rows have been added. This clears out the savepoints for each row and
starts a new transaction.
back_postgresql.py:
Implement commit mechanism in checkpoint_data(). Add two class level
attributes for tracking the number of savepoints and the limit when
the commit should happen.
roundup_admin.py:
implement pragma and dynamically create the config item
RDBMS_SAVEPOINT_LIMIT used by checkpoint_data.
Also fixed formatting of descriptions when using pragma list in
verbose mode.
admin_guide.txt, upgrading.txt:
Document change and use of pragma savepoint_limit in roundup-admin
for changing the default of 10,000.
test/db_test_base.py:
add some more asserts. In existing testAdminImportExport, set the
savepoint limit to 5 to test setting method and so that the commit
code will be run by existing tests. This provides coverage, but
does not actually test that the commit is done every 5 savepoints
8-(. The verification of every 5 savepoints was done manually
using a pdb breakpoint just before the commit.
acknowledgements.txt:
Added 2.4.0 section mentioning Norbert as he has done a ton of
testing with much larger datasets than I can test with.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 19 Oct 2023 16:11:25 -0400 |
| parents | b21ebabeb2ef |
| children | 3071db43bfb6 |
line wrap: on
line source
.. index:: mysql; deployment notes ============= MySQL Backend ============= This notes detail the MySQL backend for the Roundup issue tracker. Prerequisites ============= To use MySQL as the backend for storing roundup data, you also need to install: 1. MySQL RDBMS 4.0.18 or higher - https://www.mysql.com/. Your MySQL installation MUST support InnoDB tables (or Berkeley DB (BDB) tables if you have no other choice). If you're running < 4.0.18 (but not <4.0) then you'll need to use BDB to pass all unit tests. Edit the ``roundup/backends/back_mysql.py`` file to enable DBD instead of InnoDB. 2. Python MySQL interface - https://pypi.org/project/mysqlclient/ Other Configuration =================== If you are indexing large documents (e.g attached file contents) using MySQL, you may need to increase the max_allowed_packet size. If you don't you can see the error:: 'MySql Server has gone away (2006)' To do this edit /etc/my.conf and change:: [mysqld] max_allowed_packet = 1M the 'max_allowed_packet' value from '1M' to '64M' or larger. Alternatively you can install an alternate indexer (whoosh, xapian etc.) and force the tracker to use it by setting the ``indexer`` setting in the tracker's ``config.ini``. This fix was supplied by telsch. See issue https://issues.roundup-tracker.org/issue2550743 for further info or if you are interested in developing a patch to roundup to help work around this issue. Running the MySQL tests ======================= Roundup tests expect an empty MySQL database. Two alternate ways to provide this: 1. If you have root permissions on the MySQL server, you can create the necessary database entries using the following SQL sequence. Use ``mysql`` on the command line to enter:: CREATE DATABASE rounduptest; USE rounduptest; GRANT ALL PRIVILEGES ON rounduptest.* TO rounduptest@localhost IDENTIFIED BY 'rounduptest'; FLUSH PRIVILEGES; 2. If your administrator has provided you with database connection info, see the config values in 'test/db_test_base.py' about which database connection, name and user will be used. The MySQL database should not contain any tables. Tests will not drop the database with existing data. Note that ``rounduptest`` is a well known account. You should delete it and the ``rounduptest`` database and create a new user/database for production use. Showing MySQL who's boss ======================== If things ever get to the point where that test database is totally hosed, just:: $ su - # /etc/init.d/mysql stop # rm -rf /var/lib/mysql/rounduptest # /etc/init.d/mysql start and all will be better (note that on some systems, ``mysql`` is spelt ``mysqld``).