Mercurial > p > roundup > code
annotate test/test_instance.py @ 7668:5b41018617f2
fix: out of memory error when importing under postgresql
If you try importing more than 20k items under postgresql you can run
out of memory:
psycopg2.errors.OutOfMemory: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
Tuning memory may help, it's unknown at this point.
This checkin forces a commit to the postgres database after 10,000
rows have been added. This clears out the savepoints for each row and
starts a new transaction.
back_postgresql.py:
Implement commit mechanism in checkpoint_data(). Add two class level
attributes for tracking the number of savepoints and the limit when
the commit should happen.
roundup_admin.py:
implement pragma and dynamically create the config item
RDBMS_SAVEPOINT_LIMIT used by checkpoint_data.
Also fixed formatting of descriptions when using pragma list in
verbose mode.
admin_guide.txt, upgrading.txt:
Document change and use of pragma savepoint_limit in roundup-admin
for changing the default of 10,000.
test/db_test_base.py:
add some more asserts. In existing testAdminImportExport, set the
savepoint limit to 5 to test setting method and so that the commit
code will be run by existing tests. This provides coverage, but
does not actually test that the commit is done every 5 savepoints
8-(. The verification of every 5 savepoints was done manually
using a pdb breakpoint just before the commit.
acknowledgements.txt:
Added 2.4.0 section mentioning Norbert as he has done a ton of
testing with much larger datasets than I can test with.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 19 Oct 2023 16:11:25 -0400 |
| parents | 778a9f455067 |
| children | 9c3ec0a5c7fc |
| rev | line source |
|---|---|
|
6300
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
1 # |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
2 # Copyright (C) 2020 John Rouillard |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
3 # All rights reserved. |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
4 # For license terms see the file COPYING.txt. |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
5 # |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
6 |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
7 from __future__ import print_function |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
8 import unittest, os, shutil, errno, sys, difflib |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
9 |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
10 from roundup import instance |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
11 from roundup.instance import TrackerError |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
12 |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
13 try: |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
14 # python2 |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
15 import pathlib2 as pathlib |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
16 except ImportError: |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
17 # python3 |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
18 import pathlib |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
19 |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
20 from . import db_test_base |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
21 |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
22 class InstanceTest(unittest.TestCase): |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
23 |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
24 backend = 'anydbm' |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
25 |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
26 def setUp(self): |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
27 self.dirname = '_test_instance' |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
28 # set up and open a tracker |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
29 self.instance = db_test_base.setupTracker(self.dirname, self.backend) |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
30 |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
31 # open the database |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
32 self.db = self.instance.open('admin') |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
33 |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
34 self.db.commit() |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
35 self.db.close() |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
36 |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
37 def tearDown(self): |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
38 if self.db: |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
39 self.db.close() |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
40 try: |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
41 shutil.rmtree(self.dirname) |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
42 except OSError as error: |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
43 if error.errno not in (errno.ENOENT, errno.ESRCH): raise |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
44 |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
45 |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
46 def testOpenOldStyle(self): |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
47 pathlib.Path(os.path.join(self.dirname, "dbinit.py")).touch() |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
48 # no longer support old style tracker configs |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
49 self.assertRaises(TrackerError, instance.open, self.dirname) |
|
778a9f455067
Remove old code import imp, old style trackers db/backend_name
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
50 |
