annotate test/test_anypy.py @ 7668:5b41018617f2

fix: out of memory error when importing under postgresql If you try importing more than 20k items under postgresql you can run out of memory: psycopg2.errors.OutOfMemory: out of shared memory HINT: You might need to increase max_locks_per_transaction. Tuning memory may help, it's unknown at this point. This checkin forces a commit to the postgres database after 10,000 rows have been added. This clears out the savepoints for each row and starts a new transaction. back_postgresql.py: Implement commit mechanism in checkpoint_data(). Add two class level attributes for tracking the number of savepoints and the limit when the commit should happen. roundup_admin.py: implement pragma and dynamically create the config item RDBMS_SAVEPOINT_LIMIT used by checkpoint_data. Also fixed formatting of descriptions when using pragma list in verbose mode. admin_guide.txt, upgrading.txt: Document change and use of pragma savepoint_limit in roundup-admin for changing the default of 10,000. test/db_test_base.py: add some more asserts. In existing testAdminImportExport, set the savepoint limit to 5 to test setting method and so that the commit code will be run by existing tests. This provides coverage, but does not actually test that the commit is done every 5 savepoints 8-(. The verification of every 5 savepoints was done manually using a pdb breakpoint just before the commit. acknowledgements.txt: Added 2.4.0 section mentioning Norbert as he has done a ton of testing with much larger datasets than I can test with.
author John Rouillard <rouilj@ieee.org>
date Thu, 19 Oct 2023 16:11:25 -0400
parents e4db9d0b85c7
children c65e0a725c88
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6532
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
1 """Random tests for anypy modules"""
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
2
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
3
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
4 import unittest
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
5 from roundup.anypy.strings import repr_export, eval_import
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
6
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
7 import sys
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
8 _py3 = sys.version_info[0] > 2
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
9
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
10 class StringsTest(unittest.TestCase):
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
11
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
12 def test_import_params(self):
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
13 """ issue2551170 - handle long int in history/journal
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
14 params tuple
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
15 """
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
16 # python2 export with id as number
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
17 val = eval_import("('issue', 2345L, 'status')")
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
18 self.assertSequenceEqual(val, ('issue', 2345, 'status'))
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
19
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
20 # python3 export with id as number
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
21 val = eval_import("('issue', 2345, 'status')")
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
22 self.assertSequenceEqual(val, ('issue', 2345, 'status'))
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
23
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
24 # python2 or python3 export with id as string
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
25 val = eval_import("('issue', '2345', 'status')")
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
26 self.assertSequenceEqual(val, ('issue', '2345', 'status'))
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
27
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
28 def test_export_params(self):
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
29 """ issue2551170 - handle long int in history/journal
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
30 params tuple
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
31 """
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
32 # python2 export with id as number
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
33 if _py3:
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
34 val = repr_export(('issue', 2345, 'status'))
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
35 self.assertEqual(val, "('issue', 2345, 'status')")
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
36 else:
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
37 val = repr_export(('issue', long(2345), 'status'))
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
38 self.assertEqual(val, "('issue', 2345L, 'status')")
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
39
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
40 # python2 or python3 export with id as string
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
41 val = repr_export(('issue', '2345', 'status'))
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
42 self.assertEqual(val, "('issue', '2345', 'status')")
e4db9d0b85c7 test for issue2551170 process python 2 long under python3
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
43

Roundup Issue Tracker: http://roundup-tracker.org/