Mercurial > p > roundup > code
view test/test_token.py @ 7668:5b41018617f2
fix: out of memory error when importing under postgresql
If you try importing more than 20k items under postgresql you can run
out of memory:
psycopg2.errors.OutOfMemory: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
Tuning memory may help, it's unknown at this point.
This checkin forces a commit to the postgres database after 10,000
rows have been added. This clears out the savepoints for each row and
starts a new transaction.
back_postgresql.py:
Implement commit mechanism in checkpoint_data(). Add two class level
attributes for tracking the number of savepoints and the limit when
the commit should happen.
roundup_admin.py:
implement pragma and dynamically create the config item
RDBMS_SAVEPOINT_LIMIT used by checkpoint_data.
Also fixed formatting of descriptions when using pragma list in
verbose mode.
admin_guide.txt, upgrading.txt:
Document change and use of pragma savepoint_limit in roundup-admin
for changing the default of 10,000.
test/db_test_base.py:
add some more asserts. In existing testAdminImportExport, set the
savepoint limit to 5 to test setting method and so that the commit
code will be run by existing tests. This provides coverage, but
does not actually test that the commit is done every 5 savepoints
8-(. The verification of every 5 savepoints was done manually
using a pdb breakpoint just before the commit.
acknowledgements.txt:
Added 2.4.0 section mentioning Norbert as he has done a ton of
testing with much larger datasets than I can test with.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 19 Oct 2023 16:11:25 -0400 |
| parents | 6971c9249c6d |
| children | 9a74dfeb8620 |
line wrap: on
line source
# # Copyright (c) 2001 Richard Jones # This module is free software, and you may redistribute it and/or modify # under the same terms as Python, so long as this copyright message and # disclaimer are retained in their original form. # # This module is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. import unittest, time from roundup.token_r import token_split class TokenTestCase(unittest.TestCase): def testValid(self): l = token_split('hello world') self.assertEqual(l, ['hello', 'world']) def testIgnoreExtraSpace(self): l = token_split('hello world ') self.assertEqual(l, ['hello', 'world']) def testQuoting(self): l = token_split('"hello world"') self.assertEqual(l, ['hello world']) l = token_split("'hello world'") self.assertEqual(l, ['hello world']) def testEmbedQuote(self): l = token_split(r'Roch\'e Compaan') self.assertEqual(l, ["Roch'e", "Compaan"]) l = token_split('address="1 2 3"') self.assertEqual(l, ['address=1 2 3']) def testEscaping(self): l = token_split('"Roch\'e" Compaan') self.assertEqual(l, ["Roch'e", "Compaan"]) l = token_split(r'hello\ world') self.assertEqual(l, ['hello world']) l = token_split(r'\\') self.assertEqual(l, ['\\']) l = token_split(r'\n') self.assertEqual(l, ['\n']) def testBadQuote(self): self.assertRaises(ValueError, token_split, '"hello world') self.assertRaises(ValueError, token_split, "Roch'e Compaan") # vim: set filetype=python ts=4 sw=4 et si
