annotate test/test_pythonexpr.py @ 7668:5b41018617f2

fix: out of memory error when importing under postgresql If you try importing more than 20k items under postgresql you can run out of memory: psycopg2.errors.OutOfMemory: out of shared memory HINT: You might need to increase max_locks_per_transaction. Tuning memory may help, it's unknown at this point. This checkin forces a commit to the postgres database after 10,000 rows have been added. This clears out the savepoints for each row and starts a new transaction. back_postgresql.py: Implement commit mechanism in checkpoint_data(). Add two class level attributes for tracking the number of savepoints and the limit when the commit should happen. roundup_admin.py: implement pragma and dynamically create the config item RDBMS_SAVEPOINT_LIMIT used by checkpoint_data. Also fixed formatting of descriptions when using pragma list in verbose mode. admin_guide.txt, upgrading.txt: Document change and use of pragma savepoint_limit in roundup-admin for changing the default of 10,000. test/db_test_base.py: add some more asserts. In existing testAdminImportExport, set the savepoint limit to 5 to test setting method and so that the commit code will be run by existing tests. This provides coverage, but does not actually test that the commit is done every 5 savepoints 8-(. The verification of every 5 savepoints was done manually using a pdb breakpoint just before the commit. acknowledgements.txt: Added 2.4.0 section mentioning Norbert as he has done a ton of testing with much larger datasets than I can test with.
author John Rouillard <rouilj@ieee.org>
date Thu, 19 Oct 2023 16:11:25 -0400
parents e70885fe72a4
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
5676
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
1 """
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
2 In Python 3, sometimes TAL "python:" expressions that refer to
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
3 variables but not all variables are recognized. That is in Python 2.7
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
4 all variables used in a TAL "python:" expression are recognized as
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
5 references. In Python 3.5 (perhaps earlier), some TAL "python:"
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
6 expressions refer to variables but the reference generates an error
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
7 like this:
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
8
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
9 <class 'NameError'>: name 'some_tal_variable' is not defined
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
10
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
11 even when the variable is defined. Output after this message lists the
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
12 variable and its value.
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
13 """
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
14
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
15 import unittest
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
16
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
17 from roundup.cgi.PageTemplates.PythonExpr import PythonExpr as PythonExprClass
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
18
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
19 class ExprTest(unittest.TestCase):
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
20 def testExpr(self):
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
21 expr = '[x for x in context.assignedto ' \
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
22 'if x.realname not in user_realnames]'
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
23 pe = PythonExprClass('test', expr, None)
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
24 # Looking at the expression, only context and user_realnames are
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
25 # external variables. The names assignedto and realname are members,
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
26 # and x is local.
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
27 required_names = ['context', 'user_realnames']
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
28 got_names = pe._f_varnames
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
29 for required_name in required_names:
e70885fe72a4 issue2551026: template variable not defined even though it is.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
30 self.assertIn(required_name, got_names)

Roundup Issue Tracker: http://roundup-tracker.org/