Mercurial > p > roundup > code
annotate test/pytest_patcher.py @ 7668:5b41018617f2
fix: out of memory error when importing under postgresql
If you try importing more than 20k items under postgresql you can run
out of memory:
psycopg2.errors.OutOfMemory: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
Tuning memory may help, it's unknown at this point.
This checkin forces a commit to the postgres database after 10,000
rows have been added. This clears out the savepoints for each row and
starts a new transaction.
back_postgresql.py:
Implement commit mechanism in checkpoint_data(). Add two class level
attributes for tracking the number of savepoints and the limit when
the commit should happen.
roundup_admin.py:
implement pragma and dynamically create the config item
RDBMS_SAVEPOINT_LIMIT used by checkpoint_data.
Also fixed formatting of descriptions when using pragma list in
verbose mode.
admin_guide.txt, upgrading.txt:
Document change and use of pragma savepoint_limit in roundup-admin
for changing the default of 10,000.
test/db_test_base.py:
add some more asserts. In existing testAdminImportExport, set the
savepoint limit to 5 to test setting method and so that the commit
code will be run by existing tests. This provides coverage, but
does not actually test that the commit is done every 5 savepoints
8-(. The verification of every 5 savepoints was done manually
using a pdb breakpoint just before the commit.
acknowledgements.txt:
Added 2.4.0 section mentioning Norbert as he has done a ton of
testing with much larger datasets than I can test with.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 19 Oct 2023 16:11:25 -0400 |
| parents | 1c94afabb2cb |
| children |
| rev | line source |
|---|---|
|
5109
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
1 """ |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
2 The following code was taken from: |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
3 |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
4 https://github.com/pytest-dev/pytest/issues/568#issuecomment-216569420 |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
5 |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
6 to resolve a bug with using pytest.mark.skip(). Once the bug is resolved in |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
7 pytest this file can be removed along with all the wrapper mark_class() |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
8 references in the other test files. |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
9 """ |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
10 import types |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
11 |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
12 |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
13 def mark_class(marker): |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
14 '''Workaround for https://github.com/pytest-dev/pytest/issues/568''' |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
15 def copy_func(f): |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
16 try: |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
17 return types.FunctionType(f.__code__, f.__globals__, |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
18 name=f.__name__, argdefs=f.__defaults__, |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
19 closure=f.__closure__) |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
20 except AttributeError: |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
21 return types.FunctionType(f.func_code, f.func_globals, |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
22 name=f.func_name, |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
23 argdefs=f.func_defaults, |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
24 closure=f.func_closure) |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
25 |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
26 def mark(cls): |
|
5111
1c94afabb2cb
Fix mark_class decorator to work in all cases
John Kristensen <john@jerrykan.com>
parents:
5109
diff
changeset
|
27 if isinstance(cls, types.FunctionType): |
|
1c94afabb2cb
Fix mark_class decorator to work in all cases
John Kristensen <john@jerrykan.com>
parents:
5109
diff
changeset
|
28 return marker(copy_func(cls)) |
|
1c94afabb2cb
Fix mark_class decorator to work in all cases
John Kristensen <john@jerrykan.com>
parents:
5109
diff
changeset
|
29 |
|
5109
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
30 for method in dir(cls): |
|
5111
1c94afabb2cb
Fix mark_class decorator to work in all cases
John Kristensen <john@jerrykan.com>
parents:
5109
diff
changeset
|
31 if method.startswith('test'): |
|
5109
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
32 f = copy_func(getattr(cls, method)) |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
33 setattr(cls, method, marker(f)) |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
34 return cls |
|
43a1f7fe39f5
Improved work-around for pytest markers bug
John Kristensen <john@jerrykan.com>
parents:
diff
changeset
|
35 return mark |
