Mercurial > p > roundup > code
annotate roundup/cgi/engine_jinja2.py @ 6433:c1d3fbcdbfbd
issue2551142 - Import of retired node ... unique constraint failure.
Title: Import of retired node with username after active node fails
with unique constraint failure.
More fixes needed for mysql and postgresql.
mysql: add unique constraint for (keyvalue, __retired__) when
creating class in the database.
On schema change if class is changed, remove the unique
constraint too.
upgrade version of rdbms database from 5 to 6 to add constraint
to all version 5 databases that were created as version 5
and didn't get the unique constraint. Make no changes
on version 5 databases upgraded from version 4, the upgrade
process to 5 added the constraint. Make no changes
to other databases (sqlite, postgres) during upgrade from
version 5 to 6.
postgres: Handle the exception raised on unique constraint violation.
The exception invalidates the database connection so it
can't be used to recover from the exception.
Added two new database methods:
checkpoint_data - performs a db.commit under postgres
does nothing on other backends
restore_connection_on_error - does a db.rollback on
postgres, does nothing on other
backends
with the rollback() done on the connection I can use the
database connection to fixup the import that failed on the
unique constraint. This makes postgres slower but without the
commit after every imported object, the rollback will delete
all the entries done up to this point.
Trying to figure out how to make the caller do_import batch
and recover from this failure is beyond me.
Also dismissed having to process the export csv file before
importing. Pushing that onto a user just seems wrong. Also
since import/export isn't frequently done the lack of
surprise on having a failing import and reduced
load/frustration for the user seems worth it. Also the import
can be run in verbose mode where it prints out a row as it is
processed, so it may take a while, ut the user can get
feedback.
db_test-base.py: add test for upgrade from 5 to 6.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 10 Jun 2021 12:52:05 -0400 |
| parents | ec1f725f5c91 |
| children | 07ce4e4110f5 |
| rev | line source |
|---|---|
|
4743
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
1 """ |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
2 Experimental Jinja2 support for Roundup. It will become less |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
3 experimental when it is completely clear what information is |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
4 passed to template, and when the info is limited to the sane |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
5 minimal set (to avoid Roundup state changes from template). |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
6 |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
7 [ ] fallback mechanizm to use multiple templating engines in |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
8 parallel and aid in incremental translation from one |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
9 engine to another |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
10 |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
11 [ ] define a place for templates |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
12 probably |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
13 TRACKER_HOME/templates/jinja2 |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
14 with |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
15 TRACKER_HOME/templates/INFO.txt |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
16 describing how the dir was created, for example |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
17 "This is a copy of 'classic' template from ..." |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
18 also template fallback mechanizm for multi-engine |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
19 configuration |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
20 [ ] backward compatibility - if no engine is explicitly |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
21 specified, use TRACKER_HOME/html directory |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
22 [ ] copy TEMPLATES-INFO.txt to INFO.txt |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
23 [ ] implement VERSION file in environment for auto |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
24 upgrade |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
25 |
|
4748
e339583eae8e
templating: rename precompileTemplates to precompile
anatoly techtonik <techtonik@gmail.com>
parents:
4743
diff
changeset
|
26 [ ] precompile() is a stub |
|
4743
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
27 |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
28 [ ] add {{ debug() }} dumper to inspect available variables |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
29 https://github.com/mitsuhiko/jinja2/issues/174 |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
30 """ |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
31 |
|
5376
64b05e24dbd8
Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5015
diff
changeset
|
32 from __future__ import print_function |
|
4743
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
33 import jinja2 |
|
4751
a8c95abaf3fb
Work in progress - template based on jinja2, bootstrap and classic schema
Pradip Caulagi <caulagi@gmail.com>
parents:
4750
diff
changeset
|
34 import gettext |
|
5516
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
35 import mimetypes |
|
5449
ddf1cf299ebc
only encode output on Python 2
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5416
diff
changeset
|
36 import sys |
|
4743
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
37 |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
38 # http://jinja.pocoo.org/docs/api/#loaders |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
39 |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
40 from roundup.cgi.templating import context, LoaderBase, TemplateBase |
|
5416
56c9bcdea47f
Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5376
diff
changeset
|
41 from roundup.anypy.strings import s2u |
|
4743
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
42 |
|
6031
0a357d3b3557
Flake8 fixes: Raise NotImplementedError not NotImplemented; whitespace
John Rouillard <rouilj@ieee.org>
parents:
5516
diff
changeset
|
43 |
|
4743
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
44 class Jinja2Loader(LoaderBase): |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
45 def __init__(self, dir): |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
46 self._env = jinja2.Environment( |
|
5516
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
47 loader=jinja2.FileSystemLoader(dir), |
|
6054
ffaf89a4a9d9
use jinja2 i18n extension
Christof Meerwald <cmeerw@cmeerw.org>
parents:
6032
diff
changeset
|
48 extensions=['jinja2.ext.i18n'], |
|
ffaf89a4a9d9
use jinja2 i18n extension
Christof Meerwald <cmeerw@cmeerw.org>
parents:
6032
diff
changeset
|
49 autoescape=True |
|
5516
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
50 ) |
|
4743
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
51 |
|
5015
85484d35f1a2
issue2550811 20% fix: jinja2 template engine now has an example how to use non-ascii unicode contents with a custom filter ('|u').
Bernhard Reiter <bernhard@intevation.de>
parents:
4772
diff
changeset
|
52 # Adding a custom filter that can transform roundup's vars to unicode |
|
85484d35f1a2
issue2550811 20% fix: jinja2 template engine now has an example how to use non-ascii unicode contents with a custom filter ('|u').
Bernhard Reiter <bernhard@intevation.de>
parents:
4772
diff
changeset
|
53 # This is necessary because jinja2 can only deal with unicode objects |
|
85484d35f1a2
issue2550811 20% fix: jinja2 template engine now has an example how to use non-ascii unicode contents with a custom filter ('|u').
Bernhard Reiter <bernhard@intevation.de>
parents:
4772
diff
changeset
|
54 # and roundup uses utf-8 for the internal representation. |
|
85484d35f1a2
issue2550811 20% fix: jinja2 template engine now has an example how to use non-ascii unicode contents with a custom filter ('|u').
Bernhard Reiter <bernhard@intevation.de>
parents:
4772
diff
changeset
|
55 # The automatic conversion will assume 'ascii' and fail sometime. |
|
85484d35f1a2
issue2550811 20% fix: jinja2 template engine now has an example how to use non-ascii unicode contents with a custom filter ('|u').
Bernhard Reiter <bernhard@intevation.de>
parents:
4772
diff
changeset
|
56 # Analysed with roundup 1.5.0 and jinja 2.7.1. See issue2550811. |
|
5516
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
57 self._env.filters["u"] = s2u |
|
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
58 |
|
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
59 def _find(self, tplname): |
|
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
60 for extension in ('', '.html', '.xml'): |
|
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
61 try: |
|
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
62 filename = tplname + extension |
|
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
63 return self._env.get_template(filename) |
|
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
64 except jinja2.TemplateNotFound: |
|
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
65 continue |
|
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
66 |
|
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
67 return None |
|
5015
85484d35f1a2
issue2550811 20% fix: jinja2 template engine now has an example how to use non-ascii unicode contents with a custom filter ('|u').
Bernhard Reiter <bernhard@intevation.de>
parents:
4772
diff
changeset
|
68 |
|
4743
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
69 def check(self, tplname): |
|
5516
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
70 return bool(self._find(tplname)) |
|
4743
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
71 |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
72 def load(self, tplname): |
|
5516
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
73 tpl = self._find(tplname) |
|
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
74 pt = Jinja2ProxyPageTemplate(tpl) |
|
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
75 pt.content_type = mimetypes.guess_type(tpl.filename)[0] or 'text/html' |
|
91391a366d92
fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5449
diff
changeset
|
76 return pt |
|
4743
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
77 |
|
4748
e339583eae8e
templating: rename precompileTemplates to precompile
anatoly techtonik <techtonik@gmail.com>
parents:
4743
diff
changeset
|
78 def precompile(self): |
|
4743
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
79 pass |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
80 |
|
6031
0a357d3b3557
Flake8 fixes: Raise NotImplementedError not NotImplemented; whitespace
John Rouillard <rouilj@ieee.org>
parents:
5516
diff
changeset
|
81 |
|
4743
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
82 class Jinja2ProxyPageTemplate(TemplateBase): |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
83 def __init__(self, template): |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
84 self._tpl = template |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
85 |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
86 def render(self, client, classname, request, **options): |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
87 # [ ] limit the information passed to the minimal necessary set |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
88 c = context(client, self, classname, request) |
|
6054
ffaf89a4a9d9
use jinja2 i18n extension
Christof Meerwald <cmeerw@cmeerw.org>
parents:
6032
diff
changeset
|
89 |
|
ffaf89a4a9d9
use jinja2 i18n extension
Christof Meerwald <cmeerw@cmeerw.org>
parents:
6032
diff
changeset
|
90 c.update({'options': options, |
|
ffaf89a4a9d9
use jinja2 i18n extension
Christof Meerwald <cmeerw@cmeerw.org>
parents:
6032
diff
changeset
|
91 'gettext': lambda s: s2u(client.gettext(s)), |
|
ffaf89a4a9d9
use jinja2 i18n extension
Christof Meerwald <cmeerw@cmeerw.org>
parents:
6032
diff
changeset
|
92 'ngettext': lambda s, p, n: s2u(client.ngettext(s, p, n))}) |
|
5449
ddf1cf299ebc
only encode output on Python 2
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5416
diff
changeset
|
93 s = self._tpl.render(c) |
|
6031
0a357d3b3557
Flake8 fixes: Raise NotImplementedError not NotImplemented; whitespace
John Rouillard <rouilj@ieee.org>
parents:
5516
diff
changeset
|
94 return s if sys.version_info[0] > 2 else \ |
|
0a357d3b3557
Flake8 fixes: Raise NotImplementedError not NotImplemented; whitespace
John Rouillard <rouilj@ieee.org>
parents:
5516
diff
changeset
|
95 s.encode(client.STORAGE_CHARSET, ) |
|
4743
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
96 |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
97 def __getitem__(self, name): |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
98 # [ ] figure out what are these for |
|
6031
0a357d3b3557
Flake8 fixes: Raise NotImplementedError not NotImplemented; whitespace
John Rouillard <rouilj@ieee.org>
parents:
5516
diff
changeset
|
99 raise NotImplementedError |
|
6032
238dafbe521c
Flake8 couple of more whitespace changes.
John Rouillard <rouilj@ieee.org>
parents:
6031
diff
changeset
|
100 # return self._pt[name] |
|
4743
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
101 |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
102 def __getattr__(self, name): |
|
2d6959f1d2df
templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff
changeset
|
103 # [ ] figure out what are these for |
|
6031
0a357d3b3557
Flake8 fixes: Raise NotImplementedError not NotImplemented; whitespace
John Rouillard <rouilj@ieee.org>
parents:
5516
diff
changeset
|
104 raise NotImplementedError |
|
6032
238dafbe521c
Flake8 couple of more whitespace changes.
John Rouillard <rouilj@ieee.org>
parents:
6031
diff
changeset
|
105 # return getattr(self._pt, name) |
