annotate roundup/cgi/engine_jinja2.py @ 6593:e70e2789bc2c

issue2551189 - increase text search maxlength This removes I think all the magic references to 25 and 30 (varchar size) and replaces them with references to maxlength or maxlength+5. I am not sure why the db column is 5 characters larger than the size of what should be the max size of a word, but I'll keep the buffer of 5 as making it 1/5 the size of maxlength makes less sense. Also added tests for fts search in templating which were missing. Added postgres, mysql and sqlite native indexing backends in which to test fts. Added fts test to native-fts as well to make sure it's working. I want to commit this now for CI. Todo: add test cases for the use of FTS in the csv output in actions.py. There is no test coverage of the match case there. change maxlength to a higher value (50) as requested in the ticket. Modify existing extremewords test cases to allow words > 25 and < 51 write code to migrate column sizes for mysql and postgresql to match maxlength I will roll this into the version 7 schema update that supports use of database fts support.
author John Rouillard <rouilj@ieee.org>
date Tue, 25 Jan 2022 13:22:00 -0500
parents ec1f725f5c91
children 07ce4e4110f5
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
4743
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
1 """
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
2 Experimental Jinja2 support for Roundup. It will become less
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
3 experimental when it is completely clear what information is
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
4 passed to template, and when the info is limited to the sane
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
5 minimal set (to avoid Roundup state changes from template).
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
6
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
7 [ ] fallback mechanizm to use multiple templating engines in
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
8 parallel and aid in incremental translation from one
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
9 engine to another
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
10
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
11 [ ] define a place for templates
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
12 probably
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
13 TRACKER_HOME/templates/jinja2
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
14 with
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
15 TRACKER_HOME/templates/INFO.txt
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
16 describing how the dir was created, for example
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
17 "This is a copy of 'classic' template from ..."
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
18 also template fallback mechanizm for multi-engine
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
19 configuration
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
20 [ ] backward compatibility - if no engine is explicitly
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
21 specified, use TRACKER_HOME/html directory
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
22 [ ] copy TEMPLATES-INFO.txt to INFO.txt
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
23 [ ] implement VERSION file in environment for auto
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
24 upgrade
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
25
4748
e339583eae8e templating: rename precompileTemplates to precompile
anatoly techtonik <techtonik@gmail.com>
parents: 4743
diff changeset
26 [ ] precompile() is a stub
4743
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
27
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
28 [ ] add {{ debug() }} dumper to inspect available variables
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
29 https://github.com/mitsuhiko/jinja2/issues/174
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
30 """
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
31
5376
64b05e24dbd8 Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5015
diff changeset
32 from __future__ import print_function
4743
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
33 import jinja2
4751
a8c95abaf3fb Work in progress - template based on jinja2, bootstrap and classic schema
Pradip Caulagi <caulagi@gmail.com>
parents: 4750
diff changeset
34 import gettext
5516
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
35 import mimetypes
5449
ddf1cf299ebc only encode output on Python 2
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5416
diff changeset
36 import sys
4743
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
37
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
38 # http://jinja.pocoo.org/docs/api/#loaders
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
39
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
40 from roundup.cgi.templating import context, LoaderBase, TemplateBase
5416
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5376
diff changeset
41 from roundup.anypy.strings import s2u
4743
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
42
6031
0a357d3b3557 Flake8 fixes: Raise NotImplementedError not NotImplemented; whitespace
John Rouillard <rouilj@ieee.org>
parents: 5516
diff changeset
43
4743
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
44 class Jinja2Loader(LoaderBase):
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
45 def __init__(self, dir):
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
46 self._env = jinja2.Environment(
5516
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
47 loader=jinja2.FileSystemLoader(dir),
6054
ffaf89a4a9d9 use jinja2 i18n extension
Christof Meerwald <cmeerw@cmeerw.org>
parents: 6032
diff changeset
48 extensions=['jinja2.ext.i18n'],
ffaf89a4a9d9 use jinja2 i18n extension
Christof Meerwald <cmeerw@cmeerw.org>
parents: 6032
diff changeset
49 autoescape=True
5516
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
50 )
4743
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
51
5015
85484d35f1a2 issue2550811 20% fix: jinja2 template engine now has an example how to use non-ascii unicode contents with a custom filter ('|u').
Bernhard Reiter <bernhard@intevation.de>
parents: 4772
diff changeset
52 # Adding a custom filter that can transform roundup's vars to unicode
85484d35f1a2 issue2550811 20% fix: jinja2 template engine now has an example how to use non-ascii unicode contents with a custom filter ('|u').
Bernhard Reiter <bernhard@intevation.de>
parents: 4772
diff changeset
53 # This is necessary because jinja2 can only deal with unicode objects
85484d35f1a2 issue2550811 20% fix: jinja2 template engine now has an example how to use non-ascii unicode contents with a custom filter ('|u').
Bernhard Reiter <bernhard@intevation.de>
parents: 4772
diff changeset
54 # and roundup uses utf-8 for the internal representation.
85484d35f1a2 issue2550811 20% fix: jinja2 template engine now has an example how to use non-ascii unicode contents with a custom filter ('|u').
Bernhard Reiter <bernhard@intevation.de>
parents: 4772
diff changeset
55 # The automatic conversion will assume 'ascii' and fail sometime.
85484d35f1a2 issue2550811 20% fix: jinja2 template engine now has an example how to use non-ascii unicode contents with a custom filter ('|u').
Bernhard Reiter <bernhard@intevation.de>
parents: 4772
diff changeset
56 # Analysed with roundup 1.5.0 and jinja 2.7.1. See issue2550811.
5516
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
57 self._env.filters["u"] = s2u
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
58
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
59 def _find(self, tplname):
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
60 for extension in ('', '.html', '.xml'):
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
61 try:
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
62 filename = tplname + extension
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
63 return self._env.get_template(filename)
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
64 except jinja2.TemplateNotFound:
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
65 continue
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
66
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
67 return None
5015
85484d35f1a2 issue2550811 20% fix: jinja2 template engine now has an example how to use non-ascii unicode contents with a custom filter ('|u').
Bernhard Reiter <bernhard@intevation.de>
parents: 4772
diff changeset
68
4743
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
69 def check(self, tplname):
5516
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
70 return bool(self._find(tplname))
4743
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
71
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
72 def load(self, tplname):
5516
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
73 tpl = self._find(tplname)
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
74 pt = Jinja2ProxyPageTemplate(tpl)
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
75 pt.content_type = mimetypes.guess_type(tpl.filename)[0] or 'text/html'
91391a366d92 fixed issue2550967: support .xml files in addition to .html in the
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5449
diff changeset
76 return pt
4743
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
77
4748
e339583eae8e templating: rename precompileTemplates to precompile
anatoly techtonik <techtonik@gmail.com>
parents: 4743
diff changeset
78 def precompile(self):
4743
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
79 pass
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
80
6031
0a357d3b3557 Flake8 fixes: Raise NotImplementedError not NotImplemented; whitespace
John Rouillard <rouilj@ieee.org>
parents: 5516
diff changeset
81
4743
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
82 class Jinja2ProxyPageTemplate(TemplateBase):
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
83 def __init__(self, template):
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
84 self._tpl = template
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
85
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
86 def render(self, client, classname, request, **options):
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
87 # [ ] limit the information passed to the minimal necessary set
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
88 c = context(client, self, classname, request)
6054
ffaf89a4a9d9 use jinja2 i18n extension
Christof Meerwald <cmeerw@cmeerw.org>
parents: 6032
diff changeset
89
ffaf89a4a9d9 use jinja2 i18n extension
Christof Meerwald <cmeerw@cmeerw.org>
parents: 6032
diff changeset
90 c.update({'options': options,
ffaf89a4a9d9 use jinja2 i18n extension
Christof Meerwald <cmeerw@cmeerw.org>
parents: 6032
diff changeset
91 'gettext': lambda s: s2u(client.gettext(s)),
ffaf89a4a9d9 use jinja2 i18n extension
Christof Meerwald <cmeerw@cmeerw.org>
parents: 6032
diff changeset
92 'ngettext': lambda s, p, n: s2u(client.ngettext(s, p, n))})
5449
ddf1cf299ebc only encode output on Python 2
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5416
diff changeset
93 s = self._tpl.render(c)
6031
0a357d3b3557 Flake8 fixes: Raise NotImplementedError not NotImplemented; whitespace
John Rouillard <rouilj@ieee.org>
parents: 5516
diff changeset
94 return s if sys.version_info[0] > 2 else \
0a357d3b3557 Flake8 fixes: Raise NotImplementedError not NotImplemented; whitespace
John Rouillard <rouilj@ieee.org>
parents: 5516
diff changeset
95 s.encode(client.STORAGE_CHARSET, )
4743
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
96
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
97 def __getitem__(self, name):
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
98 # [ ] figure out what are these for
6031
0a357d3b3557 Flake8 fixes: Raise NotImplementedError not NotImplemented; whitespace
John Rouillard <rouilj@ieee.org>
parents: 5516
diff changeset
99 raise NotImplementedError
6032
238dafbe521c Flake8 couple of more whitespace changes.
John Rouillard <rouilj@ieee.org>
parents: 6031
diff changeset
100 # return self._pt[name]
4743
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
101
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
102 def __getattr__(self, name):
2d6959f1d2df templating: proof of concept for Jinja2 support. Select 'jinja2'
anatoly techtonik <techtonik@gmail.com>
parents:
diff changeset
103 # [ ] figure out what are these for
6031
0a357d3b3557 Flake8 fixes: Raise NotImplementedError not NotImplemented; whitespace
John Rouillard <rouilj@ieee.org>
parents: 5516
diff changeset
104 raise NotImplementedError
6032
238dafbe521c Flake8 couple of more whitespace changes.
John Rouillard <rouilj@ieee.org>
parents: 6031
diff changeset
105 # return getattr(self._pt, name)

Roundup Issue Tracker: http://roundup-tracker.org/