Mercurial > p > roundup > code
annotate roundup/cgi/TAL/XMLParser.py @ 6593:e70e2789bc2c
issue2551189 - increase text search maxlength
This removes I think all the magic references to 25 and 30 (varchar
size) and replaces them with references to maxlength or maxlength+5.
I am not sure why the db column is 5 characters larger than the size
of what should be the max size of a word, but I'll keep the buffer
of 5 as making it 1/5 the size of maxlength makes less sense.
Also added tests for fts search in templating which were missing.
Added postgres, mysql and sqlite native indexing backends in which to
test fts. Added fts test to native-fts as well to make sure it's
working.
I want to commit this now for CI.
Todo:
add test cases for the use of FTS in the csv output in
actions.py. There is no test coverage of the match case there.
change maxlength to a higher value (50) as requested in the ticket.
Modify existing extremewords test cases to allow words > 25 and < 51
write code to migrate column sizes for mysql and postgresql to match
maxlength I will roll this into the version 7 schema update that
supports use of database fts support.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Tue, 25 Jan 2022 13:22:00 -0500 |
| parents | 14a61eabcea8 |
| children |
| rev | line source |
|---|---|
| 1049 | 1 ############################################################################## |
| 2 # | |
| 3 # Copyright (c) 2001, 2002 Zope Corporation and Contributors. | |
| 4 # All Rights Reserved. | |
|
2348
8c2402a78bb0
beginning getting ZPT up to date: TAL first
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
5 # |
| 1049 | 6 # This software is subject to the provisions of the Zope Public License, |
| 7 # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. | |
| 8 # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED | |
| 9 # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED | |
| 10 # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS | |
| 11 # FOR A PARTICULAR PURPOSE | |
|
2348
8c2402a78bb0
beginning getting ZPT up to date: TAL first
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
12 # |
| 1049 | 13 ############################################################################## |
|
2348
8c2402a78bb0
beginning getting ZPT up to date: TAL first
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
14 # Modifications for Roundup: |
|
8c2402a78bb0
beginning getting ZPT up to date: TAL first
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
15 # 1. commented out zLOG references |
|
5402
88dbacd11cd1
Python 3 preparation: update urllib / urllib2 / urlparse imports.
Joseph Myers <jsm@polyomino.org.uk>
parents:
2348
diff
changeset
|
16 # 2. use roundup.anypy.urllib_ |
|
2348
8c2402a78bb0
beginning getting ZPT up to date: TAL first
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
17 """ |
|
8c2402a78bb0
beginning getting ZPT up to date: TAL first
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
18 Generic expat-based XML parser base class. |
|
8c2402a78bb0
beginning getting ZPT up to date: TAL first
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
19 """ |
|
1071
c08b3820edd1
Adhering to ZPL
Richard Jones <richard@users.sourceforge.net>
parents:
1049
diff
changeset
|
20 |
|
2348
8c2402a78bb0
beginning getting ZPT up to date: TAL first
Richard Jones <richard@users.sourceforge.net>
parents:
2005
diff
changeset
|
21 #import zLOG |
| 1049 | 22 |
| 23 class XMLParser: | |
| 24 | |
| 25 ordered_attributes = 0 | |
| 26 | |
| 27 handler_names = [ | |
| 28 "StartElementHandler", | |
| 29 "EndElementHandler", | |
| 30 "ProcessingInstructionHandler", | |
| 31 "CharacterDataHandler", | |
| 32 "UnparsedEntityDeclHandler", | |
| 33 "NotationDeclHandler", | |
| 34 "StartNamespaceDeclHandler", | |
| 35 "EndNamespaceDeclHandler", | |
| 36 "CommentHandler", | |
| 37 "StartCdataSectionHandler", | |
| 38 "EndCdataSectionHandler", | |
| 39 "DefaultHandler", | |
| 40 "DefaultHandlerExpand", | |
| 41 "NotStandaloneHandler", | |
| 42 "ExternalEntityRefHandler", | |
| 43 "XmlDeclHandler", | |
| 44 "StartDoctypeDeclHandler", | |
| 45 "EndDoctypeDeclHandler", | |
| 46 "ElementDeclHandler", | |
| 47 "AttlistDeclHandler" | |
| 48 ] | |
| 49 | |
| 50 def __init__(self, encoding=None): | |
| 51 self.parser = p = self.createParser() | |
|
5519
14a61eabcea8
Fixed unicode issues for XML template with Python 2
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5402
diff
changeset
|
52 # Make sure we don't get fed unicode strings in Python 2 as we |
|
14a61eabcea8
Fixed unicode issues for XML template with Python 2
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5402
diff
changeset
|
53 # can't handle those |
|
14a61eabcea8
Fixed unicode issues for XML template with Python 2
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5402
diff
changeset
|
54 if hasattr(self.parser, 'returns_unicode'): |
|
14a61eabcea8
Fixed unicode issues for XML template with Python 2
Christof Meerwald <cmeerw@cmeerw.org>
parents:
5402
diff
changeset
|
55 self.parser.returns_unicode = False |
| 1049 | 56 if self.ordered_attributes: |
| 57 try: | |
| 58 self.parser.ordered_attributes = self.ordered_attributes | |
| 59 except AttributeError: | |
| 60 #zLOG.LOG("TAL.XMLParser", zLOG.INFO, | |
| 61 # "Can't set ordered_attributes") | |
| 62 self.ordered_attributes = 0 | |
| 63 for name in self.handler_names: | |
| 64 method = getattr(self, name, None) | |
| 65 if method is not None: | |
| 66 try: | |
| 67 setattr(p, name, method) | |
| 68 except AttributeError: | |
| 69 #zLOG.LOG("TAL.XMLParser", zLOG.PROBLEM, | |
| 70 # "Can't set expat handler %s" % name) | |
| 71 pass | |
| 72 | |
| 73 def createParser(self, encoding=None): | |
| 74 global XMLParseError | |
| 75 try: | |
| 76 from Products.ParsedXML.Expat import pyexpat | |
| 77 XMLParseError = pyexpat.ExpatError | |
| 78 return pyexpat.ParserCreate(encoding, ' ') | |
| 79 except ImportError: | |
| 80 from xml.parsers import expat | |
| 81 XMLParseError = expat.ExpatError | |
| 82 return expat.ParserCreate(encoding, ' ') | |
| 83 | |
| 84 def parseFile(self, filename): | |
| 85 self.parseStream(open(filename)) | |
| 86 | |
| 87 def parseString(self, s): | |
| 88 self.parser.Parse(s, 1) | |
| 89 | |
| 90 def parseURL(self, url): | |
|
5402
88dbacd11cd1
Python 3 preparation: update urllib / urllib2 / urlparse imports.
Joseph Myers <jsm@polyomino.org.uk>
parents:
2348
diff
changeset
|
91 import roundup.anypy.urllib_ |
|
88dbacd11cd1
Python 3 preparation: update urllib / urllib2 / urlparse imports.
Joseph Myers <jsm@polyomino.org.uk>
parents:
2348
diff
changeset
|
92 self.parseStream(roundup.anypy.urllib_.urlopen(url)) |
| 1049 | 93 |
| 94 def parseStream(self, stream): | |
| 95 self.parser.ParseFile(stream) | |
| 96 | |
| 97 def parseFragment(self, s, end=0): | |
| 98 self.parser.Parse(s, end) |
