Mercurial > p > roundup > code
view test/test_token.py @ 6588:91ab3e0ffcd0
Summary: Add test cases for sqlite fts
Add support for using the FTS5 full text query engine for sqlite.
Also stubbed out some sections for adding postgresql FTS support as
well.
Added nee indexer type native-fts. It is not selected by default. The
indexer=native is used if no indexer is set. This prevents an upgrade
from seeming to wipe out the native index if upgraded and
indexer=native is not explicitly set.
Docs updated. Also changed section headers to sentence case for the
current release notes.
Indexing backend can control if the full text search phrase is broken
into a list of words or passed intact. For backends with query
languages (sqlite and can be enabled for whoosh and xapian) we do not
want the phrase "tokenized" on whitespace.
This also updates the rdbms database version to version 7 to add FTS
table. I will be using the same version when I add postgresql. If
somebody runs this version on postgresql, they will have to manually
add the fts tables for postgresql if they want to use it.
Added a new renderError method to client. This allows errors to be
reported still using page.html rather than raw html. It also supports
templates for any error code. If no template for the error code
(e.g. 400) is found, the error in raw html with no page frame is
shown.
New IndexerQueryError exception to pass back message about query syntax
errors.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Sun, 23 Jan 2022 18:57:45 -0500 |
| parents | 364c54991861 |
| children | 6971c9249c6d |
line wrap: on
line source
# # Copyright (c) 2001 Richard Jones # This module is free software, and you may redistribute it and/or modify # under the same terms as Python, so long as this copyright message and # disclaimer are retained in their original form. # # This module is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. import unittest, time from roundup.token import token_split class TokenTestCase(unittest.TestCase): def testValid(self): l = token_split('hello world') self.assertEqual(l, ['hello', 'world']) def testIgnoreExtraSpace(self): l = token_split('hello world ') self.assertEqual(l, ['hello', 'world']) def testQuoting(self): l = token_split('"hello world"') self.assertEqual(l, ['hello world']) l = token_split("'hello world'") self.assertEqual(l, ['hello world']) def testEmbedQuote(self): l = token_split(r'Roch\'e Compaan') self.assertEqual(l, ["Roch'e", "Compaan"]) l = token_split('address="1 2 3"') self.assertEqual(l, ['address=1 2 3']) def testEscaping(self): l = token_split('"Roch\'e" Compaan') self.assertEqual(l, ["Roch'e", "Compaan"]) l = token_split(r'hello\ world') self.assertEqual(l, ['hello world']) l = token_split(r'\\') self.assertEqual(l, ['\\']) l = token_split(r'\n') self.assertEqual(l, ['\n']) def testBadQuote(self): self.assertRaises(ValueError, token_split, '"hello world') self.assertRaises(ValueError, token_split, "Roch'e Compaan") # vim: set filetype=python ts=4 sw=4 et si
