Mercurial > p > roundup > code

annotate test/test_token.py @ 8526:0586f25eb25e

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
ruff: exclusions on imports ignore import locations and import used for side effect
author John Rouillard <rouilj@ieee.org>
date Fri, 20 Feb 2026 16:30:50 -0500
parents 9a74dfeb8620
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
470
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
1 #
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
2 # Copyright (c) 2001 Richard Jones
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
3 # This module is free software, and you may redistribute it and/or modify
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
4 # under the same terms as Python, so long as this copyright message and
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
5 # disclaimer are retained in their original form.
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
6 #
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
7 # This module is distributed in the hope that it will be useful,
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
8 # but WITHOUT ANY WARRANTY; without even the implied warranty of
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
9 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
10
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
11 import unittest, time
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
12
7181
6971c9249c6d Fix missed roundup.token import.
John Rouillard <rouilj@ieee.org>
parents: 5037
diff changeset
13 from roundup.token_r import token_split
470
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
14
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
15 class TokenTestCase(unittest.TestCase):
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
16 def testValid(self):
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
17 l = token_split('hello world')
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
18 self.assertEqual(l, ['hello', 'world'])
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
19
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
20 def testIgnoreExtraSpace(self):
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
21 l = token_split('hello world ')
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
22 self.assertEqual(l, ['hello', 'world'])
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
23
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
24 def testQuoting(self):
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
25 l = token_split('"hello world"')
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
26 self.assertEqual(l, ['hello world'])
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
27 l = token_split("'hello world'")
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
28 self.assertEqual(l, ['hello world'])
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
29
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
30 def testEmbedQuote(self):
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
31 l = token_split(r'Roch\'e Compaan')
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
32 self.assertEqual(l, ["Roch'e", "Compaan"])
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
33 l = token_split('address="1 2 3"')
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
34 self.assertEqual(l, ['address=1 2 3'])
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
35
7859
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
36 def testEmbedEscapeQuote(self):
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
37 l = token_split(r'"Roch\'e Compaan"')
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
38 self.assertEqual(l, ["Roch'e Compaan"])
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
39
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
40 l = token_split(r'"Roch\"e Compaan"')
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
41 self.assertEqual(l, ['Roch"e Compaan'])
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
42
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
43 l = token_split(r'sql "COLLATE = \"utf8mb4_unicode_ci\";"')
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
44 self.assertEqual(l, ["sql", 'COLLATE = "utf8mb4_unicode_ci";'])
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
45
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
46 l = token_split(r'''sql 'COLLATE = "utf8mb4_unicode_ci";' ''')
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
47 self.assertEqual(l, ["sql", 'COLLATE = "utf8mb4_unicode_ci";'])
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
48
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
49 l = token_split(r'''sql 'COLLATE = \"utf8mb4_unicode_ci\";' ''')
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
50 self.assertEqual(l, ["sql", 'COLLATE = "utf8mb4_unicode_ci";'])
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
51
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
52 l = token_split(r'''sql 'COLLATE = \'utf8mb4_unicode_ci\';' ''')
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
53 self.assertEqual(l, ["sql", "COLLATE = 'utf8mb4_unicode_ci';"])
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
54
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
55 l = token_split(r'''sql 'new\nline\rneed \ttab' ''')
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
56 self.assertEqual(l, ["sql", "new\nline\rneed \ttab"])
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
57
470
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
58 def testEscaping(self):
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
59 l = token_split('"Roch\'e" Compaan')
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
60 self.assertEqual(l, ["Roch'e", "Compaan"])
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
61 l = token_split(r'hello\ world')
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
62 self.assertEqual(l, ['hello world'])
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
63 l = token_split(r'\\')
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
64 self.assertEqual(l, ['\\'])
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
65 l = token_split(r'\n')
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
66 self.assertEqual(l, ['\n'])
7859
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
67 l = token_split(r'\r')
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
68 self.assertEqual(l, ['\r'])
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
69 l = token_split(r'\t')
9a74dfeb8620 feat: can use escaped tokens inside quotes including quotes.
John Rouillard <rouilj@ieee.org>
parents: 7181
diff changeset
70 self.assertEqual(l, ['\t'])
470
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
71
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
72 def testBadQuote(self):
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
73 self.assertRaises(ValueError, token_split, '"hello world')
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
74 self.assertRaises(ValueError, token_split, "Roch'e Compaan")
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
75
9f7320624bc2 Added better tokenising to roundup-admin - handles spaces and stuff.
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
76 # vim: set filetype=python ts=4 sw=4 et si
Roundup Issue Tracker: http://roundup-tracker.org/