Mercurial > p > roundup > code
comparison roundup/token.py @ 6017:f023edbeb24d
flake8 cleanup: remove unused var; whitespace changes.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Tue, 31 Dec 2019 21:57:29 -0500 |
| parents | 4b2c6f7bc9b4 |
| children |
comparison
equal
deleted
inserted
replaced
| 6016:9223ed67af05 | 6017:f023edbeb24d |
|---|---|
| 5 # disclaimer are retained in their original form. | 5 # disclaimer are retained in their original form. |
| 6 # | 6 # |
| 7 # This module is distributed in the hope that it will be useful, | 7 # This module is distributed in the hope that it will be useful, |
| 8 # but WITHOUT ANY WARRANTY; without even the implied warranty of | 8 # but WITHOUT ANY WARRANTY; without even the implied warranty of |
| 9 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. | 9 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. |
| 10 # | 10 # |
| 11 | 11 |
| 12 """This module provides the tokeniser used by roundup-admin. | 12 """This module provides the tokeniser used by roundup-admin. |
| 13 """ | 13 """ |
| 14 __docformat__ = 'restructuredtext' | 14 __docformat__ = 'restructuredtext' |
| 15 | 15 |
| 16 | |
| 16 def token_split(s, whitespace=' \r\n\t', quotes='\'"', | 17 def token_split(s, whitespace=' \r\n\t', quotes='\'"', |
| 17 escaped={'r':'\r', 'n':'\n', 't':'\t'}): | 18 escaped={'r': '\r', 'n': '\n', 't': '\t'}): |
| 18 r'''Split the string up into tokens. An occurence of a ``'`` or ``"`` in | 19 r'''Split the string up into tokens. An occurence of a ``'`` or ``"`` in |
| 19 the input will cause the splitter to ignore whitespace until a matching | 20 the input will cause the splitter to ignore whitespace until a matching |
| 20 quote char is found. Embedded non-matching quote chars are also skipped. | 21 quote char is found. Embedded non-matching quote chars are also skipped. |
| 21 | 22 |
| 22 Whitespace and quoting characters may be escaped using a backslash. | 23 Whitespace and quoting characters may be escaped using a backslash. |
| 47 ESCAPE = 'escape' | 48 ESCAPE = 'escape' |
| 48 quotechar = '' | 49 quotechar = '' |
| 49 state = NEWTOKEN | 50 state = NEWTOKEN |
| 50 oldstate = '' # one-level state stack ;) | 51 oldstate = '' # one-level state stack ;) |
| 51 length = len(s) | 52 length = len(s) |
| 52 finish = 0 | |
| 53 token = '' | 53 token = '' |
| 54 while 1: | 54 while 1: |
| 55 # end of string, finish off the current token | 55 # end of string, finish off the current token |
| 56 if pos == length: | 56 if pos == length: |
| 57 if state == QUOTE: raise ValueError | 57 if state == QUOTE: raise ValueError |