Mercurial > p > roundup > code

comparison tools/pygettext.py @ 465:70b0809cd15c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Updated to version 1.4 (python 2.2) version of pygettext
author Jürgen Hermann <jhermann@users.sourceforge.net>
date Wed, 19 Dec 2001 00:32:13 +0000
parents a787b2413997
children b2d6657cd2a6
comparison
equal deleted inserted replaced
464:29f5ac8a0d2b 465:70b0809cd15c
1 #! /usr/bin/env python 1 #! /usr/bin/env python
2 # Originally written by Barry Warsaw <bwarsaw@python.org> 2 # Originally written by Barry Warsaw <barry@zope.com>
3 # 3 #
4 # minimally patched to make it even more xgettext compatible 4 # Minimally patched to make it even more xgettext compatible
5 # by Peter Funk <pf@artcom-gmbh.de> 5 # by Peter Funk <pf@artcom-gmbh.de>
6 # 6 #
7 # 2001-11-21 Jürgen Hermann <jh@web.de> 7 # 2001-12-18 Jürgen Hermann <jh@web.de>
8 # Checks that _() only contains string literals added, and 8 # Added checks that _() only contains string literals, and
9 # command line args are resolved to module lists, i.e. you 9 # command line args are resolved to module lists, i.e. you
10 # can now pass a filename, a module or package name, or a 10 # can now pass a filename, a module or package name, or a
11 # directory (including globbing chars, important for Win32). 11 # directory (including globbing chars, important for Win32).
12 # Made docstring fit in 80 chars wide displays using pydoc.
12 # 13 #
13 14
14 # for selftesting 15 # for selftesting
15 try: 16 try:
16 import fintl 17 import fintl
17 _ = fintl.gettext 18 _ = fintl.gettext
18 except ImportError: 19 except ImportError:
19 _ = lambda s: s 20 _ = lambda s: s
20 21
21
22 __doc__ = _("""pygettext -- Python equivalent of xgettext(1) 22 __doc__ = _("""pygettext -- Python equivalent of xgettext(1)
23 23
24 Many systems (Solaris, Linux, Gnu) provide extensive tools that ease the 24 Many systems (Solaris, Linux, Gnu) provide extensive tools that ease the
25 internationalization of C programs. Most of these tools are independent of 25 internationalization of C programs. Most of these tools are independent of
26 the programming language and can be used from within Python programs. Martin 26 the programming language and can be used from within Python programs.
27 von Loewis' work[1] helps considerably in this regard. 27 Martin von Loewis' work[1] helps considerably in this regard.
28 28
29 There's one problem though; xgettext is the program that scans source code 29 There's one problem though; xgettext is the program that scans source code
30 looking for message strings, but it groks only C (or C++). Python introduces 30 looking for message strings, but it groks only C (or C++). Python
31 a few wrinkles, such as dual quoting characters, triple quoted strings, and 31 introduces a few wrinkles, such as dual quoting characters, triple quoted
32 raw strings. xgettext understands none of this. 32 strings, and raw strings. xgettext understands none of this.
33 33
34 Enter pygettext, which uses Python's standard tokenize module to scan Python 34 Enter pygettext, which uses Python's standard tokenize module to scan
35 source code, generating .pot files identical to what GNU xgettext[2] generates 35 Python source code, generating .pot files identical to what GNU xgettext[2]
36 for C and C++ code. From there, the standard GNU tools can be used. 36 generates for C and C++ code. From there, the standard GNU tools can be
37 37 used.
38 A word about marking Python strings as candidates for translation. GNU 38
39 xgettext recognizes the following keywords: gettext, dgettext, dcgettext, and 39 A word about marking Python strings as candidates for translation. GNU
40 gettext_noop. But those can be a lot of text to include all over your code. 40 xgettext recognizes the following keywords: gettext, dgettext, dcgettext,
41 C and C++ have a trick: they use the C preprocessor. Most internationalized C 41 and gettext_noop. But those can be a lot of text to include all over your
42 source includes a #define for gettext() to _() so that what has to be written 42 code. C and C++ have a trick: they use the C preprocessor. Most
43 in the source is much less. Thus these are both translatable strings: 43 internationalized C source includes a #define for gettext() to _() so that
44 what has to be written in the source is much less. Thus these are both
45 translatable strings:
44 46
45 gettext("Translatable String") 47 gettext("Translatable String")
46 _("Translatable String") 48 _("Translatable String")
47 49
48 Python of course has no preprocessor so this doesn't work so well. Thus, 50 Python of course has no preprocessor so this doesn't work so well. Thus,
50 below for how to augment this. 52 below for how to augment this.
51 53
52 [1] http://www.python.org/workshops/1997-10/proceedings/loewis.html 54 [1] http://www.python.org/workshops/1997-10/proceedings/loewis.html
53 [2] http://www.gnu.org/software/gettext/gettext.html 55 [2] http://www.gnu.org/software/gettext/gettext.html
54 56
55 NOTE: pygettext attempts to be option and feature compatible with GNU xgettext 57 NOTE: pygettext attempts to be option and feature compatible with GNU
56 where ever possible. However some options are still missing or are not fully 58 xgettext where ever possible. However some options are still missing or are
57 implemented. Also, xgettext's use of command line switches with option 59 not fully implemented. Also, xgettext's use of command line switches with
58 arguments is broken, and in these cases, pygettext just defines additional 60 option arguments is broken, and in these cases, pygettext just defines
59 switches. 61 additional switches.
60 62
61 Usage: pygettext [options] inputfile ... 63 Usage: pygettext [options] inputfile ...
62 64
63 Options: 65 Options:
64 66
65 -a 67 -a
66 --extract-all 68 --extract-all
67 Extract all strings 69 Extract all strings.
68 70
69 -d name 71 -d name
70 --default-domain=name 72 --default-domain=name
71 Rename the default output file from messages.pot to name.pot 73 Rename the default output file from messages.pot to name.pot.
72 74
73 -E 75 -E
74 --escape 76 --escape
75 replace non-ASCII characters with octal escape sequences. 77 Replace non-ASCII characters with octal escape sequences.
78
79 -D
80 --docstrings
81 Extract module, class, method, and function docstrings. These do
82 not need to be wrapped in _() markers, and in fact cannot be for
83 Python to consider them docstrings. (See also the -X option).
76 84
77 -h 85 -h
78 --help 86 --help
79 print this help message and exit 87 Print this help message and exit.
80 88
81 -k word 89 -k word
82 --keyword=word 90 --keyword=word
83 Keywords to look for in addition to the default set, which are: 91 Keywords to look for in addition to the default set, which are:
84 %(DEFAULTKEYWORDS)s 92 %(DEFAULTKEYWORDS)s
98 Write filename/lineno location comments indicating where each 106 Write filename/lineno location comments indicating where each
99 extracted string is found in the source. These lines appear before 107 extracted string is found in the source. These lines appear before
100 each msgid. The style of comments is controlled by the -S/--style 108 each msgid. The style of comments is controlled by the -S/--style
101 option. This is the default. 109 option. This is the default.
102 110
111 -o filename
112 --output=filename
113 Rename the default output file from messages.pot to filename. If
114 filename is `-' then the output is sent to standard out.
115
116 -p dir
117 --output-dir=dir
118 Output files will be placed in directory dir.
119
103 -S stylename 120 -S stylename
104 --style stylename 121 --style stylename
105 Specify which style to use for location comments. Two styles are 122 Specify which style to use for location comments. Two styles are
106 supported: 123 supported:
107 124
108 Solaris # File: filename, line: line-number 125 Solaris # File: filename, line: line-number
109 GNU #: filename:line 126 GNU #: filename:line
110 127
111 The style name is case insensitive. GNU style is the default. 128 The style name is case insensitive. GNU style is the default.
112
113 -o filename
114 --output=filename
115 Rename the default output file from messages.pot to filename. If
116 filename is `-' then the output is sent to standard out.
117
118 -p dir
119 --output-dir=dir
120 Output files will be placed in directory dir.
121 129
122 -v 130 -v
123 --verbose 131 --verbose
124 Print the names of the files being processed. 132 Print the names of the files being processed.
125 133
135 --exclude-file=filename 143 --exclude-file=filename
136 Specify a file that contains a list of strings that are not be 144 Specify a file that contains a list of strings that are not be
137 extracted from the input files. Each string to be excluded must 145 extracted from the input files. Each string to be excluded must
138 appear on a line by itself in the file. 146 appear on a line by itself in the file.
139 147
148 -X filename
149 --no-docstrings=filename
150 Specify a file that contains a list of files (one per line) that
151 should not have their docstrings extracted. This is only useful in
152 conjunction with the -D option above.
153
140 If `inputfile' is -, standard input is read. 154 If `inputfile' is -, standard input is read.
141
142 """) 155 """)
143 156
144 import os 157 import os
145 import sys 158 import sys
146 import time 159 import time
147 import getopt 160 import getopt
148 import token 161 import token
149 import tokenize 162 import tokenize
150 163 import operator
151 __version__ = '1.1' 164
165 __version__ = '1.5'
152 166
153 default_keywords = ['_'] 167 default_keywords = ['_']
154 DEFAULTKEYWORDS = ', '.join(default_keywords) 168 DEFAULTKEYWORDS = ', '.join(default_keywords)
155 169
156 EMPTYSTRING = '' 170 EMPTYSTRING = ''
157 171
158 172
159 173
160 # The normal pot-file header. msgmerge and EMACS' po-mode work better if 174 # The normal pot-file header. msgmerge and Emacs's po-mode work better if it's
161 # it's there. 175 # there.
162 pot_header = _('''\ 176 pot_header = _('''\
163 # SOME DESCRIPTIVE TITLE. 177 # SOME DESCRIPTIVE TITLE.
164 # Copyright (C) YEAR ORGANIZATION 178 # Copyright (C) YEAR ORGANIZATION
165 # FIRST AUTHOR <EMAIL@ADDRESS>, YEAR. 179 # FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
166 # 180 #
167 msgid "" 181 msgid ""
168 msgstr "" 182 msgstr ""
169 "Project-Id-Version: PACKAGE VERSION\\n" 183 "Project-Id-Version: PACKAGE VERSION\\n"
170 "PO-Revision-Date: %(time)s\\n" 184 "POT-Creation-Date: %(time)s\\n"
185 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\\n"
171 "Last-Translator: FULL NAME <EMAIL@ADDRESS>\\n" 186 "Last-Translator: FULL NAME <EMAIL@ADDRESS>\\n"
172 "Language-Team: LANGUAGE <LL@li.org>\\n" 187 "Language-Team: LANGUAGE <LL@li.org>\\n"
173 "MIME-Version: 1.0\\n" 188 "MIME-Version: 1.0\\n"
174 "Content-Type: text/plain; charset=CHARSET\\n" 189 "Content-Type: text/plain; charset=CHARSET\\n"
175 "Content-Transfer-Encoding: ENCODING\\n" 190 "Content-Transfer-Encoding: ENCODING\\n"
177 192
178 ''') 193 ''')
179 194
180 195
181 def usage(code, msg=''): 196 def usage(code, msg=''):
182 print __doc__ % globals() 197 print >> sys.stderr, __doc__ % globals()
183 if msg: 198 if msg:
184 print msg 199 print >> sys.stderr, msg
185 sys.exit(code) 200 sys.exit(code)
186 201
187 202
188 203
189 escapes = [] 204 escapes = []
237 lineterm = '\\n"\n"' 252 lineterm = '\\n"\n"'
238 s = '""\n"' + lineterm.join(lines) + '"' 253 s = '""\n"' + lineterm.join(lines) + '"'
239 return s 254 return s
240 255
241 256
242
243 def containsAny(str, set): 257 def containsAny(str, set):
244 """ Check whether 'str' contains ANY of the chars in 'set' 258 """ Check whether 'str' contains ANY of the chars in 'set'
245 """ 259 """
246 return 1 in [c in str for c in set] 260 return 1 in [c in str for c in set]
247 261
343 self.__options = options 357 self.__options = options
344 self.__messages = {} 358 self.__messages = {}
345 self.__state = self.__waiting 359 self.__state = self.__waiting
346 self.__data = [] 360 self.__data = []
347 self.__lineno = -1 361 self.__lineno = -1
362 self.__freshmodule = 1
363 self.__curfile = None
348 364
349 def __call__(self, ttype, tstring, stup, etup, line): 365 def __call__(self, ttype, tstring, stup, etup, line):
350 # dispatch 366 # dispatch
367 ## import token
368 ## print >> sys.stderr, 'ttype:', token.tok_name[ttype], \
369 ## 'tstring:', tstring
351 self.__state(ttype, tstring, stup[0]) 370 self.__state(ttype, tstring, stup[0])
352 371
353 def __waiting(self, ttype, tstring, lineno): 372 def __waiting(self, ttype, tstring, lineno):
354 if ttype == tokenize.NAME and tstring in self.__options.keywords: 373 opts = self.__options
374 # Do docstring extractions, if enabled
375 if opts.docstrings and not opts.nodocstrings.get(self.__curfile):
376 # module docstring?
377 if self.__freshmodule:
378 if ttype == tokenize.STRING:
379 self.__addentry(safe_eval(tstring), lineno, isdocstring=1)
380 self.__freshmodule = 0
381 elif ttype not in (tokenize.COMMENT, tokenize.NL):
382 self.__freshmodule = 0
383 return
384 # class docstring?
385 if ttype == tokenize.NAME and tstring in ('class', 'def'):
386 self.__state = self.__suiteseen
387 return
388 if ttype == tokenize.NAME and tstring in opts.keywords:
355 self.__state = self.__keywordseen 389 self.__state = self.__keywordseen
390
391 def __suiteseen(self, ttype, tstring, lineno):
392 # ignore anything until we see the colon
393 if ttype == tokenize.OP and tstring == ':':
394 self.__state = self.__suitedocstring
395
396 def __suitedocstring(self, ttype, tstring, lineno):
397 # ignore any intervening noise
398 if ttype == tokenize.STRING:
399 self.__addentry(safe_eval(tstring), lineno, isdocstring=1)
400 self.__state = self.__waiting
401 elif ttype not in (tokenize.NEWLINE, tokenize.INDENT,
402 tokenize.COMMENT):
403 # there was no class docstring
404 self.__state = self.__waiting
356 405
357 def __keywordseen(self, ttype, tstring, lineno): 406 def __keywordseen(self, ttype, tstring, lineno):
358 if ttype == tokenize.OP and tstring == '(': 407 if ttype == tokenize.OP and tstring == '(':
359 self.__data = [] 408 self.__data = []
360 self.__lineno = lineno 409 self.__lineno = lineno
367 # We've seen the last of the translatable strings. Record the 416 # We've seen the last of the translatable strings. Record the
368 # line number of the first line of the strings and update the list 417 # line number of the first line of the strings and update the list
369 # of messages seen. Reset state for the next batch. If there 418 # of messages seen. Reset state for the next batch. If there
370 # were no strings inside _(), then just ignore this entry. 419 # were no strings inside _(), then just ignore this entry.
371 if self.__data: 420 if self.__data:
372 msg = EMPTYSTRING.join(self.__data) 421 self.__addentry(EMPTYSTRING.join(self.__data))
373 if not msg in self.__options.toexclude:
374 entry = (self.__curfile, self.__lineno)
375 linenos = self.__messages.get(msg)
376 if linenos is None:
377 self.__messages[msg] = [entry]
378 else:
379 linenos.append(entry)
380 self.__state = self.__waiting 422 self.__state = self.__waiting
381 elif ttype == tokenize.STRING: 423 elif ttype == tokenize.STRING:
382 self.__data.append(safe_eval(tstring)) 424 self.__data.append(safe_eval(tstring))
383 elif ttype not in [tokenize.COMMENT, token.INDENT, token.DEDENT, 425 elif ttype not in [tokenize.COMMENT, token.INDENT, token.DEDENT,
384 token.NEWLINE, tokenize.NL]: 426 token.NEWLINE, tokenize.NL]:
385 # warn if we see anything else than STRING or whitespace 427 # warn if we see anything else than STRING or whitespace
386 print >>sys.stderr, _('*** %(file)s:%(lineno)s: Seen unexpected token "%(token)s"') % { 428 print >>sys.stderr, _('*** %(file)s:%(lineno)s: Seen unexpected token "%(token)s"') % {
387 'token': tstring, 'file': self.__curfile, 'lineno': self.__lineno} 429 'token': tstring, 'file': self.__curfile, 'lineno': self.__lineno}
388 self.__state = self.__waiting 430 self.__state = self.__waiting
389 431
432 def __addentry(self, msg, lineno=None, isdocstring=0):
433 if lineno is None:
434 lineno = self.__lineno
435 if not msg in self.__options.toexclude:
436 entry = (self.__curfile, lineno)
437 self.__messages.setdefault(msg, {})[entry] = isdocstring
438
390 def set_filename(self, filename): 439 def set_filename(self, filename):
391 self.__curfile = filename 440 self.__curfile = filename
441 self.__freshmodule = 1
392 442
393 def write(self, fp): 443 def write(self, fp):
394 options = self.__options 444 options = self.__options
395 timestamp = time.ctime(time.time()) 445 timestamp = time.ctime(time.time())
396 # common header 446 # The time stamp in the header doesn't have the same format as that
397 try: 447 # generated by xgettext...
398 sys.stdout = fp 448 print >> fp, pot_header % {'time': timestamp, 'version': __version__}
399 # The time stamp in the header doesn't have the same format 449 # Sort the entries. First sort each particular entry's keys, then
400 # as that generated by xgettext... 450 # sort all the entries by their first item.
401 print pot_header % {'time': timestamp, 'version': __version__} 451 reverse = {}
402 for k, v in self.__messages.items(): 452 for k, v in self.__messages.items():
453 keys = v.keys()
454 keys.sort()
455 reverse.setdefault(tuple(keys), []).append((k, v))
456 rkeys = reverse.keys()
457 rkeys.sort()
458 for rkey in rkeys:
459 rentries = reverse[rkey]
460 rentries.sort()
461 for k, v in rentries:
462 isdocstring = 0
463 # If the entry was gleaned out of a docstring, then add a
464 # comment stating so. This is to aid translators who may wish
465 # to skip translating some unimportant docstrings.
466 if reduce(operator.__add__, v.values()):
467 isdocstring = 1
468 # k is the message string, v is a dictionary-set of (filename,
469 # lineno) tuples. We want to sort the entries in v first by
470 # file name and then by line number.
471 v = v.keys()
472 v.sort()
403 if not options.writelocations: 473 if not options.writelocations:
404 pass 474 pass
405 # location comments are different b/w Solaris and GNU: 475 # location comments are different b/w Solaris and GNU:
406 elif options.locationstyle == options.SOLARIS: 476 elif options.locationstyle == options.SOLARIS:
407 for filename, lineno in v: 477 for filename, lineno in v:
408 d = {'filename': filename, 'lineno': lineno} 478 d = {'filename': filename, 'lineno': lineno}
409 print _('# File: %(filename)s, line: %(lineno)d') % d 479 print >>fp, _(
480 '# File: %(filename)s, line: %(lineno)d') % d
410 elif options.locationstyle == options.GNU: 481 elif options.locationstyle == options.GNU:
411 # fit as many locations on one line, as long as the 482 # fit as many locations on one line, as long as the
412 # resulting line length doesn't exceeds 'options.width' 483 # resulting line length doesn't exceeds 'options.width'
413 locline = '#:' 484 locline = '#:'
414 for filename, lineno in v: 485 for filename, lineno in v:
415 d = {'filename': filename, 'lineno': lineno} 486 d = {'filename': filename, 'lineno': lineno}
416 s = _(' %(filename)s:%(lineno)d') % d 487 s = _(' %(filename)s:%(lineno)d') % d
417 if len(locline) + len(s) <= options.width: 488 if len(locline) + len(s) <= options.width:
418 locline = locline + s 489 locline = locline + s
419 else: 490 else:
420 print locline 491 print >> fp, locline
421 locline = "#:" + s 492 locline = "#:" + s
422 if len(locline) > 2: 493 if len(locline) > 2:
423 print locline 494 print >> fp, locline
424 # TBD: sorting, normalizing 495 if isdocstring:
425 print 'msgid', normalize(k) 496 print >> fp, '#, docstring'
426 print 'msgstr ""\n' 497 print >> fp, 'msgid', normalize(k)
427 finally: 498 print >> fp, 'msgstr ""\n'
428 sys.stdout = sys.__stdout__ 499
429 500
430 501
431 def main(): 502 def main():
432 global default_keywords 503 global default_keywords
433 try: 504 try:
434 opts, args = getopt.getopt( 505 opts, args = getopt.getopt(
435 sys.argv[1:], 506 sys.argv[1:],
436 'ad:Ehk:Kno:p:S:Vvw:x:', 507 'ad:DEhk:Kno:p:S:Vvw:x:X:',
437 ['extract-all', 'default-domain', 'escape', 'help', 508 ['extract-all', 'default-domain=', 'escape', 'help',
438 'keyword=', 'no-default-keywords', 509 'keyword=', 'no-default-keywords',
439 'add-location', 'no-location', 'output=', 'output-dir=', 510 'add-location', 'no-location', 'output=', 'output-dir=',
440 'style=', 'verbose', 'version', 'width=', 'exclude-file=', 511 'style=', 'verbose', 'version', 'width=', 'exclude-file=',
512 'docstrings', 'no-docstrings',
441 ]) 513 ])
442 except getopt.error, msg: 514 except getopt.error, msg:
443 usage(1, msg) 515 usage(1, msg)
444 516
445 # for holding option values 517 # for holding option values
456 writelocations = 1 528 writelocations = 1
457 locationstyle = GNU 529 locationstyle = GNU
458 verbose = 0 530 verbose = 0
459 width = 78 531 width = 78
460 excludefilename = '' 532 excludefilename = ''
533 docstrings = 0
534 nodocstrings = {}
461 535
462 options = Options() 536 options = Options()
463 locations = {'gnu' : options.GNU, 537 locations = {'gnu' : options.GNU,
464 'solaris' : options.SOLARIS, 538 'solaris' : options.SOLARIS,
465 } 539 }
472 options.extractall = 1 546 options.extractall = 1
473 elif opt in ('-d', '--default-domain'): 547 elif opt in ('-d', '--default-domain'):
474 options.outfile = arg + '.pot' 548 options.outfile = arg + '.pot'
475 elif opt in ('-E', '--escape'): 549 elif opt in ('-E', '--escape'):
476 options.escape = 1 550 options.escape = 1
551 elif opt in ('-D', '--docstrings'):
552 options.docstrings = 1
477 elif opt in ('-k', '--keyword'): 553 elif opt in ('-k', '--keyword'):
478 options.keywords.append(arg) 554 options.keywords.append(arg)
479 elif opt in ('-K', '--no-default-keywords'): 555 elif opt in ('-K', '--no-default-keywords'):
480 default_keywords = [] 556 default_keywords = []
481 elif opt in ('-n', '--add-location'): 557 elif opt in ('-n', '--add-location'):
500 options.width = int(arg) 576 options.width = int(arg)
501 except ValueError: 577 except ValueError:
502 usage(1, _('--width argument must be an integer: %s') % arg) 578 usage(1, _('--width argument must be an integer: %s') % arg)
503 elif opt in ('-x', '--exclude-file'): 579 elif opt in ('-x', '--exclude-file'):
504 options.excludefilename = arg 580 options.excludefilename = arg
581 elif opt in ('-X', '--no-docstrings'):
582 fp = open(arg)
583 try:
584 while 1:
585 line = fp.readline()
586 if not line:
587 break
588 options.nodocstrings[line[:-1]] = 1
589 finally:
590 fp.close()
505 591
506 # calculate escapes 592 # calculate escapes
507 make_escapes(options.escape) 593 make_escapes(options.escape)
508 594
509 # calculate all keywords 595 # calculate all keywords
514 try: 600 try:
515 fp = open(options.excludefilename) 601 fp = open(options.excludefilename)
516 options.toexclude = fp.readlines() 602 options.toexclude = fp.readlines()
517 fp.close() 603 fp.close()
518 except IOError: 604 except IOError:
519 sys.stderr.write(_("Can't read --exclude-file: %s") % 605 print >> sys.stderr, _(
520 options.excludefilename) 606 "Can't read --exclude-file: %s") % options.excludefilename
521 sys.exit(1) 607 sys.exit(1)
522 else: 608 else:
523 options.toexclude = [] 609 options.toexclude = []
524 610
525 # resolve args to module lists 611 # resolve args to module lists
526 expanded = [] 612 expanded = []
527 for arg in args: 613 for arg in args:
528 expanded.extend(getFilesForName(arg)) 614 if arg == '-':
615 expanded.append(arg)
616 else:
617 expanded.extend(getFilesForName(arg))
529 args = expanded 618 args = expanded
530 619
531 # slurp through all the files 620 # slurp through all the files
532 eater = TokenEater(options) 621 eater = TokenEater(options)
533 for filename in args: 622 for filename in args:
541 print _('Working on %s') % filename 630 print _('Working on %s') % filename
542 fp = open(filename) 631 fp = open(filename)
543 closep = 1 632 closep = 1
544 try: 633 try:
545 eater.set_filename(filename) 634 eater.set_filename(filename)
546 tokenize.tokenize(fp.readline, eater) 635 try:
636 tokenize.tokenize(fp.readline, eater)
637 except tokenize.TokenError, e:
638 print >> sys.stderr, '%s: %s, line %d, column %d' % (
639 e[0], filename, e[1][0], e[1][1])
547 finally: 640 finally:
548 if closep: 641 if closep:
549 fp.close() 642 fp.close()
550 643
551 # write the output 644 # write the output
Roundup Issue Tracker: http://roundup-tracker.org/