Mercurial > p > roundup > code

annotate roundup/anypy/strings.py @ 5418:55f09ca366c4

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Python 3 preparation: StringIO. This generally arranges for StringIO and cStringIO references to use io.StringIO for Python 3 but io.BytesIO for Python 2, consistent with the string representations generally used in Roundup. A special FasterStringIO in the TAL code, which referenced internals of the old Python 2 StringIO module, is cut down so it doesn't actually do anything beyond the StringIO class it inherits from (it would also be reasonable to remove FasterStringIO completely). One place in roundup_server.py clearly needing binary I/O is made to use io.BytesIO unconditionally.
author Joseph Myers <jsm@polyomino.org.uk>
date Wed, 25 Jul 2018 09:08:29 +0000
parents c749d6795bc2
children 6b0c542642be
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
5416
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
1 # Roundup represents text internally using the native Python str type.
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
2 # In Python 3, these are Unicode strings. In Python 2, these are
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
3 # encoded using UTF-8, and the Python 2 unicode type is only used in a
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
4 # few places, generally for interacting with external modules
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
5 # requiring that type to be used.
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
6
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
7 import sys
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
8 _py3 = sys.version_info[0] > 2
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
9
5418
55f09ca366c4 Python 3 preparation: StringIO.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5417
diff changeset
10 import io
55f09ca366c4 Python 3 preparation: StringIO.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5417
diff changeset
11 if _py3:
55f09ca366c4 Python 3 preparation: StringIO.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5417
diff changeset
12 StringIO = io.StringIO
55f09ca366c4 Python 3 preparation: StringIO.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5417
diff changeset
13 else:
55f09ca366c4 Python 3 preparation: StringIO.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5417
diff changeset
14 StringIO = io.BytesIO
55f09ca366c4 Python 3 preparation: StringIO.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5417
diff changeset
15
5416
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
16 def b2s(b):
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
17 """Convert a UTF-8 encoded bytes object to the internal string format."""
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
18 if _py3:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
19 return b.decode('utf-8')
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
20 else:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
21 return b
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
22
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
23 def s2b(s):
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
24 """Convert a string object to UTF-8 encoded bytes."""
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
25 if _py3:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
26 return s.encode('utf-8')
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
27 else:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
28 return s
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
29
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
30 def s2u(s, errors='strict'):
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
31 """Convert a string object to a Unicode string."""
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
32 if _py3:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
33 return s
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
34 else:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
35 return unicode(s, 'utf-8', errors)
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
36
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
37 def u2s(u):
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
38 """Convert a Unicode string to the internal string format."""
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
39 if _py3:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
40 return u
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
41 else:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
42 return u.encode('utf-8')
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
43
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
44 def us2u(s, errors='strict'):
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
45 """Convert a string or Unicode string to a Unicode string."""
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
46 if _py3:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
47 return s
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
48 else:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
49 if isinstance(s, unicode):
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
50 return s
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
51 else:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
52 return unicode(s, 'utf-8', errors)
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
53
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
54 def us2s(u):
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
55 """Convert a string or Unicode string to the internal string format."""
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
56 if _py3:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
57 return u
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
58 else:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
59 if isinstance(u, unicode):
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
60 return u.encode('utf-8')
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
61 else:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
62 return u
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
63
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
64 def uany2s(u):
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
65 """Convert a Unicode string or other object to the internal string format.
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
66
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
67 Objects that are not Unicode strings are passed to str()."""
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
68 if _py3:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
69 return str(u)
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
70 else:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
71 if isinstance(u, unicode):
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
72 return u.encode('utf-8')
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
73 else:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
74 return str(u)
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
75
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
76 def is_us(s):
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
77 """Return whether an object is a string or Unicode string."""
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
78 if _py3:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
79 return isinstance(s, str)
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
80 else:
56c9bcdea47f Python 3 preparation: unicode.
Joseph Myers <jsm@polyomino.org.uk>
parents:
diff changeset
81 return isinstance(s, str) or isinstance(s, unicode)
5417
c749d6795bc2 Python 3 preparation: unichr.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5416
diff changeset
82
c749d6795bc2 Python 3 preparation: unichr.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5416
diff changeset
83 def uchr(c):
c749d6795bc2 Python 3 preparation: unichr.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5416
diff changeset
84 """Return the Unicode string containing the given character."""
c749d6795bc2 Python 3 preparation: unichr.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5416
diff changeset
85 if _py3:
c749d6795bc2 Python 3 preparation: unichr.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5416
diff changeset
86 return chr(c)
c749d6795bc2 Python 3 preparation: unichr.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5416
diff changeset
87 else:
c749d6795bc2 Python 3 preparation: unichr.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5416
diff changeset
88 return unichr(c)
Roundup Issue Tracker: http://roundup-tracker.org/