annotate roundup/backends/__init__.py @ 5543:bc3e00a3d24b

MySQL backend fixes for Python 3. With Python 2, text sent to and from MySQL is treated as bytes in Python. The database may be recorded by MySQL as having some other encoding (latin1 being the default in some MySQL versions - Roundup does not set an encoding explicitly, unlike in back_postgresql), but as long as MySQL's notion of the connection encoding agrees with its notion of the database encoding, no conversions actually take place and the bytes are stored and returned as-is. With Python 3, text sent to and from MySQL is treated as Python Unicode strings. When the database and connection encoding is latin1, that means the bytes stored in the database under Python 2 are interpreted as latin1 and converted from that to Unicode, producing incorrect results for any non-ASCII characters; furthermore, if trying to store new non-ASCII data in the database under Python 3, any non-latin1 characters produce errors. This patch arranges for both the connection and database character sets to be UTF-8 when using Python 3, and documents a need to export and import the database when moving from Python 2 to Python 3 with this backend.
author Joseph Myers <jsm@polyomino.org.uk>
date Sun, 16 Sep 2018 16:19:20 +0000
parents 22eecc8a8bd4
children 7f00fc5958ca
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
213
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
1 #
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/)
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
3 # This module is free software, and you may redistribute it and/or modify
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
4 # under the same terms as Python, so long as this copyright message and
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
5 # disclaimer are retained in their original form.
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
6 #
214
18134bffab37 stupid typo
Richard Jones <richard@users.sourceforge.net>
parents: 213
diff changeset
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR
213
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
10 # POSSIBILITY OF SUCH DAMAGE.
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
11 #
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS"
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
1244
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1190
diff changeset
17
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
18 '''Container for the hyperdb storage backend implementations.
1244
8dd4f736370b merge from maintenance branch
Richard Jones <richard@users.sourceforge.net>
parents: 1190
diff changeset
19 '''
2005
fc52d57c6c3e documentation cleanup
Richard Jones <richard@users.sourceforge.net>
parents: 1873
diff changeset
20 __docformat__ = 'restructuredtext'
213
d45384bc6420 Added the copyright/license notice to (nearly) all files...
Richard Jones <richard@users.sourceforge.net>
parents: 159
diff changeset
21
2925
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
22 import sys
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
23
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
24 # These names are used to suppress import errors.
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
25 # If get_backend raises an ImportError with appropriate
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
26 # module name, have_backend quietly returns False.
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
27 # Otherwise the error is reraised.
2856
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
28 _modules = {
3718
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3503
diff changeset
29 'mysql': ('MySQLdb',),
5464
22eecc8a8bd4 fixed backend detection for Python 3
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5425
diff changeset
30 'postgresql': ('psycopg2',),
4269
ab4563e97d22 corrections for python2.3 compatibility:
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents: 3941
diff changeset
31 'sqlite': ('pysqlite', 'pysqlite2', 'sqlite3', '_sqlite3', 'sqlite'),
2856
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
32 }
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
33
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
34 def get_backend(name):
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
35 '''Get a specific backend by name.'''
2925
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
36 vars = globals()
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
37 # if requested backend has been imported yet, return current instance
4357
13b3155869e0 Beginnings of a big code cleanup / modernisation to make 2to3 happy
Richard Jones <richard@users.sourceforge.net>
parents: 4269
diff changeset
38 if name in vars:
2925
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
39 return vars[name]
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
40 # import the backend module
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
41 module_name = 'back_%s' % name
5425
ebd6df1126a2 Python 3 preparation: update __import__ call for relative import.
Joseph Myers <jsm@polyomino.org.uk>
parents: 5248
diff changeset
42 module = __import__(module_name, vars, level=1)
4687
4960a2c21590 Python 2.3 workarounds cleanup
anatoly techtonik <techtonik@gmail.com>
parents: 4570
diff changeset
43 vars[name] = module
4960a2c21590 Python 2.3 workarounds cleanup
anatoly techtonik <techtonik@gmail.com>
parents: 4570
diff changeset
44 return module
159
764db91c0dea Checks for ability to import the specific back-end module.
Richard Jones <richard@users.sourceforge.net>
parents: 46
diff changeset
45
2856
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
46 def have_backend(name):
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
47 '''Is backend "name" available?'''
1809
bd127cafe3a8 Simplify backend importing, by moving the imports into the backend modules.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents: 1609
diff changeset
48 try:
2856
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
49 get_backend(name)
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
50 return 1
5248
198b6e810c67 Use Python-3-compatible 'as' syntax for except statements
Eric S. Raymond <esr@thyrsus.com>
parents: 5005
diff changeset
51 except ImportError as e:
5464
22eecc8a8bd4 fixed backend detection for Python 3
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5425
diff changeset
52 if hasattr(e, 'name'):
22eecc8a8bd4 fixed backend detection for Python 3
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5425
diff changeset
53 modname = e.name
22eecc8a8bd4 fixed backend detection for Python 3
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5425
diff changeset
54 else:
22eecc8a8bd4 fixed backend detection for Python 3
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5425
diff changeset
55 modname = e.args[0][16:] if e.args[0].startswith('No module named ') else None
22eecc8a8bd4 fixed backend detection for Python 3
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5425
diff changeset
56
22eecc8a8bd4 fixed backend detection for Python 3
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5425
diff changeset
57 if modname and (modname in _modules.get(name, (name,))):
22eecc8a8bd4 fixed backend detection for Python 3
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5425
diff changeset
58 return 0
3718
0d561b24ceff support sqlite3
Richard Jones <richard@users.sourceforge.net>
parents: 3503
diff changeset
59 raise
2856
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
60 return 0
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
61
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
62 def list_backends():
2925
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
63 '''List all available backend names.
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
64
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
65 This function has side-effect of registering backward-compatible
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
66 globals for all available backends.
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
67
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
68 '''
2856
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
69 l = []
3941
9997b941dd6d remove some metakit references
Richard Jones <richard@users.sourceforge.net>
parents: 3727
diff changeset
70 for name in 'anydbm', 'mysql', 'sqlite', 'postgresql':
2856
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
71 if have_backend(name):
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
72 l.append(name)
adec352e2ce0 don't try to import all backends in backends.__init__ unless we *want* to
Richard Jones <richard@users.sourceforge.net>
parents: 2812
diff changeset
73 return l
854
3cdfa5d86cec Added metakit backend to the db tests...
Richard Jones <richard@users.sourceforge.net>
parents: 733
diff changeset
74
2925
e54c95de4c6a fix: second call to get_backend() succeeded...
Alexander Smishlajev <a1s@users.sourceforge.net>
parents: 2856
diff changeset
75 # vim: set filetype=python sts=4 sw=4 et si :

Roundup Issue Tracker: http://roundup-tracker.org/