Mercurial > p > roundup > code
annotate detectors/irker.py @ 5543:bc3e00a3d24b
MySQL backend fixes for Python 3.
With Python 2, text sent to and from MySQL is treated as bytes in
Python. The database may be recorded by MySQL as having some other
encoding (latin1 being the default in some MySQL versions - Roundup
does not set an encoding explicitly, unlike in back_postgresql), but
as long as MySQL's notion of the connection encoding agrees with its
notion of the database encoding, no conversions actually take place
and the bytes are stored and returned as-is.
With Python 3, text sent to and from MySQL is treated as Python
Unicode strings. When the database and connection encoding is latin1,
that means the bytes stored in the database under Python 2 are
interpreted as latin1 and converted from that to Unicode, producing
incorrect results for any non-ASCII characters; furthermore, if trying
to store new non-ASCII data in the database under Python 3, any
non-latin1 characters produce errors.
This patch arranges for both the connection and database character
sets to be UTF-8 when using Python 3, and documents a need to export
and import the database when moving from Python 2 to Python 3 with
this backend.
| author | Joseph Myers <jsm@polyomino.org.uk> |
|---|---|
| date | Sun, 16 Sep 2018 16:19:20 +0000 |
| parents | 64b05e24dbd8 |
| children | 72f549e68534 |
| rev | line source |
|---|---|
|
4695
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
1 # This detector sends notification on IRC through an irker daemon |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
2 # (http://www.catb.org/esr/irker/) when issues are created or messages |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
3 # are added. |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
4 # |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
5 # Written by Ezio Melotti |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
6 # |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
7 # Requires a running irkerd daemon to work. See the irker documentation |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
8 # for more information about installing, configuring, and running irker. |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
9 # |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
10 # Add the IRC channel(s) that should receive notifications in |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
11 # detectors/config.ini as a comma-separated list, using this format: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
12 # |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
13 # [irker] |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
14 # channels = irc://chat.freenode.net/channelname |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
15 # |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
16 |
|
5376
64b05e24dbd8
Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents:
4695
diff
changeset
|
17 from __future__ import print_function |
|
4695
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
18 import re |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
19 import json |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
20 import socket |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
21 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
22 IRKER_HOST = 'localhost' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
23 IRKER_PORT = 6659 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
24 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
25 max_content = 120 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
26 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
27 TEMPLATE = ('%(green)s%(author)s%(reset)s ' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
28 '%(bluish)s#%(nodeid)s%(reset)s/%(title)s%(bold)s:%(bold)s ' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
29 '%(log)s %(url)s') |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
30 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
31 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
32 def sendmsg(msg): |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
33 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
34 try: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
35 sock.connect((IRKER_HOST, IRKER_PORT)) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
36 sock.sendall(msg + "\n") |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
37 finally: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
38 sock.close() |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
39 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
40 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
41 def notify_irker(db, cl, nodeid, oldvalues): |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
42 messages = set(cl.get(nodeid, 'messages')) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
43 if oldvalues: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
44 messages -= set(oldvalues.get('messages', ())) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
45 if not messages: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
46 return |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
47 messages = list(messages) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
48 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
49 if oldvalues: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
50 oldstatus = oldvalues['status'] |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
51 else: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
52 oldstatus = None |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
53 newstatus = db.issue.get(nodeid, 'status') |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
54 if oldstatus != newstatus: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
55 if oldvalues: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
56 status = db.status.get(newstatus, 'name') |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
57 else: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
58 status = 'new' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
59 log = '[' + status + '] ' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
60 else: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
61 log = '' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
62 for msg in messages: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
63 log += db.msg.get(msg, 'content') |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
64 if len(log) > max_content: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
65 log = log[:max_content-3] + '...' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
66 log = re.sub('\s+', ' ', log) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
67 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
68 # include irc colors |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
69 params = { |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
70 'bold': '\x02', |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
71 'green': '\x0303', |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
72 'blue': '\x0302', |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
73 'bluish': '\x0310', |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
74 'yellow': '\x0307', |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
75 'brown': '\x0305', |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
76 'reset': '\x0F' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
77 } |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
78 # extend with values used in the template |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
79 params['author'] = db.user.get(db.getuid(), 'username') |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
80 params['nodeid'] = nodeid |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
81 params['title'] = db.issue.get(nodeid, 'title') |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
82 params['log'] = log |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
83 params['url'] = '%sissue%s' % (db.config.TRACKER_WEB, nodeid) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
84 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
85 # create the message and use the list of channels defined in |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
86 # detectors/config.ini |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
87 msg = json.dumps({ |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
88 'to': db.config.detectors.IRKER_CHANNELS.split(','), |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
89 'privmsg': TEMPLATE % params, |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
90 }) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
91 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
92 try: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
93 sendmsg(msg) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
94 except Exception as e: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
95 # Ignore any errors in sending the irker; |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
96 # if the server is down, that's just bad luck |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
97 # XXX might want to do some logging here |
|
5376
64b05e24dbd8
Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents:
4695
diff
changeset
|
98 print('* Sending message to irker failed', str(e)) |
|
4695
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
99 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
100 def init(db): |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
101 db.issue.react('create', notify_irker) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
102 db.issue.react('set', notify_irker) |
