annotate detectors/irker.py @ 5525:bb7865241f8a

Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975). The roundup-admin export and import commands are used for migrating between different database backends. It is desirable that they should be usable also for migrations between Python 2 and Python 3, and in some cases (e.g. with the anydbm backend) this may be required. To be usable for such migrations, the format of the generated CSV files needs to be stable, meaning the same as currently used with Python 2. The export process uses repr() to produce the fields in the CSV files and eval() to convert them back to Python data structures. repr() of strings with non-ASCII characters produces different results for Python 2 and Python 3. This patch adds repr_export and eval_import functions to roundup/anypy/strings.py which provide the required operations that are just repr() and eval() in Python 2, but are more complicated in Python 3 to use data representations compatible with Python 2. These functions are then used in the required places for export and import. repr() and eval() are also used in storing the dict of changed values in the journal for the RDBMS backends. It is similarly desirable that the database be compatible between Python 2 and Python 3, so that export and import do not need to be used for a migration between Python versions for non-anydbm back ends. Thus, this patch changes rdbms_common.py in the places involved in storing journals in the database, not just in those involved in import/export. Given this patch, import/export with non-ASCII characters appear based on some limited testing to work across Python versions, and an instance using the sqlite backend appears to be compatible between Python versions without needing import/export, *if* the sessions/otks databases (which use anydbm) are deleted when changing Python version.
author Joseph Myers <jsm@polyomino.org.uk>
date Sun, 02 Sep 2018 23:48:04 +0000
parents 64b05e24dbd8
children 72f549e68534
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
4695
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
1 # This detector sends notification on IRC through an irker daemon
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
2 # (http://www.catb.org/esr/irker/) when issues are created or messages
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
3 # are added.
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
4 #
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
5 # Written by Ezio Melotti
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
6 #
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
7 # Requires a running irkerd daemon to work. See the irker documentation
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
8 # for more information about installing, configuring, and running irker.
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
9 #
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
10 # Add the IRC channel(s) that should receive notifications in
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
11 # detectors/config.ini as a comma-separated list, using this format:
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
12 #
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
13 # [irker]
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
14 # channels = irc://chat.freenode.net/channelname
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
15 #
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
16
5376
64b05e24dbd8 Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents: 4695
diff changeset
17 from __future__ import print_function
4695
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
18 import re
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
19 import json
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
20 import socket
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
21
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
22 IRKER_HOST = 'localhost'
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
23 IRKER_PORT = 6659
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
24
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
25 max_content = 120
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
26
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
27 TEMPLATE = ('%(green)s%(author)s%(reset)s '
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
28 '%(bluish)s#%(nodeid)s%(reset)s/%(title)s%(bold)s:%(bold)s '
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
29 '%(log)s %(url)s')
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
30
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
31
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
32 def sendmsg(msg):
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
33 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
34 try:
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
35 sock.connect((IRKER_HOST, IRKER_PORT))
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
36 sock.sendall(msg + "\n")
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
37 finally:
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
38 sock.close()
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
39
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
40
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
41 def notify_irker(db, cl, nodeid, oldvalues):
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
42 messages = set(cl.get(nodeid, 'messages'))
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
43 if oldvalues:
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
44 messages -= set(oldvalues.get('messages', ()))
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
45 if not messages:
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
46 return
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
47 messages = list(messages)
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
48
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
49 if oldvalues:
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
50 oldstatus = oldvalues['status']
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
51 else:
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
52 oldstatus = None
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
53 newstatus = db.issue.get(nodeid, 'status')
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
54 if oldstatus != newstatus:
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
55 if oldvalues:
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
56 status = db.status.get(newstatus, 'name')
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
57 else:
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
58 status = 'new'
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
59 log = '[' + status + '] '
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
60 else:
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
61 log = ''
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
62 for msg in messages:
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
63 log += db.msg.get(msg, 'content')
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
64 if len(log) > max_content:
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
65 log = log[:max_content-3] + '...'
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
66 log = re.sub('\s+', ' ', log)
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
67
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
68 # include irc colors
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
69 params = {
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
70 'bold': '\x02',
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
71 'green': '\x0303',
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
72 'blue': '\x0302',
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
73 'bluish': '\x0310',
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
74 'yellow': '\x0307',
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
75 'brown': '\x0305',
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
76 'reset': '\x0F'
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
77 }
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
78 # extend with values used in the template
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
79 params['author'] = db.user.get(db.getuid(), 'username')
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
80 params['nodeid'] = nodeid
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
81 params['title'] = db.issue.get(nodeid, 'title')
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
82 params['log'] = log
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
83 params['url'] = '%sissue%s' % (db.config.TRACKER_WEB, nodeid)
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
84
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
85 # create the message and use the list of channels defined in
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
86 # detectors/config.ini
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
87 msg = json.dumps({
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
88 'to': db.config.detectors.IRKER_CHANNELS.split(','),
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
89 'privmsg': TEMPLATE % params,
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
90 })
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
91
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
92 try:
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
93 sendmsg(msg)
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
94 except Exception as e:
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
95 # Ignore any errors in sending the irker;
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
96 # if the server is down, that's just bad luck
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
97 # XXX might want to do some logging here
5376
64b05e24dbd8 Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents: 4695
diff changeset
98 print('* Sending message to irker failed', str(e))
4695
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
99
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
100 def init(db):
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
101 db.issue.react('create', notify_irker)
132650f4700a #2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff changeset
102 db.issue.react('set', notify_irker)

Roundup Issue Tracker: http://roundup-tracker.org/