Mercurial > p > roundup > code
annotate detectors/irker.py @ 6565:2c2dbfc332ba
Try to handle multiple connections better.
The session database is a hot spot. When multiple requests (e.g. 20)
come in at the same time session database contention can get great.
The original code didn't retry session database access when the open
failed. This resulted in errors at the client.
The second pass delayed 0.01 seconds and retried. It was better but we
still had multiple second stalls. I think the first request got in,
everybody else backed up and then retried at the same time. Again they
stepped on each other. With logging I would see many counters go all
the way to low single digits or to -1 indicating falure.
This pass uses randomint to generate delays from 0-.125 seconds in 5ms
increments. This performs better in testing. I rarely saw a counter
less than 13 (2 failed retries). Current logging starts after 6
failures and counts down until success or failure.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 16 Dec 2021 20:02:00 -0500 |
| parents | 64b05e24dbd8 |
| children | 72f549e68534 |
| rev | line source |
|---|---|
|
4695
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
1 # This detector sends notification on IRC through an irker daemon |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
2 # (http://www.catb.org/esr/irker/) when issues are created or messages |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
3 # are added. |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
4 # |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
5 # Written by Ezio Melotti |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
6 # |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
7 # Requires a running irkerd daemon to work. See the irker documentation |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
8 # for more information about installing, configuring, and running irker. |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
9 # |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
10 # Add the IRC channel(s) that should receive notifications in |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
11 # detectors/config.ini as a comma-separated list, using this format: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
12 # |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
13 # [irker] |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
14 # channels = irc://chat.freenode.net/channelname |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
15 # |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
16 |
|
5376
64b05e24dbd8
Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents:
4695
diff
changeset
|
17 from __future__ import print_function |
|
4695
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
18 import re |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
19 import json |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
20 import socket |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
21 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
22 IRKER_HOST = 'localhost' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
23 IRKER_PORT = 6659 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
24 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
25 max_content = 120 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
26 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
27 TEMPLATE = ('%(green)s%(author)s%(reset)s ' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
28 '%(bluish)s#%(nodeid)s%(reset)s/%(title)s%(bold)s:%(bold)s ' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
29 '%(log)s %(url)s') |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
30 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
31 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
32 def sendmsg(msg): |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
33 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
34 try: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
35 sock.connect((IRKER_HOST, IRKER_PORT)) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
36 sock.sendall(msg + "\n") |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
37 finally: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
38 sock.close() |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
39 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
40 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
41 def notify_irker(db, cl, nodeid, oldvalues): |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
42 messages = set(cl.get(nodeid, 'messages')) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
43 if oldvalues: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
44 messages -= set(oldvalues.get('messages', ())) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
45 if not messages: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
46 return |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
47 messages = list(messages) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
48 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
49 if oldvalues: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
50 oldstatus = oldvalues['status'] |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
51 else: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
52 oldstatus = None |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
53 newstatus = db.issue.get(nodeid, 'status') |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
54 if oldstatus != newstatus: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
55 if oldvalues: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
56 status = db.status.get(newstatus, 'name') |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
57 else: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
58 status = 'new' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
59 log = '[' + status + '] ' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
60 else: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
61 log = '' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
62 for msg in messages: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
63 log += db.msg.get(msg, 'content') |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
64 if len(log) > max_content: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
65 log = log[:max_content-3] + '...' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
66 log = re.sub('\s+', ' ', log) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
67 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
68 # include irc colors |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
69 params = { |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
70 'bold': '\x02', |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
71 'green': '\x0303', |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
72 'blue': '\x0302', |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
73 'bluish': '\x0310', |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
74 'yellow': '\x0307', |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
75 'brown': '\x0305', |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
76 'reset': '\x0F' |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
77 } |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
78 # extend with values used in the template |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
79 params['author'] = db.user.get(db.getuid(), 'username') |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
80 params['nodeid'] = nodeid |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
81 params['title'] = db.issue.get(nodeid, 'title') |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
82 params['log'] = log |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
83 params['url'] = '%sissue%s' % (db.config.TRACKER_WEB, nodeid) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
84 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
85 # create the message and use the list of channels defined in |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
86 # detectors/config.ini |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
87 msg = json.dumps({ |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
88 'to': db.config.detectors.IRKER_CHANNELS.split(','), |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
89 'privmsg': TEMPLATE % params, |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
90 }) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
91 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
92 try: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
93 sendmsg(msg) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
94 except Exception as e: |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
95 # Ignore any errors in sending the irker; |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
96 # if the server is down, that's just bad luck |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
97 # XXX might want to do some logging here |
|
5376
64b05e24dbd8
Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents:
4695
diff
changeset
|
98 print('* Sending message to irker failed', str(e)) |
|
4695
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
99 |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
100 def init(db): |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
101 db.issue.react('create', notify_irker) |
|
132650f4700a
#2550782: Added a new irker detector to send notifications on IRC when an issue is created or messages are added.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
diff
changeset
|
102 db.issue.react('set', notify_irker) |
