Mercurial > p > roundup > code
annotate roundup/support.py @ 6565:2c2dbfc332ba
Try to handle multiple connections better.
The session database is a hot spot. When multiple requests (e.g. 20)
come in at the same time session database contention can get great.
The original code didn't retry session database access when the open
failed. This resulted in errors at the client.
The second pass delayed 0.01 seconds and retried. It was better but we
still had multiple second stalls. I think the first request got in,
everybody else backed up and then retried at the same time. Again they
stepped on each other. With logging I would see many counters go all
the way to low single digits or to -1 indicating falure.
This pass uses randomint to generate delays from 0-.125 seconds in 5ms
increments. This performs better in testing. I rarely saw a counter
less than 13 (2 failed retries). Current logging starts after 6
failures and counts down until success or failure.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Thu, 16 Dec 2021 20:02:00 -0500 |
| parents | 01643d37785f |
| children | 33eb82ad26ba |
| rev | line source |
|---|---|
| 2984 | 1 """Implements various support classes and functions used in a number of |
| 2 places in Roundup code. | |
| 3 """ | |
| 4 | |
|
5376
64b05e24dbd8
Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5043
diff
changeset
|
5 from __future__ import print_function |
| 2984 | 6 __docformat__ = 'restructuredtext' |
| 7 | |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
8 import os, time, sys |
|
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
9 |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2984
diff
changeset
|
10 |
| 2984 | 11 class TruthDict: |
| 12 '''Returns True for valid keys, False for others. | |
| 13 ''' | |
| 14 def __init__(self, keys): | |
| 15 if keys: | |
| 16 self.keys = {} | |
| 17 for col in keys: | |
| 18 self.keys[col] = 1 | |
| 19 | |
| 20 def __getitem__(self, name): | |
|
5442
afd9fd3a0edb
Python 3 preparation: avoid assigning to instance __getitem__ in TruthDict.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5400
diff
changeset
|
21 if hasattr(self, 'keys'): |
|
afd9fd3a0edb
Python 3 preparation: avoid assigning to instance __getitem__ in TruthDict.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5400
diff
changeset
|
22 return name in self.keys |
|
afd9fd3a0edb
Python 3 preparation: avoid assigning to instance __getitem__ in TruthDict.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5400
diff
changeset
|
23 else: |
|
afd9fd3a0edb
Python 3 preparation: avoid assigning to instance __getitem__ in TruthDict.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5400
diff
changeset
|
24 return True |
| 2984 | 25 |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
26 |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2984
diff
changeset
|
27 def ensureParentsExist(dest): |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2984
diff
changeset
|
28 if not os.path.exists(os.path.dirname(dest)): |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2984
diff
changeset
|
29 os.makedirs(os.path.dirname(dest)) |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2984
diff
changeset
|
30 |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
31 |
|
3487
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
32 class PrioList: |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
33 '''Manages a sorted list. |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
34 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
35 Currently only implements method 'append' and iteration from a |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
36 full list interface. |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
37 Implementation: We manage a "sorted" status and sort on demand. |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
38 Appending to the list will require re-sorting before use. |
|
3635
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
39 >>> p = PrioList() |
|
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
40 >>> for i in 5,7,1,-1: |
|
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
41 ... p.append(i) |
|
3487
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
42 ... |
|
3635
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
43 >>> for k in p: |
|
3487
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
44 ... print k |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
45 ... |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
46 -1 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
47 1 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
48 5 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
49 7 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
50 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
51 ''' |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
52 def __init__(self): |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
53 self.list = [] |
|
3487
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
54 self.sorted = True |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
55 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
56 def append(self, item): |
|
3635
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
57 self.list.append(item) |
|
3487
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
58 self.sorted = False |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
59 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
60 def __iter__(self): |
|
3635
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
61 if not self.sorted: |
|
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
62 self.list.sort() |
|
3487
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
63 self.sorted = True |
|
3635
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
64 return iter(self.list) |
|
3487
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
65 |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
66 |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
67 class Progress: |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
68 '''Progress display for console applications. |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
69 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
70 See __main__ block at end of file for sample usage. |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
71 ''' |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
72 def __init__(self, info, sequence): |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
73 self.info = info |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
74 self.sequence = iter(sequence) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
75 self.total = len(sequence) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
76 self.start = self.now = time.time() |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
77 self.num = 0 |
|
5400
2120f77554d5
Python 3 preparation: use // and __truediv__ as needed.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5396
diff
changeset
|
78 self.stepsize = self.total // 100 or 1 |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
79 self.steptimes = [] |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
80 self.display() |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
81 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
82 def __iter__(self): return self |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
83 |
|
5396
831787cf6694
Python 3 preparation: update next() usage for iterators.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5381
diff
changeset
|
84 def __next__(self): |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
85 self.num += 1 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
86 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
87 if self.num > self.total: |
|
5376
64b05e24dbd8
Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5043
diff
changeset
|
88 print(self.info, 'done', ' '*(75-len(self.info)-6)) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
89 sys.stdout.flush() |
|
5396
831787cf6694
Python 3 preparation: update next() usage for iterators.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5381
diff
changeset
|
90 return next(self.sequence) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
91 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
92 if self.num % self.stepsize: |
|
5396
831787cf6694
Python 3 preparation: update next() usage for iterators.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5381
diff
changeset
|
93 return next(self.sequence) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
94 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
95 self.display() |
|
5396
831787cf6694
Python 3 preparation: update next() usage for iterators.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5381
diff
changeset
|
96 return next(self.sequence) |
|
831787cf6694
Python 3 preparation: update next() usage for iterators.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5381
diff
changeset
|
97 # Python 2 compatibility: |
|
831787cf6694
Python 3 preparation: update next() usage for iterators.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5381
diff
changeset
|
98 next = __next__ |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
99 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
100 def display(self): |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
101 # figure how long we've spent - guess how long to go |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
102 now = time.time() |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
103 steptime = now - self.now |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
104 self.steptimes.insert(0, steptime) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
105 if len(self.steptimes) > 5: |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
106 self.steptimes.pop() |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
107 steptime = sum(self.steptimes) / len(self.steptimes) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
108 self.now = now |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
109 eta = steptime * ((self.total - self.num)/self.stepsize) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
110 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
111 # tell it like it is (or might be) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
112 if now - self.start > 3: |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
113 M = eta / 60 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
114 H = M / 60 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
115 M = M % 60 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
116 S = eta % 60 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
117 if self.total: |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
118 s = '%s %2d%% (ETA %02d:%02d:%02d)' % (self.info, |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
119 self.num * 100. / self.total, H, M, S) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
120 else: |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
121 s = '%s 0%% (ETA %02d:%02d:%02d)' % (self.info, H, M, S) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
122 elif self.total: |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
123 s = '%s %2d%%' % (self.info, self.num * 100. / self.total) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
124 else: |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
125 s = '%s %d done' % (self.info, self.num) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
126 sys.stdout.write(s + ' '*(75-len(s)) + '\r') |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
127 sys.stdout.flush() |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
128 |
| 2984 | 129 # vim: set et sts=4 sw=4 : |
