Mercurial > p > roundup > code
annotate roundup/support.py @ 8180:d02ce1d14acd
feat: issue2551068 - Provide way to retrieve file/msg data via rest endpoint.
Use Allow header to change format of /binary_content endpoint. If
Allow header for endpoint is not application/json, it will be matched
against the mime type for the file. */*, text/* are supported and will
return the native mime type if present.
Changes:
move */* mime type from static dict of supported types. It was
hardcoded to return json only. Now it can return a matching
non-json mime type for the /binary_content endpoint.
Edited some errors to explicitly add */* mime type.
Cleanups to use ', ' separation in lists of valid mime types rather
than just space separated.
Remove ETag header when sending raw content. See issue 2551375 for
background.
Doc added to rest.txt.
Small format fix up (add dash) in CHANGES.txt.
Make passing an unset/None/False accept_mime_type to
format_dispatch_output a 500 error. This used to be the fallback
to produce a 406 error after all processing had happened. It
should no longer be possible to take that code path as all 406
errors (with valid accept_mime_types) are generated before
processing takes place.
Make format_dispatch_output handle output other than json/xml so it
can send back binary_content data.
Removed a spurious client.response_code = 400 that seems to not be
used.
Tests added for all code paths.
Database setup for tests msg and file entry. This required a file
upload test to change so it doesn't look for file1 as the link
returned by the upload. Download the link and verify the data
rather than verifying the link.
Multiple formatting changes to error messages to make all lists of
valid mime types ', ' an not just space separated.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Sun, 08 Dec 2024 17:22:33 -0500 |
| parents | 8ef97f7cfb6d |
| children | 224ccb8b49ca |
| rev | line source |
|---|---|
| 2984 | 1 """Implements various support classes and functions used in a number of |
| 2 places in Roundup code. | |
| 3 """ | |
| 4 | |
|
5376
64b05e24dbd8
Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5043
diff
changeset
|
5 from __future__ import print_function |
| 2984 | 6 __docformat__ = 'restructuredtext' |
| 7 | |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
8 import os, time, sys |
|
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
9 |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2984
diff
changeset
|
10 |
| 2984 | 11 class TruthDict: |
| 12 '''Returns True for valid keys, False for others. | |
| 13 ''' | |
| 14 def __init__(self, keys): | |
| 15 if keys: | |
| 16 self.keys = {} | |
| 17 for col in keys: | |
| 18 self.keys[col] = 1 | |
| 19 | |
| 20 def __getitem__(self, name): | |
|
5442
afd9fd3a0edb
Python 3 preparation: avoid assigning to instance __getitem__ in TruthDict.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5400
diff
changeset
|
21 if hasattr(self, 'keys'): |
|
afd9fd3a0edb
Python 3 preparation: avoid assigning to instance __getitem__ in TruthDict.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5400
diff
changeset
|
22 return name in self.keys |
|
afd9fd3a0edb
Python 3 preparation: avoid assigning to instance __getitem__ in TruthDict.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5400
diff
changeset
|
23 else: |
|
afd9fd3a0edb
Python 3 preparation: avoid assigning to instance __getitem__ in TruthDict.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5400
diff
changeset
|
24 return True |
| 2984 | 25 |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
26 |
|
3019
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2984
diff
changeset
|
27 def ensureParentsExist(dest): |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2984
diff
changeset
|
28 if not os.path.exists(os.path.dirname(dest)): |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2984
diff
changeset
|
29 os.makedirs(os.path.dirname(dest)) |
|
293a17149765
First cut at exporting/importing file content from and to the hyperdb.
Johannes Gijsbers <jlgijsbers@users.sourceforge.net>
parents:
2984
diff
changeset
|
30 |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
31 |
|
3487
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
32 class PrioList: |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
33 '''Manages a sorted list. |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
34 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
35 Currently only implements method 'append' and iteration from a |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
36 full list interface. |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
37 Implementation: We manage a "sorted" status and sort on demand. |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
38 Appending to the list will require re-sorting before use. |
|
3635
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
39 >>> p = PrioList() |
|
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
40 >>> for i in 5,7,1,-1: |
|
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
41 ... p.append(i) |
|
3487
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
42 ... |
|
3635
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
43 >>> for k in p: |
|
7114
33eb82ad26ba
issue2551250: Fix sorting of detectors
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6018
diff
changeset
|
44 ... print (k) |
|
3487
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
45 ... |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
46 -1 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
47 1 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
48 5 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
49 7 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
50 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
51 ''' |
|
7114
33eb82ad26ba
issue2551250: Fix sorting of detectors
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6018
diff
changeset
|
52 def __init__(self, key=None): |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
53 self.list = [] |
|
7114
33eb82ad26ba
issue2551250: Fix sorting of detectors
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6018
diff
changeset
|
54 self.key = key |
|
3487
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
55 self.sorted = True |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
56 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
57 def append(self, item): |
|
3635
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
58 self.list.append(item) |
|
3487
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
59 self.sorted = False |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
60 |
|
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
61 def __iter__(self): |
|
3635
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
62 if not self.sorted: |
|
7114
33eb82ad26ba
issue2551250: Fix sorting of detectors
Ralf Schlatterbeck <rsc@runtux.com>
parents:
6018
diff
changeset
|
63 self.list.sort(key=self.key) |
|
3487
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
64 self.sorted = True |
|
3635
53987aa153d2
Transitive-property support.
Ralf Schlatterbeck <schlatterbeck@users.sourceforge.net>
parents:
3634
diff
changeset
|
65 return iter(self.list) |
|
3487
a2ae11191968
added class PrioList (patch from rfe [SF#413165])
Alexander Smishlajev <a1s@users.sourceforge.net>
parents:
3019
diff
changeset
|
66 |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
67 |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
68 class Progress: |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
69 '''Progress display for console applications. |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
70 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
71 ''' |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
72 def __init__(self, info, sequence): |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
73 self.info = info |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
74 self.sequence = iter(sequence) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
75 self.total = len(sequence) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
76 self.start = self.now = time.time() |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
77 self.num = 0 |
|
5400
2120f77554d5
Python 3 preparation: use // and __truediv__ as needed.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5396
diff
changeset
|
78 self.stepsize = self.total // 100 or 1 |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
79 self.steptimes = [] |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
80 self.display() |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
81 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
82 def __iter__(self): return self |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
83 |
|
5396
831787cf6694
Python 3 preparation: update next() usage for iterators.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5381
diff
changeset
|
84 def __next__(self): |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
85 self.num += 1 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
86 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
87 if self.num > self.total: |
|
5376
64b05e24dbd8
Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5043
diff
changeset
|
88 print(self.info, 'done', ' '*(75-len(self.info)-6)) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
89 sys.stdout.flush() |
|
5396
831787cf6694
Python 3 preparation: update next() usage for iterators.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5381
diff
changeset
|
90 return next(self.sequence) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
91 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
92 if self.num % self.stepsize: |
|
5396
831787cf6694
Python 3 preparation: update next() usage for iterators.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5381
diff
changeset
|
93 return next(self.sequence) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
94 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
95 self.display() |
|
5396
831787cf6694
Python 3 preparation: update next() usage for iterators.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5381
diff
changeset
|
96 return next(self.sequence) |
|
831787cf6694
Python 3 preparation: update next() usage for iterators.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5381
diff
changeset
|
97 # Python 2 compatibility: |
|
831787cf6694
Python 3 preparation: update next() usage for iterators.
Joseph Myers <jsm@polyomino.org.uk>
parents:
5381
diff
changeset
|
98 next = __next__ |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
99 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
100 def display(self): |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
101 # figure how long we've spent - guess how long to go |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
102 now = time.time() |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
103 steptime = now - self.now |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
104 self.steptimes.insert(0, steptime) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
105 if len(self.steptimes) > 5: |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
106 self.steptimes.pop() |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
107 steptime = sum(self.steptimes) / len(self.steptimes) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
108 self.now = now |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
109 eta = steptime * ((self.total - self.num)/self.stepsize) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
110 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
111 # tell it like it is (or might be) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
112 if now - self.start > 3: |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
113 M = eta / 60 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
114 H = M / 60 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
115 M = M % 60 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
116 S = eta % 60 |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
117 if self.total: |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
118 s = '%s %2d%% (ETA %02d:%02d:%02d)' % (self.info, |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
119 self.num * 100. / self.total, H, M, S) |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
120 else: |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
121 s = '%s 0%% (ETA %02d:%02d:%02d)' % (self.info, H, M, S) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
122 elif self.total: |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
123 s = '%s %2d%%' % (self.info, self.num * 100. / self.total) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
124 else: |
|
6018
01643d37785f
flake8 cleanup: remove unused re import; whitespace normalization.
John Rouillard <rouilj@ieee.org>
parents:
5442
diff
changeset
|
125 s = '%s %d done' % (self.info, self.num) |
|
3544
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
126 sys.stdout.write(s + ' '*(75-len(s)) + '\r') |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
127 sys.stdout.flush() |
|
5cd1c83dea50
Features and fixes.
Richard Jones <richard@users.sourceforge.net>
parents:
3487
diff
changeset
|
128 |
| 2984 | 129 # vim: set et sts=4 sw=4 : |
