Mercurial > p > roundup > code
annotate scripts/stats.xmlrpc.py @ 7752:b2dbab2b34bc
fix(refactor): multiple fixups using ruff linter; more testing.
Converting to using the ruff linter and its rulesets. Fixed a number
of issues.
admin.py:
sort imports
use immutable tuples as default value markers for parameters where a
None value is valid.
reduced some loops to list comprehensions for performance
used ternary to simplify some if statements
named some variables to make them less magic
(e.g. _default_savepoint_setting = 1000)
fixed some tests for argument counts < 2 becomes != 2 so 3 is an
error.
moved exception handlers outside of loops for performance where
exception handler will abort loop anyway.
renamed variables called 'id' or 'dir' as they shadow builtin
commands.
fix translations of form _("string %s" % value) -> _("string %s") %
value so translation will be looked up with the key before
substitution.
end dicts, tuples with a trailing comma to reduce missing comma
errors if modified
simplified sorted(list(self.setting.keys())) to
sorted(self.setting.keys()) as sorted consumes whole list.
in if conditions put compared variable on left and threshold condition
on right. (no yoda conditions)
multiple noqa: suppression
removed unneeded noqa as lint rulesets are a bit different
do_get - refactor output printing logic: Use fast return if not
special formatting is requested; use isinstance with a tuple
rather than two isinstance calls; cleaned up flow and removed
comments on algorithm as it can be easily read from the code.
do_filter, do_find - refactor output printing logic. Reduce
duplicate code.
do_find - renamed variable 'value' that was set inside a loop. The
loop index variable was also named 'value'.
do_pragma - added hint to use list subcommand if setting was not
found. Replaced condition 'type(x) is bool' with 'isinstance(x,
bool)' for various types.
test_admin.py
added testing for do_list
better test coverage for do_get includes: -S and -d for multilinks,
error case for -d with non-link.
better testing for do_find including all output modes
better testing for do_filter including all output modes
fixed expected output for do_pragma that now includes hint to use
pragma list if setting not found.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Fri, 01 Mar 2024 14:53:18 -0500 |
| parents | 75da037d1c54 |
| children |
| rev | line source |
|---|---|
|
6553
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
1 """Count how many issues use each bpo field and print a report.""" |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
2 """ sample output: https://github.com/psf/gh-migration/issues/5#issuecomment-935697646""" |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
3 |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
4 import xmlrpc.client |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
5 |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
6 from collections import defaultdict |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
7 |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
8 class SpecialTransport(xmlrpc.client.SafeTransport): |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
9 def send_content(self, connection, request_body): |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
10 connection.putheader("Referer", "https://bugs.python.org/") |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
11 connection.putheader("Origin", "https://bugs.python.org") |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
12 connection.putheader("X-Requested-With", "XMLHttpRequest") |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
13 xmlrpc.client.SafeTransport.send_content(self, connection, request_body) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
14 |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
15 # connect to bpo |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
16 roundup = xmlrpc.client.ServerProxy('https://bugs.python.org/xmlrpc', |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
17 transport=SpecialTransport(), |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
18 allow_none=True) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
19 |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
20 # map bpo classes -> propname |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
21 # the class is the name of the class (e.g. issue_type, keyword -- |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
22 # also used in e.g. in https://bugs.python.org/keyword) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
23 # the propname is the name used as attribute on the issue class |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
24 # (e.g. issue.type, issue.keywords) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
25 classes = { |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
26 # 'status': 'status', # skip this |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
27 'issue_type': 'type', |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
28 'stage': 'stage', |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
29 'component': 'components', |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
30 'version': 'versions', |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
31 'resolution': 'resolution', |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
32 'priority': 'priority', |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
33 'keyword': 'keywords', |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
34 } |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
35 |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
36 # find the id for the 'open' status |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
37 open_id = roundup.lookup('status', 'open') |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
38 |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
39 print(f'* Counting total issues...') |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
40 total_issues_num = len(roundup.filter('issue', None, {})) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
41 |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
42 print(f'* Counting open issues...') |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
43 # use this list to filter only the open issues |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
44 open_issues = roundup.filter('issue', None, {'status': open_id}) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
45 open_issues_num = len(open_issues) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
46 |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
47 # save the totals in a dict with this structure: |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
48 # totals[propname][open/all][num/perc][name] |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
49 # where propname is e.g. 'keyword' and name is e.g. 'easy' |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
50 totals = defaultdict(lambda: {'all': {'perc': {}, 'num': {}}, |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
51 'open': {'perc': {}, 'num': {}}}) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
52 for cls, propname in classes.items(): |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
53 print(f'* Counting <{cls}>...') |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
54 # get the list of ids/names for the given class (e.g. 'easy' is 6) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
55 ids = roundup.list(cls, 'id') |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
56 names = roundup.list(cls, 'name') |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
57 for id, name in zip(ids, names): |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
58 # filter and count on *all* issues with the given propname |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
59 tot_all = len(roundup.filter('issue', None, {propname: id})) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
60 totals[propname]['all']['num'][name] = tot_all |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
61 totals[propname]['all']['perc'][name] = tot_all / total_issues_num |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
62 # filter and count on *open* issues with the given propname |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
63 tot_open = len(roundup.filter('issue', open_issues, {propname: id})) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
64 totals[propname]['open']['num'][name] = tot_open |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
65 totals[propname]['open']['perc'][name] = tot_open / open_issues_num |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
66 |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
67 |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
68 print(f'Issues (open/all): {open_issues_num}/{total_issues_num}') |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
69 |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
70 # print a list of markdown tables for each bpo class name |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
71 for propname in classes.values(): |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
72 print(f'### {propname}') |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
73 print('| bpo field | open | all |') |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
74 print('| :--- | ---: | ---: |') |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
75 # pick the dict for the given propname (e.g. keywords) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
76 proptots = totals[propname] |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
77 names = proptots['open']['num'] |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
78 # sort the names (e.g. 'easy') in reverse order |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
79 # based on the number of open issues |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
80 for name in sorted(names, key=names.get, reverse=True): |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
81 # get and print num/perc for all/open issues |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
82 issues_all = proptots['all']['num'][name] |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
83 issues_open = proptots['open']['num'][name] |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
84 perc_all = proptots['all']['perc'][name] |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
85 perc_open = proptots['open']['perc'][name] |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
86 print(f'| {name:20} | {issues_open:>5} ({perc_open:5.1%}) |' |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
87 f' {issues_all:>5} ({perc_all:5.1%}) |') |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
88 # calc and print num/perc for all/open issues |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
89 tot_issues_all = sum(proptots['all']['num'].values()) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
90 tot_issues_open = sum(proptots['open']['num'].values()) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
91 tot_perc_all = sum(proptots['all']['perc'].values()) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
92 tot_perc_open = sum(proptots['open']['perc'].values()) |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
93 print(f'| **Total** | {tot_issues_open:>5} ({tot_perc_open:5.1%}) |' |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
94 f' {tot_issues_all:>5} ({tot_perc_all:5.1%}) |') |
|
75da037d1c54
Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
95 |
