Mercurial > p > roundup > code

annotate scripts/stats.xmlrpc.py @ 6610:db3f0ba75b4a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Change checkpoint_data and restore_connection_on_error to subtransaction checkpoint_data and restore_connection_on_error used to commit() and rollback() the db connection. This causes additional I/O and load. Changed them to use 'SAVEPOINT name' and 'ROLLBACK TO name' to get a faster method for handling errors within a tranaction. One thing to note is that postgresql (unlike SQL std) doesn't overwrite an older savepoint with he same name. It keeps all savepoints but only rolls back to the newest one with a given name. This could be a resource issue. I left a commented out release statement in case somebody runs into an issue due to too many savepoints. I expect it to slow down the import but....
author John Rouillard <rouilj@ieee.org>
date Sat, 29 Jan 2022 11:29:36 -0500
parents 75da037d1c54
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6553
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
1 """Count how many issues use each bpo field and print a report."""
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
2 """ sample output: https://github.com/psf/gh-migration/issues/5#issuecomment-935697646"""
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
3
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
4 import xmlrpc.client
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
5
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
6 from collections import defaultdict
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
7
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
8 class SpecialTransport(xmlrpc.client.SafeTransport):
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
9 def send_content(self, connection, request_body):
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
10 connection.putheader("Referer", "https://bugs.python.org/")
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
11 connection.putheader("Origin", "https://bugs.python.org")
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
12 connection.putheader("X-Requested-With", "XMLHttpRequest")
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
13 xmlrpc.client.SafeTransport.send_content(self, connection, request_body)
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
14
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
15 # connect to bpo
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
16 roundup = xmlrpc.client.ServerProxy('https://bugs.python.org/xmlrpc',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
17 transport=SpecialTransport(),
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
18 allow_none=True)
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
19
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
20 # map bpo classes -> propname
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
21 # the class is the name of the class (e.g. issue_type, keyword --
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
22 # also used in e.g. in https://bugs.python.org/keyword)
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
23 # the propname is the name used as attribute on the issue class
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
24 # (e.g. issue.type, issue.keywords)
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
25 classes = {
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
26 # 'status': 'status', # skip this
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
27 'issue_type': 'type',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
28 'stage': 'stage',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
29 'component': 'components',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
30 'version': 'versions',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
31 'resolution': 'resolution',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
32 'priority': 'priority',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
33 'keyword': 'keywords',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
34 }
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
35
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
36 # find the id for the 'open' status
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
37 open_id = roundup.lookup('status', 'open')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
38
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
39 print(f'* Counting total issues...')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
40 total_issues_num = len(roundup.filter('issue', None, {}))
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
41
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
42 print(f'* Counting open issues...')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
43 # use this list to filter only the open issues
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
44 open_issues = roundup.filter('issue', None, {'status': open_id})
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
45 open_issues_num = len(open_issues)
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
46
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
47 # save the totals in a dict with this structure:
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
48 # totals[propname][open/all][num/perc][name]
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
49 # where propname is e.g. 'keyword' and name is e.g. 'easy'
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
50 totals = defaultdict(lambda: {'all': {'perc': {}, 'num': {}},
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
51 'open': {'perc': {}, 'num': {}}})
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
52 for cls, propname in classes.items():
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
53 print(f'* Counting <{cls}>...')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
54 # get the list of ids/names for the given class (e.g. 'easy' is 6)
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
55 ids = roundup.list(cls, 'id')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
56 names = roundup.list(cls, 'name')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
57 for id, name in zip(ids, names):
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
58 # filter and count on *all* issues with the given propname
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
59 tot_all = len(roundup.filter('issue', None, {propname: id}))
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
60 totals[propname]['all']['num'][name] = tot_all
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
61 totals[propname]['all']['perc'][name] = tot_all / total_issues_num
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
62 # filter and count on *open* issues with the given propname
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
63 tot_open = len(roundup.filter('issue', open_issues, {propname: id}))
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
64 totals[propname]['open']['num'][name] = tot_open
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
65 totals[propname]['open']['perc'][name] = tot_open / open_issues_num
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
66
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
67
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
68 print(f'Issues (open/all): {open_issues_num}/{total_issues_num}')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
69
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
70 # print a list of markdown tables for each bpo class name
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
71 for propname in classes.values():
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
72 print(f'### {propname}')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
73 print('| bpo field | open | all |')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
74 print('| :--- | ---: | ---: |')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
75 # pick the dict for the given propname (e.g. keywords)
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
76 proptots = totals[propname]
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
77 names = proptots['open']['num']
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
78 # sort the names (e.g. 'easy') in reverse order
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
79 # based on the number of open issues
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
80 for name in sorted(names, key=names.get, reverse=True):
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
81 # get and print num/perc for all/open issues
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
82 issues_all = proptots['all']['num'][name]
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
83 issues_open = proptots['open']['num'][name]
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
84 perc_all = proptots['all']['perc'][name]
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
85 perc_open = proptots['open']['perc'][name]
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
86 print(f'| {name:20} | {issues_open:>5} ({perc_open:5.1%}) |'
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
87 f' {issues_all:>5} ({perc_all:5.1%}) |')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
88 # calc and print num/perc for all/open issues
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
89 tot_issues_all = sum(proptots['all']['num'].values())
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
90 tot_issues_open = sum(proptots['open']['num'].values())
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
91 tot_perc_all = sum(proptots['all']['perc'].values())
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
92 tot_perc_open = sum(proptots['open']['perc'].values())
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
93 print(f'| **Total** | {tot_issues_open:>5} ({tot_perc_open:5.1%}) |'
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
94 f' {tot_issues_all:>5} ({tot_perc_all:5.1%}) |')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
95
Roundup Issue Tracker: http://roundup-tracker.org/