Mercurial > p > roundup > code

annotate scripts/stats.xmlrpc.py @ 7719:3071db43bfb6

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
feat: issue2550852 - support using a specified PostgreSQL db schema Finally after 7 years this is closed. roundup/backends/back_postgresql.py: Support use of schema when specified in RDBMS_NAME. Stuart McGraws code is finally merged 8-). test/test_postgresql.py, test/conftest.py: Run all postgresql tests in the schema db as well. Also make sure that db_nuke raises an error when trying to delete the schema test database. Conftest defines pg_schema mark that can be used to exclude schema tests with pytest -m "not pg_schema". roundup/configuration.py: change doc on RDBMS_NAME to include db.schema form. .travis.yml, .github/workflows/ci-test.yml: create schema test db; add user for testing with schema; grant new user create privs for schema. doc/installation.txt: Reference to roundup-admin init deleting schema added. doc/mysql.txt doc/postgresql.txt: New documentation on psql/mysql commands to set up a production db. doc/upgrading.txt: mention schema support, also document service setting for selecting connection from pg_service.conf. doc/reference.txt: update config.ini documentation for RDBMS_NAME.
author John Rouillard <rouilj@ieee.org>
date Wed, 27 Dec 2023 22:52:14 -0500
parents 75da037d1c54
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6553
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
1 """Count how many issues use each bpo field and print a report."""
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
2 """ sample output: https://github.com/psf/gh-migration/issues/5#issuecomment-935697646"""
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
3
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
4 import xmlrpc.client
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
5
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
6 from collections import defaultdict
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
7
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
8 class SpecialTransport(xmlrpc.client.SafeTransport):
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
9 def send_content(self, connection, request_body):
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
10 connection.putheader("Referer", "https://bugs.python.org/")
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
11 connection.putheader("Origin", "https://bugs.python.org")
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
12 connection.putheader("X-Requested-With", "XMLHttpRequest")
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
13 xmlrpc.client.SafeTransport.send_content(self, connection, request_body)
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
14
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
15 # connect to bpo
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
16 roundup = xmlrpc.client.ServerProxy('https://bugs.python.org/xmlrpc',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
17 transport=SpecialTransport(),
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
18 allow_none=True)
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
19
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
20 # map bpo classes -> propname
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
21 # the class is the name of the class (e.g. issue_type, keyword --
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
22 # also used in e.g. in https://bugs.python.org/keyword)
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
23 # the propname is the name used as attribute on the issue class
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
24 # (e.g. issue.type, issue.keywords)
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
25 classes = {
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
26 # 'status': 'status', # skip this
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
27 'issue_type': 'type',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
28 'stage': 'stage',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
29 'component': 'components',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
30 'version': 'versions',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
31 'resolution': 'resolution',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
32 'priority': 'priority',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
33 'keyword': 'keywords',
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
34 }
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
35
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
36 # find the id for the 'open' status
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
37 open_id = roundup.lookup('status', 'open')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
38
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
39 print(f'* Counting total issues...')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
40 total_issues_num = len(roundup.filter('issue', None, {}))
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
41
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
42 print(f'* Counting open issues...')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
43 # use this list to filter only the open issues
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
44 open_issues = roundup.filter('issue', None, {'status': open_id})
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
45 open_issues_num = len(open_issues)
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
46
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
47 # save the totals in a dict with this structure:
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
48 # totals[propname][open/all][num/perc][name]
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
49 # where propname is e.g. 'keyword' and name is e.g. 'easy'
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
50 totals = defaultdict(lambda: {'all': {'perc': {}, 'num': {}},
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
51 'open': {'perc': {}, 'num': {}}})
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
52 for cls, propname in classes.items():
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
53 print(f'* Counting <{cls}>...')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
54 # get the list of ids/names for the given class (e.g. 'easy' is 6)
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
55 ids = roundup.list(cls, 'id')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
56 names = roundup.list(cls, 'name')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
57 for id, name in zip(ids, names):
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
58 # filter and count on *all* issues with the given propname
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
59 tot_all = len(roundup.filter('issue', None, {propname: id}))
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
60 totals[propname]['all']['num'][name] = tot_all
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
61 totals[propname]['all']['perc'][name] = tot_all / total_issues_num
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
62 # filter and count on *open* issues with the given propname
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
63 tot_open = len(roundup.filter('issue', open_issues, {propname: id}))
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
64 totals[propname]['open']['num'][name] = tot_open
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
65 totals[propname]['open']['perc'][name] = tot_open / open_issues_num
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
66
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
67
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
68 print(f'Issues (open/all): {open_issues_num}/{total_issues_num}')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
69
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
70 # print a list of markdown tables for each bpo class name
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
71 for propname in classes.values():
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
72 print(f'### {propname}')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
73 print('| bpo field | open | all |')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
74 print('| :--- | ---: | ---: |')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
75 # pick the dict for the given propname (e.g. keywords)
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
76 proptots = totals[propname]
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
77 names = proptots['open']['num']
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
78 # sort the names (e.g. 'easy') in reverse order
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
79 # based on the number of open issues
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
80 for name in sorted(names, key=names.get, reverse=True):
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
81 # get and print num/perc for all/open issues
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
82 issues_all = proptots['all']['num'][name]
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
83 issues_open = proptots['open']['num'][name]
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
84 perc_all = proptots['all']['perc'][name]
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
85 perc_open = proptots['open']['perc'][name]
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
86 print(f'| {name:20} | {issues_open:>5} ({perc_open:5.1%}) |'
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
87 f' {issues_all:>5} ({perc_all:5.1%}) |')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
88 # calc and print num/perc for all/open issues
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
89 tot_issues_all = sum(proptots['all']['num'].values())
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
90 tot_issues_open = sum(proptots['open']['num'].values())
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
91 tot_perc_all = sum(proptots['all']['perc'].values())
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
92 tot_perc_open = sum(proptots['open']['perc'].values())
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
93 print(f'| **Total** | {tot_issues_open:>5} ({tot_perc_open:5.1%}) |'
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
94 f' {tot_issues_all:>5} ({tot_perc_all:5.1%}) |')
75da037d1c54 Add another xmlrpc example script and update readme.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
95
Roundup Issue Tracker: http://roundup-tracker.org/