Mercurial > p > roundup > code

annotate scripts/dump_dbm_sessions_db.py @ 8408:e882a5d52ae5

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
refactor: move RateLimitExceeded to roundup.cgi.exceptions RateLimitExceeded is an HTTP exception that raises code 429. Move it to roundup.cgi.exceptions where all the other exceptions that result in http status codes are located. Also make it inherit from HTTPException since it is one. Also add docstrings for all HTTP exceptions and order HTTPExceptions by status code. BREAKING CHANGE: if somebody is importing RateLimitExceeded they will need to change their import. I consider it unlikely anybody is using RateLimitExceeded. Detectors and extensions are unlikely to raise RateLimitExceeded. So I am leaving it out of the upgrading doc. Just doc in change log.
author John Rouillard <rouilj@ieee.org>
date Sun, 10 Aug 2025 21:27:06 -0400
parents 9ba04f37896f
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
1 #! /usr/bin/env python3
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
2 """Usage: dump_dbm_sessions_db.py [filename]
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
3
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
4 Simple script to dump the otks and sessions dbm databases. Dumps
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
5 sessions db in current directory if no argument is given.
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
6
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
7 Dump format:
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
8
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
9 key: <timestamp> data
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
10
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
11 where <timestamp> is the human readable __timestamp decoded from the
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
12 data object. Data object is dumped in json format. With pretty print
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
13
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
14 key:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
15 <timestamp>
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
16 {
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
17 key: val,
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
18 ...
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
19 }
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
20
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
21 if data is not a python object, print will be key: data or
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
22 key:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
23 data
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
24
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
25 if pretty printed.
6577
61481d7bbb07 fix syntax error
John Rouillard <rouilj@ieee.org>
parents: 6571
diff changeset
26 """
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
27
7780
9ba04f37896f chore: lint cleanups
John Rouillard <rouilj@ieee.org>
parents: 6799
diff changeset
28 import argparse
9ba04f37896f chore: lint cleanups
John Rouillard <rouilj@ieee.org>
parents: 6799
diff changeset
29 import dbm
9ba04f37896f chore: lint cleanups
John Rouillard <rouilj@ieee.org>
parents: 6799
diff changeset
30 import json
9ba04f37896f chore: lint cleanups
John Rouillard <rouilj@ieee.org>
parents: 6799
diff changeset
31 import marshal
9ba04f37896f chore: lint cleanups
John Rouillard <rouilj@ieee.org>
parents: 6799
diff changeset
32 import os
9ba04f37896f chore: lint cleanups
John Rouillard <rouilj@ieee.org>
parents: 6799
diff changeset
33 import sys
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
34 from datetime import datetime
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
35
7780
9ba04f37896f chore: lint cleanups
John Rouillard <rouilj@ieee.org>
parents: 6799
diff changeset
36
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
37 def indent(text, amount, ch=" "):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
38 """ Found at: https://stackoverflow.com/a/8348914
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
39 """
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
40 padding = amount * ch
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
41 return ''.join(padding+line for line in text.splitlines(True))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
42
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
43 def print_marshal(k):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
44 d = marshal.loads(db[k])
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
45 try:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
46 t = datetime.fromtimestamp(d['__timestamp'])
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
47 except (KeyError, TypeError):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
48 # TypeError raised if marshalled data is not a dict (list, tuple etc)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
49 t = "no_timestamp"
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
50 if args.pretty:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
51 print("%s:\n %s\n%s"%(k, t, indent(json.dumps(
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
52 d, sort_keys=True, indent=4), 4)))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
53 else:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
54 print("%s: %s %s"%(k, t, d))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
55
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
56 def print_raw(k):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
57 if args.pretty:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
58 print("%s:\n %s"%(k, db[k]))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
59 else:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
60 print("%s: %s"%(k, db[k]))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
61
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
62 parser = argparse.ArgumentParser(
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
63 description='Dump DBM files used by Roundup in storage order.')
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
64 parser.add_argument('-k', '--key', action="append",
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
65 help='dump the entry for a key, can be used multiple times.')
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
66 parser.add_argument('-K', '--keysonly', action='store_true',
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
67 help='print the database keys, sorted in byte order.')
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
68 parser.add_argument('-p', '--pretty', action='store_true',
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
69 help='pretty print the output rather than printing on one line.')
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
70 parser.add_argument('file', nargs='?',
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
71 help='file to be dumped ("sessions" if not provided)')
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
72 args = parser.parse_args()
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
73
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
74 if args.file:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
75 file = args.file
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
76 else:
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
77 file="sessions"
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
78
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
79 try:
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
80 db = dbm.open(file)
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
81 except Exception as e:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
82 print("Unable to open database for %s: %s"%(file, e))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
83 try:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
84 os.stat(file)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
85 print(" perhaps file is invalid or was created with a different version of Python?")
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
86 except OSError:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
87 # the file does exist on disk.
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
88 pass
7780
9ba04f37896f chore: lint cleanups
John Rouillard <rouilj@ieee.org>
parents: 6799
diff changeset
89 sys.exit(1)
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
90
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
91 if args.keysonly:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
92 for k in sorted(db.keys()):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
93 print("%s"%k)
7780
9ba04f37896f chore: lint cleanups
John Rouillard <rouilj@ieee.org>
parents: 6799
diff changeset
94 sys.exit(0)
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
95
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
96 if args.key:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
97 for k in args.key:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
98 try:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
99 print_marshal(k)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
100 except (ValueError):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
101 print_raw(k)
7780
9ba04f37896f chore: lint cleanups
John Rouillard <rouilj@ieee.org>
parents: 6799
diff changeset
102 sys.exit(0)
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
103
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
104 k = db.firstkey()
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
105 while k is not None:
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
106 try:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
107 print_marshal(k)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
108 except (ValueError): # ValueError marshal.loads failed
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
109 print_raw(k)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
110
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
111 k = db.nextkey(k)
Roundup Issue Tracker: http://roundup-tracker.org/