Mercurial > p > roundup > code
annotate scripts/dump_dbm_sessions_db.py @ 7752:b2dbab2b34bc
fix(refactor): multiple fixups using ruff linter; more testing.
Converting to using the ruff linter and its rulesets. Fixed a number
of issues.
admin.py:
sort imports
use immutable tuples as default value markers for parameters where a
None value is valid.
reduced some loops to list comprehensions for performance
used ternary to simplify some if statements
named some variables to make them less magic
(e.g. _default_savepoint_setting = 1000)
fixed some tests for argument counts < 2 becomes != 2 so 3 is an
error.
moved exception handlers outside of loops for performance where
exception handler will abort loop anyway.
renamed variables called 'id' or 'dir' as they shadow builtin
commands.
fix translations of form _("string %s" % value) -> _("string %s") %
value so translation will be looked up with the key before
substitution.
end dicts, tuples with a trailing comma to reduce missing comma
errors if modified
simplified sorted(list(self.setting.keys())) to
sorted(self.setting.keys()) as sorted consumes whole list.
in if conditions put compared variable on left and threshold condition
on right. (no yoda conditions)
multiple noqa: suppression
removed unneeded noqa as lint rulesets are a bit different
do_get - refactor output printing logic: Use fast return if not
special formatting is requested; use isinstance with a tuple
rather than two isinstance calls; cleaned up flow and removed
comments on algorithm as it can be easily read from the code.
do_filter, do_find - refactor output printing logic. Reduce
duplicate code.
do_find - renamed variable 'value' that was set inside a loop. The
loop index variable was also named 'value'.
do_pragma - added hint to use list subcommand if setting was not
found. Replaced condition 'type(x) is bool' with 'isinstance(x,
bool)' for various types.
test_admin.py
added testing for do_list
better test coverage for do_get includes: -S and -d for multilinks,
error case for -d with non-link.
better testing for do_find including all output modes
better testing for do_filter including all output modes
fixed expected output for do_pragma that now includes hint to use
pragma list if setting not found.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Fri, 01 Mar 2024 14:53:18 -0500 |
| parents | 1188bb423f92 |
| children | 9ba04f37896f |
| rev | line source |
|---|---|
|
6571
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
1 #! /usr/bin/env python3 |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
2 """Usage: dump_dbm_sessions_db.py [filename] |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
3 |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
4 Simple script to dump the otks and sessions dbm databases. Dumps |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
5 sessions db in current directory if no argument is given. |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
6 |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
7 Dump format: |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
8 |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
9 key: <timestamp> data |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
10 |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
11 where <timestamp> is the human readable __timestamp decoded from the |
|
6799
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
12 data object. Data object is dumped in json format. With pretty print |
|
6571
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
13 |
|
6799
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
14 key: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
15 <timestamp> |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
16 { |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
17 key: val, |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
18 ... |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
19 } |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
20 |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
21 if data is not a python object, print will be key: data or |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
22 key: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
23 data |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
24 |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
25 if pretty printed. |
| 6577 | 26 """ |
|
6571
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
27 |
|
6799
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
28 import argparse, dbm, json, marshal, os, sys |
|
6571
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
29 from datetime import datetime |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
30 |
|
6799
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
31 def indent(text, amount, ch=" "): |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
32 """ Found at: https://stackoverflow.com/a/8348914 |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
33 """ |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
34 padding = amount * ch |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
35 return ''.join(padding+line for line in text.splitlines(True)) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
36 |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
37 def print_marshal(k): |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
38 d = marshal.loads(db[k]) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
39 try: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
40 t = datetime.fromtimestamp(d['__timestamp']) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
41 except (KeyError, TypeError): |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
42 # TypeError raised if marshalled data is not a dict (list, tuple etc) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
43 t = "no_timestamp" |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
44 if args.pretty: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
45 print("%s:\n %s\n%s"%(k, t, indent(json.dumps( |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
46 d, sort_keys=True, indent=4), 4))) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
47 else: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
48 print("%s: %s %s"%(k, t, d)) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
49 |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
50 def print_raw(k): |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
51 if args.pretty: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
52 print("%s:\n %s"%(k, db[k])) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
53 else: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
54 print("%s: %s"%(k, db[k])) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
55 |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
56 parser = argparse.ArgumentParser( |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
57 description='Dump DBM files used by Roundup in storage order.') |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
58 parser.add_argument('-k', '--key', action="append", |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
59 help='dump the entry for a key, can be used multiple times.') |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
60 parser.add_argument('-K', '--keysonly', action='store_true', |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
61 help='print the database keys, sorted in byte order.') |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
62 parser.add_argument('-p', '--pretty', action='store_true', |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
63 help='pretty print the output rather than printing on one line.') |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
64 parser.add_argument('file', nargs='?', |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
65 help='file to be dumped ("sessions" if not provided)') |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
66 args = parser.parse_args() |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
67 |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
68 if args.file: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
69 file = args.file |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
70 else: |
|
6571
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
71 file="sessions" |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
72 |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
73 try: |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
74 db = dbm.open(file) |
|
6799
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
75 except Exception as e: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
76 print("Unable to open database for %s: %s"%(file, e)) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
77 try: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
78 os.stat(file) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
79 print(" perhaps file is invalid or was created with a different version of Python?") |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
80 except OSError: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
81 # the file does exist on disk. |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
82 pass |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
83 exit(1) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
84 |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
85 if args.keysonly: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
86 for k in sorted(db.keys()): |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
87 print("%s"%k) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
88 exit(0) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
89 |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
90 if args.key: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
91 for k in args.key: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
92 try: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
93 print_marshal(k) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
94 except (ValueError): |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
95 print_raw(k) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
96 exit(0) |
|
6571
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
97 |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
98 k = db.firstkey() |
|
cd408eb748dd
Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff
changeset
|
99 while k is not None: |
|
6799
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
100 try: |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
101 print_marshal(k) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
102 except (ValueError): # ValueError marshal.loads failed |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
103 print_raw(k) |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
104 |
|
1188bb423f92
Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents:
6577
diff
changeset
|
105 k = db.nextkey(k) |
