Mercurial > p > roundup > code

annotate scripts/dump_dbm_sessions_db.py @ 6799:1188bb423f92

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Script dumps dbm files without marshalled data like _ids Allow the _ids database to be dumped. The _ids db (when using the anydbm backend) records the number last used for each object type. E.G. if there were 8 statuses defined, it would have the entry: status: 8 Allow other dbm databases (like the node or journal dbs) to be dumped. Used argparse for command line options and added support for: dump all keys in sorted order dump specific key(s) pretty print output
author John Rouillard <rouilj@ieee.org>
date Sun, 24 Jul 2022 01:31:45 -0400
parents 61481d7bbb07
children 9ba04f37896f
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
1 #! /usr/bin/env python3
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
2 """Usage: dump_dbm_sessions_db.py [filename]
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
3
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
4 Simple script to dump the otks and sessions dbm databases. Dumps
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
5 sessions db in current directory if no argument is given.
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
6
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
7 Dump format:
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
8
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
9 key: <timestamp> data
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
10
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
11 where <timestamp> is the human readable __timestamp decoded from the
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
12 data object. Data object is dumped in json format. With pretty print
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
13
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
14 key:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
15 <timestamp>
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
16 {
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
17 key: val,
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
18 ...
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
19 }
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
20
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
21 if data is not a python object, print will be key: data or
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
22 key:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
23 data
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
24
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
25 if pretty printed.
6577
61481d7bbb07 fix syntax error
John Rouillard <rouilj@ieee.org>
parents: 6571
diff changeset
26 """
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
27
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
28 import argparse, dbm, json, marshal, os, sys
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
29 from datetime import datetime
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
30
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
31 def indent(text, amount, ch=" "):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
32 """ Found at: https://stackoverflow.com/a/8348914
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
33 """
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
34 padding = amount * ch
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
35 return ''.join(padding+line for line in text.splitlines(True))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
36
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
37 def print_marshal(k):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
38 d = marshal.loads(db[k])
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
39 try:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
40 t = datetime.fromtimestamp(d['__timestamp'])
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
41 except (KeyError, TypeError):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
42 # TypeError raised if marshalled data is not a dict (list, tuple etc)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
43 t = "no_timestamp"
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
44 if args.pretty:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
45 print("%s:\n %s\n%s"%(k, t, indent(json.dumps(
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
46 d, sort_keys=True, indent=4), 4)))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
47 else:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
48 print("%s: %s %s"%(k, t, d))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
49
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
50 def print_raw(k):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
51 if args.pretty:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
52 print("%s:\n %s"%(k, db[k]))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
53 else:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
54 print("%s: %s"%(k, db[k]))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
55
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
56 parser = argparse.ArgumentParser(
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
57 description='Dump DBM files used by Roundup in storage order.')
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
58 parser.add_argument('-k', '--key', action="append",
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
59 help='dump the entry for a key, can be used multiple times.')
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
60 parser.add_argument('-K', '--keysonly', action='store_true',
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
61 help='print the database keys, sorted in byte order.')
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
62 parser.add_argument('-p', '--pretty', action='store_true',
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
63 help='pretty print the output rather than printing on one line.')
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
64 parser.add_argument('file', nargs='?',
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
65 help='file to be dumped ("sessions" if not provided)')
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
66 args = parser.parse_args()
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
67
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
68 if args.file:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
69 file = args.file
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
70 else:
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
71 file="sessions"
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
72
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
73 try:
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
74 db = dbm.open(file)
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
75 except Exception as e:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
76 print("Unable to open database for %s: %s"%(file, e))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
77 try:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
78 os.stat(file)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
79 print(" perhaps file is invalid or was created with a different version of Python?")
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
80 except OSError:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
81 # the file does exist on disk.
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
82 pass
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
83 exit(1)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
84
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
85 if args.keysonly:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
86 for k in sorted(db.keys()):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
87 print("%s"%k)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
88 exit(0)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
89
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
90 if args.key:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
91 for k in args.key:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
92 try:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
93 print_marshal(k)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
94 except (ValueError):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
95 print_raw(k)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
96 exit(0)
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
97
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
98 k = db.firstkey()
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
99 while k is not None:
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
100 try:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
101 print_marshal(k)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
102 except (ValueError): # ValueError marshal.loads failed
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
103 print_raw(k)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
104
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
105 k = db.nextkey(k)
Roundup Issue Tracker: http://roundup-tracker.org/