Mercurial > p > roundup > code

annotate scripts/dump_dbm_sessions_db.py @ 7399:deb8e7e6d66d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Skip redis tests if unable to communicate with the server. If the redis module is in the test environment, the redis tests will not be skipped. If connecting to redis during testing fails with a ConnectionError because there is no redis server at localhost, or if it fails with an AuthenticationError, you would fail a slew of tests. This causes the tests to report as skipped if either of the two errors occurs. It is very inefficient as it fails in setup() for the tests, but at least it does report skipping the tests. Also documented how to pass the redis password to the tests in the test part of the install docs. Future note: running tests needs proper docs in development.txt (including database setup) and a link left to that doc in installation.txt.
author John Rouillard <rouilj@ieee.org>
date Wed, 24 May 2023 12:52:53 -0400
parents 1188bb423f92
children 9ba04f37896f
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
1 #! /usr/bin/env python3
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
2 """Usage: dump_dbm_sessions_db.py [filename]
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
3
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
4 Simple script to dump the otks and sessions dbm databases. Dumps
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
5 sessions db in current directory if no argument is given.
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
6
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
7 Dump format:
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
8
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
9 key: <timestamp> data
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
10
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
11 where <timestamp> is the human readable __timestamp decoded from the
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
12 data object. Data object is dumped in json format. With pretty print
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
13
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
14 key:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
15 <timestamp>
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
16 {
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
17 key: val,
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
18 ...
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
19 }
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
20
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
21 if data is not a python object, print will be key: data or
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
22 key:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
23 data
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
24
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
25 if pretty printed.
6577
61481d7bbb07 fix syntax error
John Rouillard <rouilj@ieee.org>
parents: 6571
diff changeset
26 """
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
27
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
28 import argparse, dbm, json, marshal, os, sys
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
29 from datetime import datetime
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
30
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
31 def indent(text, amount, ch=" "):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
32 """ Found at: https://stackoverflow.com/a/8348914
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
33 """
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
34 padding = amount * ch
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
35 return ''.join(padding+line for line in text.splitlines(True))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
36
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
37 def print_marshal(k):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
38 d = marshal.loads(db[k])
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
39 try:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
40 t = datetime.fromtimestamp(d['__timestamp'])
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
41 except (KeyError, TypeError):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
42 # TypeError raised if marshalled data is not a dict (list, tuple etc)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
43 t = "no_timestamp"
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
44 if args.pretty:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
45 print("%s:\n %s\n%s"%(k, t, indent(json.dumps(
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
46 d, sort_keys=True, indent=4), 4)))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
47 else:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
48 print("%s: %s %s"%(k, t, d))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
49
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
50 def print_raw(k):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
51 if args.pretty:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
52 print("%s:\n %s"%(k, db[k]))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
53 else:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
54 print("%s: %s"%(k, db[k]))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
55
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
56 parser = argparse.ArgumentParser(
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
57 description='Dump DBM files used by Roundup in storage order.')
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
58 parser.add_argument('-k', '--key', action="append",
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
59 help='dump the entry for a key, can be used multiple times.')
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
60 parser.add_argument('-K', '--keysonly', action='store_true',
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
61 help='print the database keys, sorted in byte order.')
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
62 parser.add_argument('-p', '--pretty', action='store_true',
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
63 help='pretty print the output rather than printing on one line.')
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
64 parser.add_argument('file', nargs='?',
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
65 help='file to be dumped ("sessions" if not provided)')
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
66 args = parser.parse_args()
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
67
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
68 if args.file:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
69 file = args.file
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
70 else:
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
71 file="sessions"
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
72
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
73 try:
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
74 db = dbm.open(file)
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
75 except Exception as e:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
76 print("Unable to open database for %s: %s"%(file, e))
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
77 try:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
78 os.stat(file)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
79 print(" perhaps file is invalid or was created with a different version of Python?")
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
80 except OSError:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
81 # the file does exist on disk.
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
82 pass
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
83 exit(1)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
84
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
85 if args.keysonly:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
86 for k in sorted(db.keys()):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
87 print("%s"%k)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
88 exit(0)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
89
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
90 if args.key:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
91 for k in args.key:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
92 try:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
93 print_marshal(k)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
94 except (ValueError):
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
95 print_raw(k)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
96 exit(0)
6571
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
97
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
98 k = db.firstkey()
cd408eb748dd Add small utility script for dumping dbm based databases.
John Rouillard <rouilj@ieee.org>
parents:
diff changeset
99 while k is not None:
6799
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
100 try:
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
101 print_marshal(k)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
102 except (ValueError): # ValueError marshal.loads failed
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
103 print_raw(k)
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
104
1188bb423f92 Script dumps dbm files without marshalled data like _ids
John Rouillard <rouilj@ieee.org>
parents: 6577
diff changeset
105 k = db.nextkey(k)
Roundup Issue Tracker: http://roundup-tracker.org/