Mercurial > p > roundup > code

annotate scripts/import_sf.py @ 3542:a2fc38fd0142

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
fixes and speedup
author Richard Jones <richard@users.sourceforge.net>
date Mon, 06 Feb 2006 00:26:13 +0000
parents 07072b1b796c
children 0735c4587852
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
3536
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
1 ''' Import tracker data from Sourceforge.NET
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
2
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
3 This script needs four steps to work:
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
4
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
5 1. Export the project XML data using the admin web interface at sf.net
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
6 2. Run the file fetching (these are not included in the XML):
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
7
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
8 import_sf.py files <path to XML> <path to files dir>
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
9
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
10 this will place all the downloaded files in the files dir by file id.
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
11 3. Convert the sf.net XML to Roundup "export" format:
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
12
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
13 import_sf.py import <tracker home> <path to XML> <path to files dir>
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
14
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
15 this will generate a directory "/tmp/imported" which contains the
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
16 data to be imported into a Roundup tracker.
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
17 4. Import the data:
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
18
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
19 roundup-admin -i <tracker home> import /tmp/imported
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
20
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
21 And you're done!
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
22 '''
00a7983b4666 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3534
diff changeset
23
3537
d819ff1b3116 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3536
diff changeset
24 import sys, sets, os, csv, time, urllib2, httplib, mimetypes, urlparse
3542
a2fc38fd0142 fixes and speedup
Richard Jones <richard@users.sourceforge.net>
parents: 3541
diff changeset
25
a2fc38fd0142 fixes and speedup
Richard Jones <richard@users.sourceforge.net>
parents: 3541
diff changeset
26 try:
a2fc38fd0142 fixes and speedup
Richard Jones <richard@users.sourceforge.net>
parents: 3541
diff changeset
27 import cElementTree as ElementTree
a2fc38fd0142 fixes and speedup
Richard Jones <richard@users.sourceforge.net>
parents: 3541
diff changeset
28 except ImportError:
a2fc38fd0142 fixes and speedup
Richard Jones <richard@users.sourceforge.net>
parents: 3541
diff changeset
29 from elementtree import ElementTree
3534
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
30
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
31 from roundup import instance, hyperdb, date, support, password
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
32
3538
7f7902af7864 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3537
diff changeset
33 today = date.Date('.')
7f7902af7864 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3537
diff changeset
34
3537
d819ff1b3116 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3536
diff changeset
35 DL_URL = 'http://sourceforge.net/tracker/download.php?group_id=%(group_id)s&atid=%(atid)s&aid=%(aid)s'
d819ff1b3116 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3536
diff changeset
36
3534
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
37 def get_url(aid):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
38 """ so basically we have to jump through hoops, given an artifact id, to
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
39 figure what the URL should be to access that artifact, and hence any
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
40 attached files."""
3538
7f7902af7864 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3537
diff changeset
41 # first we hit this URL...
3534
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
42 conn = httplib.HTTPConnection("sourceforge.net")
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
43 conn.request("GET", "/support/tracker.php?aid=%s"%aid)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
44 response = conn.getresponse()
3538
7f7902af7864 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3537
diff changeset
45 # which should respond with a redirect to the correct url which has the
7f7902af7864 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3537
diff changeset
46 # magic "group_id" and "atid" values in it that we need
3534
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
47 assert response.status == 302, 'response code was %s'%response.status
3537
d819ff1b3116 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3536
diff changeset
48 location = response.getheader('location')
d819ff1b3116 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3536
diff changeset
49 query = urlparse.urlparse(response.getheader('location'))[-2]
d819ff1b3116 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3536
diff changeset
50 info = dict([param.split('=') for param in query.split('&')])
3541
07072b1b796c fix typo
Richard Jones <richard@users.sourceforge.net>
parents: 3538
diff changeset
51 return DL_URL%info
3534
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
52
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
53 def fetch_files(xml_file, file_dir):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
54 """ Fetch files referenced in the xml_file into the dir file_dir. """
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
55 root = ElementTree.parse(xml_file).getroot()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
56 to_fetch = sets.Set()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
57 deleted = sets.Set()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
58 for artifact in root.find('artifacts'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
59 for field in artifact.findall('field'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
60 if field.get('name') == 'artifact_id':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
61 aid = field.text
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
62 for field in artifact.findall('field'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
63 if field.get('name') != 'artifact_history': continue
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
64 for event in field.findall('history'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
65 d = {}
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
66 for field in event.findall('field'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
67 d[field.get('name')] = field.text
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
68 if d['field_name'] == 'File Added':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
69 fid = d['old_value'].split(':')[0]
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
70 to_fetch.add((aid, fid))
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
71 if d['field_name'] == 'File Deleted':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
72 fid = d['old_value'].split(':')[0]
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
73 deleted.add((aid, fid))
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
74 to_fetch = to_fetch - deleted
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
75
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
76 got = sets.Set(os.listdir(file_dir))
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
77 to_fetch = to_fetch - got
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
78
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
79 # load cached urls (sigh)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
80 urls = {}
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
81 if os.path.exists(os.path.join(file_dir, 'urls.txt')):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
82 for line in open(os.path.join(file_dir, 'urls.txt')):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
83 aid, url = line.strip().split()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
84 urls[aid] = url
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
85
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
86 for aid, fid in Progress('Fetching files', list(to_fetch)):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
87 if fid in got: continue
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
88 if not urls.has_key(aid):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
89 urls[aid] = get_url(aid)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
90 f = open(os.path.join(file_dir, 'urls.txt'), 'a')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
91 f.write('%s %s\n'%(aid, urls[aid]))
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
92 f.close()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
93 url = urls[aid] + '&file_id=' + fid
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
94 f = urllib2.urlopen(url)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
95 data = f.read()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
96 n = open(os.path.join(file_dir, fid), 'w')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
97 n.write(data)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
98 f.close()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
99 n.close()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
100
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
101 def import_xml(tracker_home, xml_file, file_dir):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
102 """ Generate Roundup tracker import files based on the tracker schema,
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
103 sf.net xml export and downloaded files from sf.net. """
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
104 tracker = instance.open(tracker_home)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
105 db = tracker.open('admin')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
106
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
107 resolved = db.status.lookup('resolved')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
108 unread = db.status.lookup('unread')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
109 chatting = db.status.lookup('unread')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
110 critical = db.priority.lookup('critical')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
111 urgent = db.priority.lookup('urgent')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
112 bug = db.priority.lookup('bug')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
113 feature = db.priority.lookup('feature')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
114 wish = db.priority.lookup('wish')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
115 adminuid = db.user.lookup('admin')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
116 anonuid = db.user.lookup('anonymous')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
117
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
118 root = ElementTree.parse(xml_file).getroot()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
119
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
120 def to_date(ts):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
121 return date.Date(time.gmtime(float(ts)))
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
122
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
123 # parse out the XML
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
124 artifacts = []
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
125 categories = sets.Set()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
126 users = sets.Set()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
127 add_files = sets.Set()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
128 remove_files = sets.Set()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
129 for artifact in root.find('artifacts'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
130 d = {}
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
131 op = {}
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
132 artifacts.append(d)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
133 for field in artifact.findall('field'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
134 name = field.get('name')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
135 if name == 'artifact_messages':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
136 for message in field.findall('message'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
137 l = d.setdefault('messages', [])
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
138 m = {}
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
139 l.append(m)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
140 for field in message.findall('field'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
141 name = field.get('name')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
142 if name == 'adddate':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
143 m[name] = to_date(field.text)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
144 else:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
145 m[name] = field.text
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
146 if name == 'user_name': users.add(field.text)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
147 elif name == 'artifact_history':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
148 for event in field.findall('history'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
149 l = d.setdefault('history', [])
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
150 e = {}
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
151 l.append(e)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
152 for field in event.findall('field'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
153 name = field.get('name')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
154 if name == 'entrydate':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
155 e[name] = to_date(field.text)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
156 else:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
157 e[name] = field.text
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
158 if name == 'mod_by': users.add(field.text)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
159 if e['field_name'] == 'File Added':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
160 add_files.add(e['old_value'].split(':')[0])
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
161 elif e['field_name'] == 'File Deleted':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
162 remove_files.add(e['old_value'].split(':')[0])
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
163 elif name == 'details':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
164 op['body'] = field.text
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
165 elif name == 'submitted_by':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
166 op['user_name'] = field.text
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
167 d[name] = field.text
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
168 users.add(field.text)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
169 elif name == 'open_date':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
170 thedate = to_date(field.text)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
171 op['adddate'] = thedate
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
172 d[name] = thedate
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
173 else:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
174 d[name] = field.text
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
175
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
176 categories.add(d['category'])
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
177
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
178 if op.has_key('body'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
179 l = d.setdefault('messages', [])
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
180 l.insert(0, op)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
181
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
182 add_files -= remove_files
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
183
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
184 # create users
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
185 userd = {'nobody': '2'}
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
186 users.remove('nobody')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
187 data = [
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
188 {'id': '1', 'username': 'admin', 'password': password.Password('admin'),
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
189 'roles': 'Admin', 'address': 'richard@python.org'},
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
190 {'id': '2', 'username': 'anonymous', 'roles': 'Anonymous'},
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
191 ]
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
192 for n, user in enumerate(list(users)):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
193 userd[user] = n+3
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
194 data.append({'id': str(n+3), 'username': user, 'roles': 'User',
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
195 'address': '%s@users.sourceforge.net'%user})
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
196 write_csv(db.user, data)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
197 users=userd
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
198
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
199 # create categories
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
200 categoryd = {'None': None}
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
201 categories.remove('None')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
202 data = []
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
203 for n, category in enumerate(list(categories)):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
204 categoryd[category] = n
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
205 data.append({'id': str(n), 'name': category})
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
206 write_csv(db.keyword, data)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
207 categories = categoryd
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
208
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
209 # create issues
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
210 issue_data = []
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
211 file_data = []
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
212 message_data = []
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
213 issue_journal = []
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
214 message_id = 0
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
215 for artifact in artifacts:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
216 d = {}
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
217 d['id'] = artifact['artifact_id']
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
218 d['title'] = artifact['summary']
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
219 d['assignedto'] = users[artifact['assigned_to']]
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
220 if d['assignedto'] == '2':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
221 d['assignedto'] = None
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
222 d['creation'] = artifact['open_date']
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
223 activity = artifact['open_date']
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
224 d['creator'] = users[artifact['submitted_by']]
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
225 actor = d['creator']
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
226 if categories[artifact['category']]:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
227 d['topic'] = [categories[artifact['category']]]
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
228 issue_journal.append((
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
229 d['id'], d['creation'].get_tuple(), d['creator'], "'create'", {}
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
230 ))
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
231
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
232 p = int(artifact['priority'])
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
233 if artifact['artifact_type'] == 'Feature Requests':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
234 if p > 3:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
235 d['priority'] = feature
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
236 else:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
237 d['priority'] = wish
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
238 else:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
239 if p > 7:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
240 d['priority'] = critical
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
241 elif p > 5:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
242 d['priority'] = urgent
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
243 elif p > 3:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
244 d['priority'] = bug
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
245 else:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
246 d['priority'] = feature
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
247
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
248 s = artifact['status']
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
249 if s == 'Closed':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
250 d['status'] = resolved
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
251 elif s == 'Deleted':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
252 d['status'] = resolved
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
253 d['is retired'] = True
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
254 else:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
255 d['status'] = unread
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
256
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
257 messages = []
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
258 nosy = sets.Set()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
259 for message in artifact.get('messages', []):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
260 message_id += 1
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
261 authid = users[message['user_name']]
3542
a2fc38fd0142 fixes and speedup
Richard Jones <richard@users.sourceforge.net>
parents: 3541
diff changeset
262 if not message['body']: continue
3534
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
263 body = convert_message(message['body'], message_id)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
264 if not body: continue
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
265 m = {'content': body, 'author': authid,
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
266 'date': message['adddate'], 'id': str(message_id),
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
267 'creation': message['adddate'], }
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
268 message_data.append(m)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
269 messages.append(message_id)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
270 if authid not in (None, '2'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
271 nosy.add(authid)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
272 activity = message['adddate']
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
273 actor = authid
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
274 if d['status'] == unread:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
275 d['status'] = chatting
3538
7f7902af7864 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3537
diff changeset
276
7f7902af7864 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3537
diff changeset
277 message_id += 1
7f7902af7864 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3537
diff changeset
278 m = {'content': 'IMPORT FROM SOURCEFORGE', 'author': '1',
7f7902af7864 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3537
diff changeset
279 'date': today, 'id': str(message_id), 'creation': today }
7f7902af7864 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3537
diff changeset
280 message_data.append(m)
7f7902af7864 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3537
diff changeset
281 messages.append(message_id)
7f7902af7864 *** empty log message ***
Richard Jones <richard@users.sourceforge.net>
parents: 3537
diff changeset
282
3534
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
283 d['messages'] = messages
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
284 d['nosy'] = list(nosy)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
285
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
286 files = []
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
287 for event in artifact.get('history', []):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
288 if event['field_name'] == 'File Added':
3542
a2fc38fd0142 fixes and speedup
Richard Jones <richard@users.sourceforge.net>
parents: 3541
diff changeset
289 fid, name = event['old_value'].split(':', 1)
3534
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
290 if fid in add_files:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
291 files.append(fid)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
292 name = name.strip()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
293 try:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
294 f = open(os.path.join(file_dir, fid))
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
295 content = f.read()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
296 f.close()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
297 except:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
298 content = 'content missing'
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
299 file_data.append({
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
300 'id': fid,
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
301 'creation': event['entrydate'],
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
302 'creator': users[event['mod_by']],
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
303 'name': name,
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
304 'type': mimetypes.guess_type(name)[0],
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
305 'content': content,
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
306 })
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
307 continue
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
308 elif event['field_name'] == 'close_date':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
309 action = "'set'"
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
310 info = { 'status': 'unread' }
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
311 elif event['field_name'] == 'summary':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
312 action = "'set'"
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
313 info = { 'title': event['old_value'] }
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
314 else:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
315 continue
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
316 row = [ d['id'], event['entrydate'].get_tuple(),
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
317 users[event['mod_by']], action, info ]
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
318 if event['entrydate'] > activity:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
319 activity = event['entrydate']
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
320 issue_journal.append(row)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
321 d['files'] = files
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
322
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
323 d['activity'] = activity
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
324 d['actor'] = actor
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
325 issue_data.append(d)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
326
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
327 write_csv(db.issue, issue_data)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
328 write_csv(db.msg, message_data)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
329 write_csv(db.file, file_data)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
330
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
331 f = open('/tmp/imported/issue-journals.csv', 'w')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
332 writer = csv.writer(f, colon_separated)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
333 writer.writerows(issue_journal)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
334 f.close()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
335
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
336 def convert_message(content, id):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
337 ''' Strip off the useless sf message header crap '''
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
338 if content[:14] == 'Logged In: YES':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
339 return '\n'.join(content.splitlines()[3:]).strip()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
340 return content
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
341
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
342 class colon_separated(csv.excel):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
343 delimiter = ':'
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
344
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
345 def write_csv(klass, data):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
346 props = klass.getprops()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
347 if not os.path.exists('/tmp/imported'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
348 os.mkdir('/tmp/imported')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
349 f = open('/tmp/imported/%s.csv'%klass.classname, 'w')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
350 writer = csv.writer(f, colon_separated)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
351 propnames = klass.export_propnames()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
352 propnames.append('is retired')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
353 writer.writerow(propnames)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
354 for entry in data:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
355 row = []
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
356 for name in propnames:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
357 if name == 'is retired':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
358 continue
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
359 prop = props[name]
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
360 if entry.has_key(name):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
361 if isinstance(prop, hyperdb.Date) or \
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
362 isinstance(prop, hyperdb.Interval):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
363 row.append(repr(entry[name].get_tuple()))
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
364 elif isinstance(prop, hyperdb.Password):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
365 row.append(repr(str(entry[name])))
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
366 else:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
367 row.append(repr(entry[name]))
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
368 elif isinstance(prop, hyperdb.Multilink):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
369 row.append('[]')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
370 elif name in ('creator', 'actor'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
371 row.append("'1'")
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
372 elif name in ('created', 'activity'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
373 row.append(repr(today.get_tuple()))
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
374 else:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
375 row.append('None')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
376 row.append(entry.get('is retired', False))
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
377 writer.writerow(row)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
378
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
379 if isinstance(klass, hyperdb.FileClass) and entry.get('content'):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
380 fname = klass.exportFilename('/tmp/imported/', entry['id'])
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
381 support.ensureParentsExist(fname)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
382 c = open(fname, 'w')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
383 if isinstance(entry['content'], unicode):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
384 c.write(entry['content'].encode('utf8'))
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
385 else:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
386 c.write(entry['content'])
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
387 c.close()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
388
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
389 f.close()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
390 f = open('/tmp/imported/%s-journals.csv'%klass.classname, 'w')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
391 f.close()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
392
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
393 class Progress:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
394 '''Progress display for console applications.
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
395
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
396 See __main__ block at end of file for sample usage.
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
397 '''
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
398 def __init__(self, info, sequence):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
399 self.info = info
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
400 self.sequence = iter(sequence)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
401 self.total = len(sequence)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
402 self.start = self.now = time.time()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
403 self.num = 0
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
404 self.stepsize = self.total / 100 or 1
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
405 self.steptimes = []
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
406 self.display()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
407
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
408 def __iter__(self): return self
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
409
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
410 def next(self):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
411 self.num += 1
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
412
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
413 if self.num > self.total:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
414 print self.info, 'done', ' '*(75-len(self.info)-6)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
415 sys.stdout.flush()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
416 return self.sequence.next()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
417
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
418 if self.num % self.stepsize:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
419 return self.sequence.next()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
420
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
421 self.display()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
422 return self.sequence.next()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
423
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
424 def display(self):
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
425 # figure how long we've spent - guess how long to go
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
426 now = time.time()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
427 steptime = now - self.now
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
428 self.steptimes.insert(0, steptime)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
429 if len(self.steptimes) > 5:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
430 self.steptimes.pop()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
431 steptime = sum(self.steptimes) / len(self.steptimes)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
432 self.now = now
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
433 eta = steptime * ((self.total - self.num)/self.stepsize)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
434
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
435 # tell it like it is (or might be)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
436 if now - self.start > 3:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
437 M = eta / 60
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
438 H = M / 60
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
439 M = M % 60
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
440 S = eta % 60
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
441 s = '%s %2d%% (ETA %02d:%02d:%02d)'%(self.info,
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
442 self.num * 100. / self.total, H, M, S)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
443 else:
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
444 s = '%s %2d%%'%(self.info, self.num * 100. / self.total)
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
445 sys.stdout.write(s + ' '*(75-len(s)) + '\r')
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
446 sys.stdout.flush()
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
447
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
448 if __name__ == '__main__':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
449 if sys.argv[1] == 'import':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
450 import_xml(*sys.argv[2:])
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
451 elif sys.argv[1] == 'files':
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
452 fetch_files(*sys.argv[2:])
ad9f06e32f7c scripts/import_sf.py will import a tracker from Sourceforge.NET
Richard Jones <richard@users.sourceforge.net>
parents:
diff changeset
453
Roundup Issue Tracker: http://roundup-tracker.org/