annotate roundup/dist/command/build.py @ 5710:0b79bfcb3312

Add support for making an idempotent POST. This allows retrying a POST that was interrupted. It involves creating a post once only (poe) url /rest/data/<class>/@poe/<random_token>. This url acts the same as a post to /rest/data/<class>. However once the @poe url is used, it can't be used for a second POST. To make these changes: 1) Take the body of post_collection into a new post_collection_inner function. Have post_collection call post_collection_inner. 2) Add a handler for POST to rest/data/class/@poe. This will return a unique POE url. By default the url expires after 30 minutes. The POE random token is only good for a specific user and is stored in the session db. 3) Add a handler for POST to rest/data/<class>/@poe/<random token>. The random token generated in 2 is validated for proper class (if token is not generic) and proper user and must not have expired. If everything is valid, call post_collection_inner to process the input and generate the new entry. To make recognition of 2 stable (so it's not confused with rest/data/<:class_name>/<:item_id>), removed @ from Routing::url_to_regex. The current Routing.execute method stops on the first regular expression to match the URL. Since item_id doesn't accept a POST, I was getting 405 bad method sometimes. My guess is the order of the regular expressions is not stable, so sometime I would get the right regexp for /data/<class>/@poe and sometime I would get the one for /data/<class>/<item_id>. By removing the @ from the url_to_regexp, there was no way for the item_id case to match @poe. There are alternate fixes we may need to look at. If a regexp matches but the method does not, return to the regexp matching loop in execute() looking for another match. Only once every possible match has failed should the code return a 405 method failure. Another fix is to implement a more sophisticated mechanism so that @Routing.route("/data/<:class_name>/<:item_id>/<:attr_name>", 'PATCH') has different regexps for matching <:class_name> <:item_id> and <:attr_name>. Currently the regexp specified by url_to_regex is used for every component. Other fixes: Made failure to find any props in props_from_args return an empty dict rather than throwing an unhandled error. Make __init__ for SimulateFieldStorageFromJson handle an empty json doc. Useful for POSTing to rest/data/class/@poe with an empty document. Testing: added testPostPOE to test/rest_common.py that I think covers all the code that was added. Documentation: Add doc to rest.txt in the "Client API" section titled: Safely Re-sending POST". Move existing section "Adding new rest endpoints" in "Client API" to a new second level section called "Programming the REST API". Also a minor change to the simple rest client moving the header setting to continuation lines rather than showing one long line.
author John Rouillard <rouilj@ieee.org>
date Sun, 14 Apr 2019 21:07:11 -0400
parents f2fade4552c5
children 42bf0a707763
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
4068
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
1 #
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
2 # Copyright (C) 2009 Stefan Seefeld
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
3 # All rights reserved.
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
4 # For license terms see the file COPYING.txt.
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
5 #
5376
64b05e24dbd8 Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents: 4516
diff changeset
6 from __future__ import print_function
4068
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
7 from roundup import msgfmt
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
8 from distutils.command.build import build as base
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
9 import os
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
10 from glob import glob
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
11
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
12 def list_message_files(suffix=".po"):
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
13 """Return list of all found message files and their intallation paths"""
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
14 _files = glob("locale/*" + suffix)
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
15 _list = []
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
16 for _file in _files:
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
17 # basename (without extension) is a locale name
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
18 _locale = os.path.splitext(os.path.basename(_file))[0]
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
19 _list.append((_file, os.path.join(
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
20 "share", "locale", _locale, "LC_MESSAGES", "roundup.mo")))
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
21 return _list
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
22
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
23 def check_manifest():
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
24 """Check that the files listed in the MANIFEST are present when the
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
25 source is unpacked.
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
26 """
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
27 try:
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
28 f = open('MANIFEST')
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
29 except:
5376
64b05e24dbd8 Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents: 4516
diff changeset
30 print('\n*** SOURCE WARNING: The MANIFEST file is missing!')
4068
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
31 return
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
32 try:
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
33 manifest = [l.strip() for l in f.readlines()]
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
34 finally:
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
35 f.close()
4394
d4cd0a264098 fixed reporting of source missing warnings
Richard Jones <richard@users.sourceforge.net>
parents: 4068
diff changeset
36 err = set([line for line in manifest if not os.path.exists(line)])
4068
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
37 # ignore auto-generated files
4394
d4cd0a264098 fixed reporting of source missing warnings
Richard Jones <richard@users.sourceforge.net>
parents: 4068
diff changeset
38 err = err - set(['roundup-admin', 'roundup-demo', 'roundup-gettext',
d4cd0a264098 fixed reporting of source missing warnings
Richard Jones <richard@users.sourceforge.net>
parents: 4068
diff changeset
39 'roundup-mailgw', 'roundup-server', 'roundup-xmlrpc-server'])
4068
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
40 if err:
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
41 n = len(manifest)
5376
64b05e24dbd8 Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents: 4516
diff changeset
42 print('\n*** SOURCE WARNING: There are files missing (%d/%d found)!'%(
64b05e24dbd8 Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents: 4516
diff changeset
43 n-len(err), n))
64b05e24dbd8 Python 3 preparation: convert print to a function.
Joseph Myers <jsm@polyomino.org.uk>
parents: 4516
diff changeset
44 print('Missing:', '\nMissing: '.join(err))
4068
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
45
4516
85dfe17c182e Installation:
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4394
diff changeset
46 def build_message_files(command):
85dfe17c182e Installation:
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4394
diff changeset
47 """For each locale/*.po, build .mo file in target locale directory"""
85dfe17c182e Installation:
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4394
diff changeset
48 for (_src, _dst) in list_message_files():
85dfe17c182e Installation:
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4394
diff changeset
49 _build_dst = os.path.join("build", _dst)
85dfe17c182e Installation:
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4394
diff changeset
50 command.mkpath(os.path.dirname(_build_dst))
85dfe17c182e Installation:
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4394
diff changeset
51 command.announce("Compiling %s -> %s" % (_src, _build_dst))
5450
f2fade4552c5 replaced msgfmt.py with latest version supporting Python 3
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5376
diff changeset
52 mo = msgfmt.Msgfmt(_src).get()
f2fade4552c5 replaced msgfmt.py with latest version supporting Python 3
Christof Meerwald <cmeerw@cmeerw.org>
parents: 5376
diff changeset
53 open(_build_dst, 'wb').write(mo)
4516
85dfe17c182e Installation:
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4394
diff changeset
54
4068
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
55
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
56 class build(base):
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
57
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
58 def run(self):
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
59 check_manifest()
4516
85dfe17c182e Installation:
Bernhard Reiter <Bernhard.Reiter@intevation.de>
parents: 4394
diff changeset
60 build_message_files(self)
4068
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
61 base.run(self)
e233d7a66343 Refactor setup.py.
Stefan Seefeld <stefan@seefeld.name>
parents:
diff changeset
62

Roundup Issue Tracker: http://roundup-tracker.org/