Mercurial > p > roundup > code
comparison doc/rest.txt @ 5710:0b79bfcb3312
Add support for making an idempotent POST. This allows retrying a POST
that was interrupted. It involves creating a post once only (poe) url
/rest/data/<class>/@poe/<random_token>. This url acts the same as a
post to /rest/data/<class>. However once the @poe url is used, it
can't be used for a second POST.
To make these changes:
1) Take the body of post_collection into a new post_collection_inner
function. Have post_collection call post_collection_inner.
2) Add a handler for POST to rest/data/class/@poe. This will return a
unique POE url. By default the url expires after 30 minutes. The
POE random token is only good for a specific user and is stored in
the session db.
3) Add a handler for POST to rest/data/<class>/@poe/<random token>.
The random token generated in 2 is validated for proper class (if
token is not generic) and proper user and must not have expired.
If everything is valid, call post_collection_inner to process the
input and generate the new entry.
To make recognition of 2 stable (so it's not confused with
rest/data/<:class_name>/<:item_id>), removed @ from
Routing::url_to_regex.
The current Routing.execute method stops on the first regular
expression to match the URL. Since item_id doesn't accept a POST, I
was getting 405 bad method sometimes. My guess is the order of the
regular expressions is not stable, so sometime I would get the right
regexp for /data/<class>/@poe and sometime I would get the one for
/data/<class>/<item_id>. By removing the @ from the url_to_regexp,
there was no way for the item_id case to match @poe.
There are alternate fixes we may need to look at. If a regexp matches
but the method does not, return to the regexp matching loop in
execute() looking for another match. Only once every possible match
has failed should the code return a 405 method failure.
Another fix is to implement a more sophisticated mechanism so that
@Routing.route("/data/<:class_name>/<:item_id>/<:attr_name>", 'PATCH')
has different regexps for matching <:class_name> <:item_id> and
<:attr_name>. Currently the regexp specified by url_to_regex is used
for every component.
Other fixes:
Made failure to find any props in props_from_args return an empty
dict rather than throwing an unhandled error.
Make __init__ for SimulateFieldStorageFromJson handle an empty json
doc. Useful for POSTing to rest/data/class/@poe with an empty
document.
Testing:
added testPostPOE to test/rest_common.py that I think covers
all the code that was added.
Documentation:
Add doc to rest.txt in the "Client API" section titled: Safely
Re-sending POST". Move existing section "Adding new rest endpoints" in
"Client API" to a new second level section called "Programming the
REST API". Also a minor change to the simple rest client moving the
header setting to continuation lines rather than showing one long
line.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Sun, 14 Apr 2019 21:07:11 -0400 |
| parents | c7dd1cae3416 |
| children | 59a3bbd3603a |
comparison
equal
deleted
inserted
replaced
| 5709:e2378b6afdb5 | 5710:0b79bfcb3312 |
|---|---|
| 183 >>> r = s.get (u + 'issue/42') | 183 >>> r = s.get (u + 'issue/42') |
| 184 >>> etag = r.headers['ETag'] | 184 >>> etag = r.headers['ETag'] |
| 185 >>> print("ETag: %s" % etag) | 185 >>> print("ETag: %s" % etag) |
| 186 >>> etag = r.json()['data']['@etag'] | 186 >>> etag = r.json()['data']['@etag'] |
| 187 >>> print("@etag: %s" % etag) | 187 >>> print("@etag: %s" % etag) |
| 188 >>> h = {'If-Match': etag, 'X-Requested-With': 'rest', 'Referer': 'http://tracker.example.com/demo/'} | 188 >>> h = {'If-Match': etag, |
| 189 ... 'X-Requested-With': 'rest', | |
| 190 ... 'Referer': 'http://tracker.example.com/demo/'} | |
| 189 >>> d = {'@op:'action', '@action_name':'retire'} | 191 >>> d = {'@op:'action', '@action_name':'retire'} |
| 190 >>> r = s.patch(u + 'issue/42', data = d, headers = h) | 192 >>> r = s.patch(u + 'issue/42', data = d, headers = h) |
| 191 >>> print(r.json()) | 193 >>> print(r.json()) |
| 192 >>> d = {'@op:'action', '@action_name':'restore'} | 194 >>> d = {'@op:'action', '@action_name':'restore'} |
| 193 >>> r = s.patch(u + 'issue/42', data = d, headers = h) | 195 >>> r = s.patch(u + 'issue/42', data = d, headers = h) |
| 194 >>> print(r.json()) | 196 >>> print(r.json()) |
| 195 | 197 |
| 196 Note the addition of headers for: x-requested-with and referer. This | 198 Note the addition of headers for: x-requested-with and referer. This |
| 197 allows the request to pass the CSRF protection mechanism. You may need | 199 allows the request to pass the CSRF protection mechanism. You may need |
| 198 to add Origin if this check is enabled in your tracker's config.ini. | 200 to add an Origin header if this check is enabled in your tracker's |
| 199 | 201 config.ini (look for csrf_enforce_header_origin). |
| 202 | |
| 203 Safely Re-sending POST | |
| 204 ====================== | |
| 205 | |
| 206 POST is used to create new object in a class. E.G. a new issue. One | |
| 207 problem is that a POST may time out. Because it is not idempotent like | |
| 208 a PUT or DELETE, retrying the interrupted POST may result in the | |
| 209 creation of a duplicate issue. | |
| 210 | |
| 211 To solve this problem, a two step process inspired by the POE - Post | |
| 212 Once Exactly spec: | |
| 213 https://tools.ietf.org/html/draft-nottingham-http-poe-00 is provided. | |
| 214 | |
| 215 This mechanism returns a single use URL. POSTing to the URL creates | |
| 216 a new object in the class. | |
| 217 | |
| 218 First we get the URL. Here is an example using curl:: | |
| 219 | |
| 220 curl -u demo:demo -s -X POST -H "Referer: https://.../demo/" \ | |
| 221 -H "X-requested-with: rest" \ | |
| 222 -H "Content-Type: application/json" \ | |
| 223 --data '' \ | |
| 224 https://.../demo/rest/data/issue/@poe | |
| 225 | |
| 226 This will return a json payload like:: | |
| 227 | |
| 228 { | |
| 229 "data": { | |
| 230 "expires": 1555266310.4457426, | |
| 231 "link": "https://.../demo/rest/data/issue/@poe/vizl713xHtIzANRW9jPb3bWXePRzmehdmSXzEta1" | |
| 232 } | |
| 233 } | |
| 234 | |
| 235 The value of expires is a Unix timestamp in seconds. In this case it | |
| 236 has the default lifetime of 30 minutes after the current time. Using | |
| 237 the link more than 30 minutes into the future will cause a 400 error. | |
| 238 | |
| 239 Within 30 minutes, the link can be used to post an issue with the same | |
| 240 payload that would normally be sent to: | |
| 241 ``https://.../demo/rest/data/issue``. | |
| 242 | |
| 243 For example:: | |
| 244 | |
| 245 curl -u demo:demo -s -X POST \ | |
| 246 -H "Referer: https://.../demo/" \ | |
| 247 -H "X-requested-with: rest" \ | |
| 248 -H "Content-Type: application/json" \ | |
| 249 --data-binary '{ "title": "a problem" }' \ | |
| 250 https://.../demo/rest/data/issue/@poe/vizl713xHtIzANRW9jPb3bWXePRzmehdmSXzEta1 | |
| 251 | |
| 252 returns:: | |
| 253 | |
| 254 { | |
| 255 "data": { | |
| 256 "link": "https://.../demo/rest/data/issue/2280", | |
| 257 "id": "2280" | |
| 258 } | |
| 259 } | |
| 260 | |
| 261 Once the @poe link is used and creates an issue, it becomes invalid | |
| 262 and can't be used again. Posting to it after the issue, or other | |
| 263 object, is created, results in a 400 error [#poe_retry]_. | |
| 264 | |
| 265 Note that POE links are by restricted to the class that was used to | |
| 266 get the link. So you can only create an issue using the link returned | |
| 267 from ``rest/data/issue/@poe``. You can create a generic POE link by adding | |
| 268 the "generic" field to the post payload:: | |
| 269 | |
| 270 curl -u demo:demo -s -X POST -H "Referer: https://.../demo/" \ | |
| 271 -H "X-requested-with: rest" \ | |
| 272 --data 'lifetime=100&generic=1' \ | |
| 273 https://.../demo/rest/data/issue/@poe | |
| 274 | |
| 275 This will return a link under: ``https://.../demo/rest/data/issue/@poe``:: | |
| 276 | |
| 277 { | |
| 278 "data": { | |
| 279 "expires": 1555268640.9606116, | |
| 280 "link": | |
| 281 "https://.../demo/rest/data/issue/@poe/slPrzmEq6Q9BTjvcKhfxMNZL4uHXjbHCidY1ludZ" | |
| 282 } | |
| 283 } | |
| 284 | |
| 285 You could use the link and change 'issue' to 'user' and it would work | |
| 286 to create a user. Creating generic POE tokens is *not* recommended, | |
| 287 but is available if a use case requires it. | |
| 288 | |
| 289 This example also changes the lifetime of the POE url. This link has | |
| 290 a lifetime of 15 minutes (900 seconds). Using it after 16 minutes will | |
| 291 result in a 400 error. A lifetime up to 1 hour can be specified. | |
| 292 | |
| 293 POE url's are an optional mechanism. If: | |
| 294 | |
| 295 * you do not expect your client to retry a failed post, | |
| 296 * a failed post is unlikely (e.g. you are running over a local lan), | |
| 297 * there is a human using the client and who can intervene if a post | |
| 298 fails | |
| 299 | |
| 300 you can use the url ``https://.../demo/data/<class>``. However if you | |
| 301 are using this mechanism to automate creation of objects and will | |
| 302 automatically retry a post until it succeeds, please use the POE | |
| 303 mechanism. | |
| 304 | |
| 305 .. [#poe_retry] At some future date, performing a POST to the POE link | |
| 306 soon after it has been used to create an object will | |
| 307 change. It will not return a 400 error. It will will trigger a | |
| 308 redirect to the url for the created object. After some period | |
| 309 of time (maybe a week) the POE link will be removed and return | |
| 310 a 400 error. This is meant to allow the client (a time limited | |
| 311 way) to retrieve the created resource when the response was | |
| 312 lost. | |
| 313 | |
| 314 Searches and selection | |
| 315 ====================== | |
| 316 | |
| 317 One difficult interface issue is selection of items from a long list. | |
| 318 Using multi-item selects requires loading a lot of data (e.g. consider | |
| 319 a selection tool to select one or more issues as in the classic | |
| 320 superseder field). | |
| 321 | |
| 322 This can be made easier using javascript selection tools like select2, | |
| 323 selectize.js, chosen etc. These tools can query a remote data provider | |
| 324 to get a list of items for the user to select from. | |
| 325 | |
| 326 Consider a multi-select box for the superseder property. Using | |
| 327 selectize.js (and jquery) code similar to:: | |
| 328 | |
| 329 $('#superseder').selectize({ | |
| 330 valueField: 'id', | |
| 331 labelField: 'title', | |
| 332 searchField: 'title', ... | |
| 333 load: function(query, callback) { | |
| 334 if (!query.length) return callback(); | |
| 335 $.ajax({ | |
| 336 url: '.../rest/data/issue?@verbose=2&title=' | |
| 337 + encodeURIComponent(query), | |
| 338 type: 'GET', | |
| 339 error: function() {callback();}, | |
| 340 success: function(res) { | |
| 341 callback(res.data.collection);} | |
| 342 | |
| 343 Sets up a box that a user can type the word "request" into. Then | |
| 344 selectize.js will use that word to generate an ajax request with the | |
| 345 url: ``.../rest/data/issue?@verbose=2&title=request`` | |
| 346 | |
| 347 This will return data like:: | |
| 348 | |
| 349 { | |
| 350 "data": { | |
| 351 "@total_size": 440, | |
| 352 "collection": [ | |
| 353 { | |
| 354 "link": ".../rest/data/issue/8", | |
| 355 "id": "8", | |
| 356 "title": "Request for Power plugs" | |
| 357 }, | |
| 358 { | |
| 359 "link": ".../rest/data/issue/27", | |
| 360 "id": "27", | |
| 361 "title": "Request for foo" | |
| 362 }, | |
| 363 ... | |
| 364 | |
| 365 selectize.js will look at these objects (as passed to | |
| 366 callback(res.data.collection)) and create a select list from the each | |
| 367 object showing the user the labelField (title) for each object and | |
| 368 associating each title with the corresponding valueField (id). The | |
| 369 example above has 440 issues returned from a total of 2000 | |
| 370 issues. Only 440 had the word "request" somewhere in the title greatly | |
| 371 reducing the amount of data that needed to be transferred. | |
| 372 | |
| 373 Similar code can be set up to search a large list of keywords using:: | |
| 374 | |
| 375 .../rest/data/keyword?@verbose=2&name=some | |
| 376 | |
| 377 which would return: "some keyword" "awesome" "somebody" making | |
| 378 selections for links and multilinks much easier. | |
| 379 | |
| 380 Hopefully future enhancements will allow get on a collection to | |
| 381 include other fields. Why do we want this? Selectize.js can set up | |
| 382 option groups (optgroups) in the select pulldown. So by including | |
| 383 status in the returned data:: | |
| 384 | |
| 385 { | |
| 386 "link": ".../rest/data/issue/27", | |
| 387 "id": "27", | |
| 388 "title": "Request for foo", | |
| 389 'status": "open" | |
| 390 }, | |
| 391 | |
| 392 a select widget like:: | |
| 393 | |
| 394 === New === | |
| 395 A request | |
| 396 === Open === | |
| 397 Request for bar | |
| 398 Request for foo | |
| 399 | |
| 400 etc. can be generated. Also depending on the javascript library, other | |
| 401 fields can be used for subsearch and sorting. | |
| 402 | |
| 403 | |
| 404 Programming the REST API | |
| 405 ------------------------ | |
| 406 | |
| 407 You can extend the rest api for a tracker. This describes how to add | |
| 408 new rest end points. At some point it will also describe the rest.py | |
| 409 structure and implementation. | |
| 200 | 410 |
| 201 Adding new rest endpoints | 411 Adding new rest endpoints |
| 202 ========================= | 412 ========================= |
| 203 | 413 |
| 204 Add or edit the file interfaces.py at the root of the tracker | 414 Add or edit the file interfaces.py at the root of the tracker |
| 269 } | 479 } |
| 270 | 480 |
| 271 | 481 |
| 272 Adding other endpoints (e.g. to allow an OPTIONS query against | 482 Adding other endpoints (e.g. to allow an OPTIONS query against |
| 273 ``/data/issue/@schema``) is left as an exercise for the reader. | 483 ``/data/issue/@schema``) is left as an exercise for the reader. |
| 274 | |
| 275 Searches and selection | |
| 276 ====================== | |
| 277 | |
| 278 One difficult interface issue is selection of items from a long list. | |
| 279 Using multi-item selects requires loading a lot of data (e.g. consider | |
| 280 a selection tool to select one or more issues as in the classic | |
| 281 superseder field). | |
| 282 | |
| 283 This can be made easier using javascript selection tools like select2, | |
| 284 selectize.js, chosen etc. These tools can query a remote data provider | |
| 285 to get a list of items for the user to select from. | |
| 286 | |
| 287 Consider a multi-select box for the superseder property. Using | |
| 288 selectize.js (and jquery) code similar to:: | |
| 289 | |
| 290 $('#superseder').selectize({ | |
| 291 valueField: 'id', | |
| 292 labelField: 'title', | |
| 293 searchField: 'title', ... | |
| 294 load: function(query, callback) { | |
| 295 if (!query.length) return callback(); | |
| 296 $.ajax({ | |
| 297 url: '.../rest/data/issue?@verbose=2&title=' | |
| 298 + encodeURIComponent(query), | |
| 299 type: 'GET', | |
| 300 error: function() {callback();}, | |
| 301 success: function(res) { | |
| 302 callback(res.data.collection);} | |
| 303 | |
| 304 Sets up a box that a user can type the word "request" into. Then | |
| 305 selectize.js will use that word to generate an ajax request with the | |
| 306 url: ``.../rest/data/issue?@verbose=2&title=request`` | |
| 307 | |
| 308 This will return data like:: | |
| 309 | |
| 310 { | |
| 311 "data": { | |
| 312 "@total_size": 440, | |
| 313 "collection": [ | |
| 314 { | |
| 315 "link": ".../rest/data/issue/8", | |
| 316 "id": "8", | |
| 317 "title": "Request for Power plugs" | |
| 318 }, | |
| 319 { | |
| 320 "link": ".../rest/data/issue/27", | |
| 321 "id": "27", | |
| 322 "title": "Request for foo" | |
| 323 }, | |
| 324 ... | |
| 325 | |
| 326 selectize.js will look at these objects (as passed to | |
| 327 callback(res.data.collection)) and create a select list from the each | |
| 328 object showing the user the labelField (title) for each object and | |
| 329 associating each title with the corresponding valueField (id). The | |
| 330 example above has 440 issues returned from a total of 2000 | |
| 331 issues. Only 440 had the word "request" somewhere in the title greatly | |
| 332 reducing the amount of data that needed to be transferred. | |
| 333 | |
| 334 Similar code can be set up to search a large list of keywords using:: | |
| 335 | |
| 336 .../rest/data/keyword?@verbose=2&name=some | |
| 337 | |
| 338 which would return: "some keyword" "awesome" "somebody" making | |
| 339 selections for links and multilinks much easier. | |
| 340 | |
| 341 Hopefully future enhancements will allow get on a collection to | |
| 342 include other fields. Why do we want this? Selectize.js can set up | |
| 343 option groups (optgroups) in the select pulldown. So by including | |
| 344 status in the returned data:: | |
| 345 | |
| 346 { | |
| 347 "link": ".../rest/data/issue/27", | |
| 348 "id": "27", | |
| 349 "title": "Request for foo", | |
| 350 'status": "open" | |
| 351 }, | |
| 352 | |
| 353 a select widget like:: | |
| 354 | |
| 355 === New === | |
| 356 A request | |
| 357 === Open === | |
| 358 Request for bar | |
| 359 Request for foo | |
| 360 | |
| 361 etc. can be generated. Also depending on the javascript library, other | |
| 362 fields can be used for subsearch and sorting. |