====================
REST API for Roundup
====================

.. contents::
   :local:

Introduction
------------

After the last 1.6.0 Release, a REST-API developed in 2015 during a Google
Summer of Code (GSOC) by Chau Nguyen, supervised by Ezio Melotti was
integrated. The code was then updated by John Rouillard and Ralf
Schlatterbeck to fix some shortcomings and provide the necessary
functions for a single page web application, e.g. etag support among
others.

Enabling the REST API
---------------------

The REST API can be disabled in the ``[web]`` section of ``config.ini``
via the variable ``enable_rest`` which is ``yes`` by default.

The REST api is reached via the ``/rest/`` endpoint of the tracker URL.


Client API
----------

The top-level REST url ``/rest/`` will display the current version of
the REST API (Version 1 as of this writing) and some links to relevant
endpoints of the API. In the following the ``/rest`` prefix is ommitted
from relative REST-API links for brevety.

Summary
=======

A Summary page can be reached via ``/summary`` via the ``GET`` method.
This is currently hard-coded for the standard tracker schema shipped
with roundup and will display a summary of open issues.

Data
====

All the links mentioned in the following support the http method ``GET``. 
Results of a ``GET`` request will always return the results as a
dictionary with the entry ``data`` referring to the returned data.

The ``/data`` link will display a set of classes of the tracker. All
classes can be reached via ``/data/<classname>`` where ``<classname>``
is replace with the name of the class to query, e.g. ``/data/issue``.
Individual items of a class (e.g. a single issue) can be queried by
giving the issue-id, e.g., ``/data/issue/42``. Individual properties of
an item can be queried by appending the property, e.g.,
``/data/issue/42/title``.

When performing the ``GET`` method on a class (e.g. ``/data/issue``), the
number of items is returned in ``@total_size``. Then a ``collection``
list follows which contains the id and link to the respective item.
This endpoint supports pagination. With the attributes @page_size and
@page_index, pagination is controlled. The @page_size specifies how many
items are displayed at once. The @page_index (which defaults to 1 if not
given) specifies which page number of @page_size items is displayed. If
no @page_size is specified, all items are returned.

Adding the query parameter @verbose=2 to the GET will include the label
property in addition to id and link for the items. This is useful as
documented below in "Searches and selection".

In addition this method supports searching. Search parameters are names
of properties of the given class, e.g., ``status`` for ``issue``. Links
and Multilinks can be specified numerically or symbolically, e.g.,
searching for issues in status ``closed`` can be achieved by searching
for ``status=closed`` or ``status=3`` (provided the ``closed`` status
has ID 3). Note that searching for strings (e.g. the issue title, or a
keyword name) performs a case-insensitive substring search, so searching
for title=Something will find all issues with "Something" or "someThing",
etc. in the title. There is currently no way to perform an exact string
match.

When performing the ``GET`` method on an item (e.g. ``/data/issue/42``), a
``link`` attribute contains the link to the item, ``id`` contains the
id, type contains the class name (e.g. ``issue`` in the example) and an
``etag`` property can be used to detect modifications since the last
query. The individual properties of the item are returned in an
``attributes`` dictionary. The properties returned depend on the
permissions of the account used for the query.
Link and Multilink properties are displayed as a dictionary with a link
and an id property by default. This is controlled by the @verbose
attribute which is set to 1 by default. If set to 0, only the id is
shown for Link and Multilink attributes. If set to 2, the label property
(usually ``name`` e.g. for status) is also put into the dictionary.
Content properties of message and file object are by default also shown
as a dictionary with a sole link attribute. The link is the download
link for the file or message. If @verbose is >= 3, the content property
is shown in json as a (possibly very long) string. Currently the json
serializer cannot handle files not properly utf-8 encoded, so specifying
@verbose=3 for files is currently discouraged.
Note that if the class has a key attribute (like e.g., the 'status'
class in the classic tracker), you can get an individual status by
specifying the key-attribute e.g. ``/data/status/name=closed``. Note
that ``name`` in this example must be the key-attribute of the class.
A short-form (which might no longer be supported in future version of
the API) is to specify only the value, e.g. ``/data/status/closed``.
This short-form only works when you're sure that the key of the class is
not numeric.
The long-form (with ``=``) is different from a query-parameter like
``/data/status?@name=closed`` which would find all stati that have
``closed`` as a substring.

A ``GET`` method on a property (e.g. ``/data/issue/42/title``) returns the
link, an ``@etag``, the type of the property (e.g. "<type str>") the id
of the item and the content of the property in ``data``.

Only class links support the ``POST`` method for creation of new items
of a class, e.g., a new issue via the ``/data/issue`` link. The post
gets a dictionary of keys/values for the new item. It returns the same
parameters as the GET method after successful creation.

All endpoints support an ``OPTIONS`` method for determining which
methods are allowed on a given endpoint.

The method ``PUT`` is allowed on individual items, e.g.
``/data/issue/42`` as well as properties, e.g.,
``/data/issue/42/title``. On success it returns the same parameters as
the respective ``GET`` method. Note that for ``PUT`` an Etag has to be
supplied, either in the request header or as an @etag parameter.

The method ``DELETE`` is allowed on items, e.g., ``/data/issue/42`` and
will retire (mark as deleted) the respective item. On success it will
only return a status code. It is also possible to call ``DELETE`` on a
property of an item, e.g., ``/data/issue/42/nosy`` to delete the nosy
list. The same effect can be achieved with a ``PUT`` request and an
empty new value.

Finally the ``PATCH`` method can be applied to individual items, e.g.,
``/data/issue/42`` and to properties, e.g., ``/data/issue/42/title``.
This method gets an operator ``@op=<method>`` where ``<method>`` is one
of ``add``, ``replace``, ``remove``, only for an item (not for a
property) an additional operator ``action`` is supported. If no operator
is specified, the default is ``replace``. The first three operators are
self explanatory. For an ``action`` operator an ``@action_name`` and
optional ``@action_argsXXX`` parameters have to be supplied. Currently
there are only two actions, neither has args, namely ``retire`` and
``restore``. The ``retire`` action on an item is the same as a
``DELETE`` method, it retires the item. The ``restore`` action is the
inverse of ``retire``, the item is again visible.
On success the returned value is the same as the respective ``GET``
method.

Note that the ``GET`` method on an item (e.g. ``/data/issue/43``)
returns an ETag in the http header *and* the ``@etag`` value. When
modifying the item via ``PUT`` or ``PATCH`` either a ``If-Match`` header
or an ``@etag`` value in the form have to be provided.

sample python client
====================

The client uses the python ``requests`` library for easier interaction
with a REST API supporting JSON encoding::


        >>> import requests
        >>> u = 'http://user:password@tracker.example.com/demo/rest/data/'
        >>> s = requests.session()
        >>> r = s.get(u + 'issue/42/title')
        >>> if r.status_code != 200:
        ...     print("Failed: %s: %s" % (r.status_code, r.reason))
        ...     exit(1)
        >>> print (r.json() ['data']['data']
        TEST Title
        >>> h = {'X-Requested-With': 'rest', 'Referer': 'http://tracker.example.com/demo/'}
        >>> r = s.post (u + 'issue', data = dict (title = 'TEST Issue'), headers=h)
        >>> if not 200 <= r.status_code <= 201:
        ...     print("Failed: %s: %s" % (r.status_code, r.reason))
        ...     exit(1)
        >>> print(r.json())

Retire/Restore::
        >>> r = s.delete (u + 'issue/42')
        >>> print (r.json())
        >>> r = s.get (u + 'issue/42')
        >>> etag = r.headers['ETag']
        >>> print("ETag: %s" % etag)
        >>> etag = r.json()['data']['@etag']
        >>> print("@etag: %s" % etag)
        >>> h = {'If-Match': etag, 
        ...   'X-Requested-With': 'rest',
        ...   'Referer': 'http://tracker.example.com/demo/'}
        >>> d = {'@op:'action', '@action_name':'retire'}
        >>> r = s.patch(u + 'issue/42', data = d, headers = h)
        >>> print(r.json())
        >>> d = {'@op:'action', '@action_name':'restore'}
        >>> r = s.patch(u + 'issue/42', data = d, headers = h)
        >>> print(r.json())

Note the addition of headers for: x-requested-with and referer. This
allows the request to pass the CSRF protection mechanism. You may need
to add an Origin header if this check is enabled in your tracker's
config.ini (look for csrf_enforce_header_origin).

Safely Re-sending POST
======================

POST is used to create new object in a class. E.G. a new issue.  One
problem is that a POST may time out. Because it is not idempotent like
a PUT or DELETE, retrying the interrupted POST may result in the
creation of a duplicate issue.

To solve this problem, a two step process inspired by the POE - Post
Once Exactly spec:
https://tools.ietf.org/html/draft-nottingham-http-poe-00 is provided.

This mechanism returns a single use URL. POSTing to the URL creates
a new object in the class.

First we get the URL. Here is an example using curl::

  curl -u demo:demo -s -X POST -H "Referer: https://.../demo/" \
      -H "X-requested-with: rest" \
      -H "Content-Type: application/json" \
      --data '' \
      https://.../demo/rest/data/issue/@poe

This will return a json payload like::

  {
    "data": {
        "expires": 1555266310.4457426,
        "link": "https://.../demo/rest/data/issue/@poe/vizl713xHtIzANRW9jPb3bWXePRzmehdmSXzEta1"
    }
  }

The value of expires is a Unix timestamp in seconds. In this case it
has the default lifetime of 30 minutes after the current time. Using
the link more than 30 minutes into the future will cause a 400 error.

Within 30 minutes, the link can be used to post an issue with the same
payload that would normally be sent to:
``https://.../demo/rest/data/issue``.

For example::

   curl -u demo:demo -s -X POST \
     -H "Referer: https://.../demo/" \
     -H "X-requested-with: rest" \
     -H "Content-Type: application/json"  \
     --data-binary '{ "title": "a problem" }' \
     https://.../demo/rest/data/issue/@poe/vizl713xHtIzANRW9jPb3bWXePRzmehdmSXzEta1

returns::

  {
    "data": {
        "link": "https://.../demo/rest/data/issue/2280",
        "id": "2280"
    }
  }

Once the @poe link is used and creates an issue, it becomes invalid
and can't be used again.  Posting to it after the issue, or other
object, is created, results in a 400 error [#poe_retry]_.

Note that POE links are by restricted to the class that was used to
get the link. So you can only create an issue using the link returned
from ``rest/data/issue/@poe``. You can create a generic POE link by adding
the "generic" field to the post payload::

  curl -u demo:demo -s -X POST -H "Referer: https://.../demo/" \
      -H "X-requested-with: rest" \
      --data 'lifetime=100&generic=1' \
      https://.../demo/rest/data/issue/@poe

This will return a link under: ``https://.../demo/rest/data/issue/@poe``::

  {
    "data": {
        "expires": 1555268640.9606116,
        "link":
    "https://.../demo/rest/data/issue/@poe/slPrzmEq6Q9BTjvcKhfxMNZL4uHXjbHCidY1ludZ"
    }
  }

You could use the link and change 'issue' to 'user' and it would work
to create a user. Creating generic POE tokens is *not* recommended,
but is available if a use case requires it.

This example also changes the lifetime of the POE url.  This link has
a lifetime of 15 minutes (900 seconds). Using it after 16 minutes will
result in a 400 error. A lifetime up to 1 hour can be specified.

POE url's are an optional mechanism. If:

* you do not expect your client to retry a failed post,
* a failed post is unlikely (e.g. you are running over a local lan),
* there is a human using the client and who can intervene if a post
  fails

you can use the url ``https://.../demo/data/<class>``. However if you
are using this mechanism to automate creation of objects and will
automatically retry a post until it succeeds, please use the POE
mechanism.

.. [#poe_retry] At some future date, performing a POST to the POE link
        soon after it has been used to create an object will
        change. It will not return a 400 error. It will will trigger a
        redirect to the url for the created object. After some period
        of time (maybe a week) the POE link will be removed and return
        a 400 error. This is meant to allow the client (a time limited
        way) to retrieve the created resource when the response was
        lost.

Searches and selection
======================

One difficult interface issue is selection of items from a long list.
Using multi-item selects requires loading a lot of data (e.g. consider
a selection tool to select one or more issues as in the classic
superseder field).

This can be made easier using javascript selection tools like select2,
selectize.js, chosen etc. These tools can query a remote data provider
to get a list of items for the user to select from.

Consider a multi-select box for the superseder property.  Using
selectize.js (and jquery) code similar to::

    $('#superseder').selectize({
	valueField: 'id',
	labelField: 'title',
	searchField: 'title', ...
	load: function(query, callback) {
		if (!query.length) return callback();
		$.ajax({
			url: '.../rest/data/issue?@verbose=2&title='
			    + encodeURIComponent(query),
			type: 'GET',
			error: function() {callback();},
			success: function(res) {
			  callback(res.data.collection);}

Sets up a box that a user can type the word "request" into. Then
selectize.js will use that word to generate an ajax request with the
url: ``.../rest/data/issue?@verbose=2&title=request``

This will return data like::

  {
    "data": {
    "@total_size": 440,
    "collection": [
      {
	  "link": ".../rest/data/issue/8",
	  "id": "8",
	  "title": "Request for Power plugs"
      },
      {
	  "link": ".../rest/data/issue/27",
	  "id": "27",
	  "title": "Request for foo"
      },
  ...

selectize.js will look at these objects (as passed to
callback(res.data.collection)) and create a select list from the each
object showing the user the labelField (title) for each object and
associating each title with the corresponding valueField (id). The
example above has 440 issues returned from a total of 2000
issues. Only 440 had the word "request" somewhere in the title greatly
reducing the amount of data that needed to be transferred.

Similar code can be set up to search a large list of keywords using::

  .../rest/data/keyword?@verbose=2&name=some

which would return: "some keyword" "awesome" "somebody" making
selections for links and multilinks much easier.

Hopefully future enhancements will allow get on a collection to
include other fields. Why do we want this?  Selectize.js can set up
option groups (optgroups) in the select pulldown. So by including
status in the returned data::

      {
	  "link": ".../rest/data/issue/27",
	  "id": "27",
	  "title": "Request for foo",
	  'status": "open"
      },

a select widget like::

  === New ===
  A request
  === Open ===
  Request for bar
  Request for foo

etc. can be generated. Also depending on the javascript library, other
fields can be used for subsearch and sorting.


Programming the REST API
------------------------

You can extend the rest api for a tracker. This describes how to add
new rest end points. At some point it will also describe the rest.py
structure and implementation.

Adding new rest endpoints
=========================

Add or edit the file interfaces.py at the root of the tracker
directory.

In that file add::

    from roundup.rest import Routing, RestfulInstance, _data_decorator
    from roundup.exceptions import Unauthorised

    class RestfulInstance:

	@Routing.route("/summary2")
	@_data_decorator
	def summary2(self, input):
	    result = { "hello": "world" }
	    return 200, result

will make a new endpoint .../rest/summary2 that you can test with::

    $ curl -X GET .../rest/summary2
    {
        "data": {
            "hello": "world"
        }
    }

Similarly appending this to interfaces.py after summary2::

    # handle more endpoints
        @Routing.route("/data/<:class_name>/@schema", 'GET')
        def get_element_schema(self, class_name, input):
	    result = { "schema": {} }
	    uid = self.db.getuid ()
	    if not self.db.security.hasPermission('View', uid, class_name) :
		raise Unauthorised('Permission to view %s denied' % class_name)

	    class_obj = self.db.getclass(class_name)
	    props = class_obj.getprops(protected=False)
	    schema = result['schema']

	    for prop in props:
		schema[prop] = { "type": repr(class_obj.properties[prop]) }

	    return result

..
  the # comment in the example is needed to preserve indention under Class.

returns some data about the class::

    $ curl -X GET .../rest/data/issue/@schema
    {   
	"schema": {
	    "keyword": {
		"type": "<roundup.hyperdb.Multilink to \"keyword\">"
	    },
	    "title": {
		"type": "<roundup.hyperdb.String>"
	    },
	    "files": {
		"type": "<roundup.hyperdb.Multilink to \"file\">"
	    },
	    "status": {
		"type": "<roundup.hyperdb.Link to \"status\">"
	    }, ...
	}
    }


Adding other endpoints (e.g. to allow an OPTIONS query against
``/data/issue/@schema``) is left as an exercise for the reader.