====================
REST API for Roundup
====================

.. contents::
   :local:

Introduction
------------

After the last 1.6.0 Release, a REST-API developed in 2015 during a Google
Summer of Code (GSOC) by Chau Nguyen, supervised by Ezio Melotti was
integrated. The code was then updated by John Rouillard and Ralf
Schlatterbeck to fix some shortcomings and provide the necessary
functions for a single page web application, e.g. etag support among
others.

Enabling the REST API
---------------------

The REST API can be disabled in the ``[web]`` section of ``config.ini``
via the variable ``enable_rest`` which is ``yes`` by default.

The REST api is reached via the ``/rest/`` endpoint of the tracker URL.


Client API
----------

The top-level REST url ``/rest/`` will display the current version of
the REST API (Version 1 as of this writing) and some links to relevant
endpoints of the API. In the following the ``/rest`` prefix is ommitted
from relative REST-API links for brevety.

Summary
=======

A Summary page can be reached via ``/summary`` via the ``GET`` method.
This is currently hard-coded for the standard tracker schema shipped
with roundup and will display a summary of open issues.

Data
====

All the links mentioned in the following support the http method ``GET``. 
Results of a ``GET`` request will always return the results as a
dictionary with the entry ``data`` referring to the returned data.

The ``/data`` link will display a set of classes of the tracker. All
classes can be reached via ``/data/<classname>`` where ``<classname>``
is replace with the name of the class to query, e.g. ``/data/issue``.
Individual items of a class (e.g. a single issue) can be queried by
giving the issue-id, e.g., ``/data/issue/42``. Individual properties of
an item can be queried by appending the property, e.g.,
``/data/issue/42/title``.

When performing the ``GET`` method on a class (e.g. ``/data/issue``), the
number of items is returned in ``@total_size``. Then a ``collection``
list follows which contains the id and link to the respective item.
This endpoint supports pagination. With the attributes @page_size and
@page_index, pagination is controlled. The @page_size specifies how many
items are displayed at once. The @page_index (which defaults to 1 if not
given) specifies which page number of @page_size items is displayed. If
no @page_size is specified, all items are returned.

Adding the query parameter @verbose=2 to the GET will include the label
property in addition to id and link for the items. This is useful as
documented below in "Searches and selection".

In addition this method supports searching. Search parameters are names
of properties of the given class, e.g., ``status`` for ``issue``. Links
and Multilinks can be specified numerically or symbolically, e.g.,
searching for issues in status ``closed`` can be achieved by searching
for ``status=closed`` or ``status=3`` (provided the ``closed`` status
has ID 3). Note that searching for strings (e.g. the issue title, or a
keyword name) performs a case-insensitive substring search, so searching
for title=Something will find all issues with "Something" or "someThing",
etc. in the title. There is currently no way to perform an exact string
match.

When performing the ``GET`` method on an item (e.g. ``/data/issue/42``), a
``link`` attribute contains the link to the item, ``id`` contains the
id, type contains the class name (e.g. ``issue`` in the example) and an
``etag`` property can be used to detect modifications since the last
query. The individual properties of the item are returned in an
``attributes`` dictionary. The properties returned depend on the
permissions of the account used for the query.
Link and Multilink properties are displayed as a dictionary with a link
and an id property by default. This is controlled by the @verbose
attribute which is set to 1 by default. If set to 0, only the id is
shown for Link and Multilink attributes. If set to 2, the label property
(usually ``name`` e.g. for status) is also put into the dictionary.
Content properties of message and file object are by default also shown
as a dictionary with a sole link attribute. The link is the download
link for the file or message. If @verbose is >= 3, the content property
is shown in json as a (possibly very long) string. Currently the json
serializer cannot handle files not properly utf-8 encoded, so specifying
@verbose=3 for files is currently discouraged.
Note that if the class has a key attribute (like e.g., the 'status'
class in the classic tracker), you can get an individual status by
specifying the key-attribute e.g. ``/data/status/name=closed``. Note
that ``name`` in this example must be the key-attribute of the class.
A short-form (which might no longer be supported in future version of
the API) is to specify only the value, e.g. ``/data/status/closed``.
This short-form only works when you're sure that the key of the class is
not numeric.
The long-form (with ``=``) is different from a query-parameter like
``/data/status?@name=closed`` which would find all stati that have
``closed`` as a substring.

A ``GET`` method on a property (e.g. ``/data/issue/42/title``) returns the
link, an ``@etag``, the type of the property (e.g. "<type str>") the id
of the item and the content of the property in ``data``.

Only class links support the ``POST`` method for creation of new items
of a class, e.g., a new issue via the ``/data/issue`` link. The post
gets a dictionary of keys/values for the new item. It returns the same
parameters as the GET method after successful creation.

All endpoints support an ``OPTIONS`` method for determining which
methods are allowed on a given endpoint.

The method ``PUT`` is allowed on individual items, e.g.
``/data/issue/42`` as well as properties, e.g.,
``/data/issue/42/title``. On success it returns the same parameters as
the respective ``GET`` method. Note that for ``PUT`` an Etag has to be
supplied, either in the request header or as an @etag parameter.

The method ``DELETE`` is allowed on items, e.g., ``/data/issue/42`` and
will retire (mark as deleted) the respective item. On success it will
only return a status code. It is also possible to call ``DELETE`` on a
property of an item, e.g., ``/data/issue/42/nosy`` to delete the nosy
list. The same effect can be achieved with a ``PUT`` request and an
empty new value.

Finally the ``PATCH`` method can be applied to individual items, e.g.,
``/data/issue/42`` and to properties, e.g., ``/data/issue/42/title``.
This method gets an operator ``@op=<method>`` where ``<method>`` is one
of ``add``, ``replace``, ``remove``, only for an item (not for a
property) an additional operator ``action`` is supported. If no operator
is specified, the default is ``replace``. The first three operators are
self explanatory. For an ``action`` operator an ``@action_name`` and
optional ``@action_argsXXX`` parameters have to be supplied. Currently
there are only two actions, neither has args, namely ``retire`` and
``restore``. The ``retire`` action on an item is the same as a
``DELETE`` method, it retires the item. The ``restore`` action is the
inverse of ``retire``, the item is again visible.
On success the returned value is the same as the respective ``GET``
method.

Note that the ``GET`` method on an item (e.g. ``/data/issue/43``)
returns an ETag in the http header *and* the ``@etag`` value. When
modifying the item via ``PUT`` or ``PATCH`` either a ``If-Match`` header
or an ``@etag`` value in the form have to be provided.

sample python client
====================

The client uses the python ``requests`` library for easier interaction
with a REST API supporting JSON encoding::


        >>> import requests
        >>> u = 'http://user:password@tracker.example.com/demo/rest/data/'
        >>> s = requests.session()
        >>> r = s.get(u + 'issue/42/title')
        >>> if r.status_code != 200:
        ...     print("Failed: %s: %s" % (r.status_code, r.reason))
        ...     exit(1)
        >>> print (r.json() ['data']['data']
        TEST Title
        >>> h = {'X-Requested-With': 'rest', 'Referer': 'http://tracker.example.com/demo/'}
        >>> r = s.post (u + 'issue', data = dict (title = 'TEST Issue'), headers=h)
        >>> if not 200 <= r.status_code <= 201:
        ...     print("Failed: %s: %s" % (r.status_code, r.reason))
        ...     exit(1)
        >>> print(r.json())

Retire/Restore::
        >>> r = s.delete (u + 'issue/42')
        >>> print (r.json())
        >>> r = s.get (u + 'issue/42')
        >>> etag = r.headers['ETag']
        >>> print("ETag: %s" % etag)
        >>> etag = r.json()['data']['@etag']
        >>> print("@etag: %s" % etag)
        >>> h = {'If-Match': etag, 'X-Requested-With': 'rest', 'Referer': 'http://tracker.example.com/demo/'}
        >>> d = {'@op:'action', '@action_name':'retire'}
        >>> r = s.patch(u + 'issue/42', data = d, headers = h)
        >>> print(r.json())
        >>> d = {'@op:'action', '@action_name':'restore'}
        >>> r = s.patch(u + 'issue/42', data = d, headers = h)
        >>> print(r.json())

Note the addition of headers for: x-requested-with and referer. This
allows the request to pass the CSRF protection mechanism. You may need
to add Origin if this check is enabled in your tracker's config.ini.


Adding new rest endpoints
=========================

Add or edit the file interfaces.py at the root of the tracker
directory.

In that file add::

    from roundup.rest import Routing, RestfulInstance, _data_decorator
    from roundup.exceptions import Unauthorised

    class RestfulInstance:

	@Routing.route("/summary2")
	@_data_decorator
	def summary2(self, input):
	    result = { "hello": "world" }
	    return 200, result

will make a new endpoint .../rest/summary2 that you can test with::

    $ curl -X GET .../rest/summary2
    {
        "data": {
            "hello": "world"
        }
    }

Similarly appending this to interfaces.py after summary2::

    # handle more endpoints
        @Routing.route("/data/<:class_name>/@schema", 'GET')
        def get_element_schema(self, class_name, input):
	    result = { "schema": {} }
	    uid = self.db.getuid ()
	    if not self.db.security.hasPermission('View', uid, class_name) :
		raise Unauthorised('Permission to view %s denied' % class_name)

	    class_obj = self.db.getclass(class_name)
	    props = class_obj.getprops(protected=False)
	    schema = result['schema']

	    for prop in props:
		schema[prop] = { "type": repr(class_obj.properties[prop]) }

	    return result

..
  the # comment in the example is needed to preserve indention under Class.

returns some data about the class::

    $ curl -X GET .../rest/data/issue/@schema
    {   
	"schema": {
	    "keyword": {
		"type": "<roundup.hyperdb.Multilink to \"keyword\">"
	    },
	    "title": {
		"type": "<roundup.hyperdb.String>"
	    },
	    "files": {
		"type": "<roundup.hyperdb.Multilink to \"file\">"
	    },
	    "status": {
		"type": "<roundup.hyperdb.Link to \"status\">"
	    }, ...
	}
    }


Adding other endpoints (e.g. to allow an OPTIONS query against
``/data/issue/@schema``) is left as an exercise for the reader.

Searches and selection
======================

One difficult interface issue is selection of items from a long list.
Using multi-item selects requires loading a lot of data (e.g. consider
a selection tool to select one or more issues as in the classic
superseder field).

This can be made easier using javascript selection tools like select2,
selectize.js, chosen etc. These tools can query a remote data provider
to get a list of items for the user to select from.

Consider a multi-select box for the superseder property.  Using
selectize.js (and jquery) code similar to::

    $('#superseder').selectize({
	valueField: 'id',
	labelField: 'title',
	searchField: 'title', ...
	load: function(query, callback) {
		if (!query.length) return callback();
		$.ajax({
			url: '.../rest/data/issue?@verbose=2&title='
			    + encodeURIComponent(query),
			type: 'GET',
			error: function() {callback();},
			success: function(res) {
			  callback(res.data.collection);}

Sets up a box that a user can type the word "request" into. Then
selectize.js will use that word to generate an ajax request with the
url: ``.../rest/data/issue?@verbose=2&title=request``

This will return data like::

  {
    "data": {
    "@total_size": 440,
    "collection": [
      {
	  "link": ".../rest/data/issue/8",
	  "id": "8",
	  "title": "Request for Power plugs"
      },
      {
	  "link": ".../rest/data/issue/27",
	  "id": "27",
	  "title": "Request for foo"
      },
  ...

selectize.js will look at these objects (as passed to
callback(res.data.collection)) and create a select list from the each
object showing the user the labelField (title) for each object and
associating each title with the corresponding valueField (id). The
example above has 440 issues returned from a total of 2000
issues. Only 440 had the word "request" somewhere in the title greatly
reducing the amount of data that needed to be transferred.

Similar code can be set up to search a large list of keywords using::

  .../rest/data/keyword?@verbose=2&name=some

which would return: "some keyword" "awesome" "somebody" making
selections for links and multilinks much easier.

Hopefully future enhancements will allow get on a collection to
include other fields. Why do we want this?  Selectize.js can set up
option groups (optgroups) in the select pulldown. So by including
status in the returned data::

      {
	  "link": ".../rest/data/issue/27",
	  "id": "27",
	  "title": "Request for foo",
	  'status": "open"
      },

a select widget like::

  === New ===
  A request
  === Open ===
  Request for bar
  Request for foo

etc. can be generated. Also depending on the javascript library, other
fields can be used for subsearch and sorting.