Mercurial Repository: p/roundup/code: doc/rest.txt comparison

comparison doc/rest.txt @ 7853:03c1b7ae3a68

issue2551328/issue2551264 unneeded next link and total_count incorrect Fix: issue2551328 - REST results show next link if number of results is a multiple of page size. (Found by members of team 3 in the UMass-Boston CS682 Spring 2024 class.) issue2551264 - REST X-Total-Count header and @total_size count incorrect when paginated These issues arose because we retrieved the exact number of rows from the database as requested by the user using the @page_size parameter. With this changeset, we retrieve up to 10 million + 1 rows from the database. If the total number of rows exceeds 10 million, we set the total_count indicators to -1 as an invalid size. (The max number of requested rows (default 10 million +1) can be modified by the admin through interfaces.py.) By retrieving more data than necessary, we can calculate the total count by adding @page_index*@page_size to the number of rows returned by the query. Furthermore, since we return more than @page_size rows, we can determine the existence of a row at @page_size+1 and use that information to determine if a next link should be provided. Previously, a next link was returned if @page_size rows were retrieved. This change does not guarantee that the user will get @page_size rows returned. Access policy filtering occurs after the rows are returned, and discards rows inaccessible by the user. Using the current @page_index/@page_size it would be difficult to have the roundup code refetch data and make sure that a full @page_size set of rows is returned. E.G. @page_size=100 and 5 of them are dropped due to access restrictions. We then fetch 10 items and add items 1-4 and 6 (5 is inaccessible). There is no way to calculate the new database offset at: @page_index*@page_size + 6 from the URL. We would need to add an @page_offset=6 or something. This could work since the client isn't adding 1 to @page_index to get the next page. Thanks to HATEOAS, the client just uses the 'next' url. But I am not going to cross that bridge without a concrete use case. This can also be handled client side by merging a short response with the next response and re-paginating client side. Also added extra index markers to the docs to highlight use of interfaces.py.

author	John Rouillard <rouilj@ieee.org>
date	Mon, 01 Apr 2024 09:57:16 -0400
parents	ee0062411160
children	171ff2e487df

comparison

equal deleted inserted replaced

-:9e686e3f39c1
+:03c1b7ae3a68
 Details are in the sections below.
 /data/\ *class* Collection
 --------------------------
-When performing the ``GET`` method on a class (e.g. ``/data/issue``),
+When you use the ``GET`` method on a class (like ``/data/issue``), the
-the ``data`` object includes the number of items available in
+``data`` will include the number of available items in
-``@total_size``. A a ``collection`` list follows which contains the id
+``@total_size``. If the size exceeds the administrative limit (which
-and link to the respective item.  For example a get on
+is 10 million by default), ``@total_size`` will be set to ``-1``. To
-https://.../rest/data/issue returns::
+navigate to the last page of results, you can use the ``next`` links
+or increment ``@page_index`` until the result does not include a
+``next`` ``@link`` or ``@total_size`` is not ``-1``. The value of the
+HTTP header ``X-Count-Total`` is the same as ``@total_size``.
+A ``collection`` list contains the id and link to the
+respective item.  For example a get on https://.../rest/data/issue
+returns::
 {
 "data": {
 "collection": [
 {
 }
 Collection endpoints support a number of features as seen in the next
 sections.
-A server may implement a default maximum number of items in the
+Having an empty ``collection`` does not mean next next link will not
-collection.  This can be used to prevent denial of service (DOS).  As
+return more data. The row limit is applied when the query is made to
-a result all clients must be programmed to expect pagination
+the database. The result set is then filtered, removing rows that the
-decorations in the response. See the section on pagination below for
+user does not have permission to access. So it is possible to have no
-details.
+data items on a page because the user does not have access to them. If
+you use ``@page_size`` near the administrative limit, you may receive
+fewer rows than requested. However, this does not mean you are out of
+data.
+All clients must be programmed to expect pagination decorations in the
+response. See the section on pagination below for details.
 Searching
 ~~~~~~~~~
 Searching is done by adding roundup field names and values as query
 ``name`` using ``https://.../rest/data/keyword?name=Foo`` (note
 searching keyword class not issue class) will return matches for
 ``Foo``, ``foobar``, ``foo taz`` etc.
 In all cases the field ``@total_size`` is reported which is the total
-number of items available if you were to retrieve all of them.
+number of items available if you were to retrieve all of them. See
+more details in the parent section about ``@total_size`` and when it
+can return ``-1``.
 Other data types: Date, Interval, Integer, Number need examples and may
 need work to allow range searches. Full text search (e.g. over the
 body of a msg) is a work in progress.
 Does not support PUT, DELETE or PATCH.
 /data/\ *class*/\ *id* item
 ---------------------------
-When performing the ``GET`` method on an item
+When you use the ``GET`` method on an item
 (e.g. ``/data/issue/42``), a ``link`` attribute contains the link to
 the item, ``id`` contains the id, ``type`` contains the class name
 (e.g. ``issue`` in the example) and an ``etag`` property can be used
 to detect modifications since the last query.

Mercurial > p > roundup > code

comparison doc/rest.txt @ 7853:03c1b7ae3a68