issue2551328/issue2551264 unneeded next link and total_count incorrect Fix: issue2551328 - REST results show next link if number of results is a multiple of page size. (Found by members of team 3 in the UMass-Boston CS682 Spring 2024 class.) issue2551264 - REST X-Total-Count header and @total_size count incorrect when paginated These issues arose because we retrieved the exact number of rows from the database as requested by the user using the @page_size parameter. With this changeset, we retrieve up to 10 million + 1 rows from the database. If the total number of rows exceeds 10 million, we set the total_count indicators to -1 as an invalid size. (The max number of requested rows (default 10 million +1) can be modified by the admin through interfaces.py.) By retrieving more data than necessary, we can calculate the total count by adding @page_index*@page_size to the number of rows returned by the query. Furthermore, since we return more than @page_size rows, we can determine the existence of a row at @page_size+1 and use that information to determine if a next link should be provided. Previously, a next link was returned if @page_size rows were retrieved. This change does not guarantee that the user will get @page_size rows returned. Access policy filtering occurs after the rows are returned, and discards rows inaccessible by the user. Using the current @page_index/@page_size it would be difficult to have the roundup code refetch data and make sure that a full @page_size set of rows is returned. E.G. @page_size=100 and 5 of them are dropped due to access restrictions. We then fetch 10 items and add items 1-4 and 6 (5 is inaccessible). There is no way to calculate the new database offset at: @page_index*@page_size + 6 from the URL. We would need to add an @page_offset=6 or something. This could work since the client isn't adding 1 to @page_index to get the next page. Thanks to HATEOAS, the client just uses the 'next' url. But I am not going to cross that bridge without a concrete use case. This can also be handled client side by merging a short response with the next response and re-paginating client side. Also added extra index markers to the docs to highlight use of interfaces.py.

--- a/doc/admin_guide.txt	Sun Mar 31 01:49:07 2024 -0400
+++ b/doc/admin_guide.txt	Mon Apr 01 09:57:16 2024 -0400
@@ -259,6 +259,8 @@
 gzip support. For brotli or zstd you will need to install packages. See
 the `installation documentation`_ for details.
 
+.. index:: single: interfaces.py; configuring http compression
+
 Some assets will not be compressed on the fly. Assets with mime types
 of "image/png" or "image/jpeg" will not be compressed. You
 can add mime types to the list by using ``interfaces.py`` as discussed
@@ -320,6 +322,31 @@
 mechanism allows the admin to allow use of brotli and zstd for
 dynamic content, but not for static content.
 
+.. index:: single: interfaces.py; setting REST maximum result limit
+
+Configuring REST Maximum Result Limit
+=====================================
+
+To prevent denial of service (DOS) and limit user wait time for an
+unbounded request, the REST endpoint has a maximum limit on the number
+of rows that can be returned. By default, this is set to 10 million.
+This setting applies to all users of the REST interface. If you want
+to change this limit, you can add the following code to the
+``interfaces.py`` file in your tracker::
+
+    # change max response rows
+    from roundup.rest import RestfulInstance
+    RestfulInstance.max_response_row_size = 26
+
+This code will set the maximum number of rows to 25 (one less than the
+value). Note that this setting is rarely used and is not available in
+the tracker's ``config.ini`` file. Setting it through this mechanism
+allows you to enter a string or number that may break Roundup, such as
+"asdf" or 0. In general, it is recommended to keep the limit at its
+default value. However, this option is available for cases when a
+request requires more than 10 million rows and pagination using
+``@page_index`` and ``@page_size=9999999`` is not possible.
+
 Adding a Web Content Security Policy (CSP)
 ==========================================