Mercurial > p > roundup > code
comparison doc/admin_guide.txt @ 7853:03c1b7ae3a68
issue2551328/issue2551264 unneeded next link and total_count incorrect
Fix: issue2551328 - REST results show next link if number of
results is a multiple of page size. (Found by members of
team 3 in the UMass-Boston CS682 Spring 2024 class.)
issue2551264 - REST X-Total-Count header and @total_size
count incorrect when paginated
These issues arose because we retrieved the exact number of rows
from the database as requested by the user using the @page_size
parameter. With this changeset, we retrieve up to 10 million + 1
rows from the database. If the total number of rows exceeds 10
million, we set the total_count indicators to -1 as an invalid
size. (The max number of requested rows (default 10 million +1)
can be modified by the admin through interfaces.py.)
By retrieving more data than necessary, we can calculate the
total count by adding @page_index*@page_size to the number of
rows returned by the query.
Furthermore, since we return more than @page_size rows, we can
determine the existence of a row at @page_size+1 and use that
information to determine if a next link should be
provided. Previously, a next link was returned if @page_size rows
were retrieved.
This change does not guarantee that the user will get @page_size
rows returned. Access policy filtering occurs after the rows are
returned, and discards rows inaccessible by the user.
Using the current @page_index/@page_size it would be difficult to
have the roundup code refetch data and make sure that a full
@page_size set of rows is returned. E.G. @page_size=100 and 5 of
them are dropped due to access restrictions. We then fetch 10
items and add items 1-4 and 6 (5 is inaccessible). There is no
way to calculate the new database offset at:
@page_index*@page_size + 6 from the URL. We would need to add an
@page_offset=6 or something.
This could work since the client isn't adding 1 to @page_index to
get the next page. Thanks to HATEOAS, the client just uses the
'next' url. But I am not going to cross that bridge without a
concrete use case.
This can also be handled client side by merging a short response
with the next response and re-paginating client side.
Also added extra index markers to the docs to highlight use of
interfaces.py.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Mon, 01 Apr 2024 09:57:16 -0400 |
| parents | 8bdf0484215c |
| children | 2946354d6ccd |
comparison
equal
deleted
inserted
replaced
| 7852:9e686e3f39c1 | 7853:03c1b7ae3a68 |
|---|---|
| 257 your proxy (e.g. nginx or apache) or wsgi server (uwsgi) is configured | 257 your proxy (e.g. nginx or apache) or wsgi server (uwsgi) is configured |
| 258 to compress responses on the fly. The python standard library includes | 258 to compress responses on the fly. The python standard library includes |
| 259 gzip support. For brotli or zstd you will need to install packages. See | 259 gzip support. For brotli or zstd you will need to install packages. See |
| 260 the `installation documentation`_ for details. | 260 the `installation documentation`_ for details. |
| 261 | 261 |
| 262 .. index:: single: interfaces.py; configuring http compression | |
| 263 | |
| 262 Some assets will not be compressed on the fly. Assets with mime types | 264 Some assets will not be compressed on the fly. Assets with mime types |
| 263 of "image/png" or "image/jpeg" will not be compressed. You | 265 of "image/png" or "image/jpeg" will not be compressed. You |
| 264 can add mime types to the list by using ``interfaces.py`` as discussed | 266 can add mime types to the list by using ``interfaces.py`` as discussed |
| 265 in the `customisation documentation`_. As an example adding:: | 267 in the `customisation documentation`_. As an example adding:: |
| 266 | 268 |
| 317 files, the data will be compressed dynamically (on the fly) using | 319 files, the data will be compressed dynamically (on the fly) using |
| 318 brotli. If there is a precompressed gzip file present the client will | 320 brotli. If there is a precompressed gzip file present the client will |
| 319 get the gzip version and not a brotli compressed version. This | 321 get the gzip version and not a brotli compressed version. This |
| 320 mechanism allows the admin to allow use of brotli and zstd for | 322 mechanism allows the admin to allow use of brotli and zstd for |
| 321 dynamic content, but not for static content. | 323 dynamic content, but not for static content. |
| 324 | |
| 325 .. index:: single: interfaces.py; setting REST maximum result limit | |
| 326 | |
| 327 Configuring REST Maximum Result Limit | |
| 328 ===================================== | |
| 329 | |
| 330 To prevent denial of service (DOS) and limit user wait time for an | |
| 331 unbounded request, the REST endpoint has a maximum limit on the number | |
| 332 of rows that can be returned. By default, this is set to 10 million. | |
| 333 This setting applies to all users of the REST interface. If you want | |
| 334 to change this limit, you can add the following code to the | |
| 335 ``interfaces.py`` file in your tracker:: | |
| 336 | |
| 337 # change max response rows | |
| 338 from roundup.rest import RestfulInstance | |
| 339 RestfulInstance.max_response_row_size = 26 | |
| 340 | |
| 341 This code will set the maximum number of rows to 25 (one less than the | |
| 342 value). Note that this setting is rarely used and is not available in | |
| 343 the tracker's ``config.ini`` file. Setting it through this mechanism | |
| 344 allows you to enter a string or number that may break Roundup, such as | |
| 345 "asdf" or 0. In general, it is recommended to keep the limit at its | |
| 346 default value. However, this option is available for cases when a | |
| 347 request requires more than 10 million rows and pagination using | |
| 348 ``@page_index`` and ``@page_size=9999999`` is not possible. | |
| 322 | 349 |
| 323 Adding a Web Content Security Policy (CSP) | 350 Adding a Web Content Security Policy (CSP) |
| 324 ========================================== | 351 ========================================== |
| 325 | 352 |
| 326 A Content Security Policy (`CSP`_) adds a layer of security to | 353 A Content Security Policy (`CSP`_) adds a layer of security to |