issue2551328/issue2551264 unneeded next link and total_count incorrect Fix: issue2551328 - REST results show next link if number of results is a multiple of page size. (Found by members of team 3 in the UMass-Boston CS682 Spring 2024 class.) issue2551264 - REST X-Total-Count header and @total_size count incorrect when paginated These issues arose because we retrieved the exact number of rows from the database as requested by the user using the @page_size parameter. With this changeset, we retrieve up to 10 million + 1 rows from the database. If the total number of rows exceeds 10 million, we set the total_count indicators to -1 as an invalid size. (The max number of requested rows (default 10 million +1) can be modified by the admin through interfaces.py.) By retrieving more data than necessary, we can calculate the total count by adding @page_index*@page_size to the number of rows returned by the query. Furthermore, since we return more than @page_size rows, we can determine the existence of a row at @page_size+1 and use that information to determine if a next link should be provided. Previously, a next link was returned if @page_size rows were retrieved. This change does not guarantee that the user will get @page_size rows returned. Access policy filtering occurs after the rows are returned, and discards rows inaccessible by the user. Using the current @page_index/@page_size it would be difficult to have the roundup code refetch data and make sure that a full @page_size set of rows is returned. E.G. @page_size=100 and 5 of them are dropped due to access restrictions. We then fetch 10 items and add items 1-4 and 6 (5 is inaccessible). There is no way to calculate the new database offset at: @page_index*@page_size + 6 from the URL. We would need to add an @page_offset=6 or something. This could work since the client isn't adding 1 to @page_index to get the next page. Thanks to HATEOAS, the client just uses the 'next' url. But I am not going to cross that bridge without a concrete use case. This can also be handled client side by merging a short response with the next response and re-paginating client side. Also added extra index markers to the docs to highlight use of interfaces.py.

import unittest, os, shutil, time

from roundup import hyperdb

from .db_test_base import DBTest, ROTest, SchemaTest, config, setupSchema
from roundup.test import memorydb

from roundup.anypy import strings

class memorydbOpener:
    module = memorydb

    def nuke_database(self):
        # really kill it
        memorydb.db_nuke('')
        self.db = None

    db = None
    def open_database(self, user='admin'):
        if self.db:
            self.db.close()
        self.db = self.module.Database(config, user)
        return self.db

    def setUp(self):
        self.open_database()
        setupSchema(self.db, 1, self.module)

    def tearDown(self):
        if self.db is not None:
            self.db.close()
            self.db = None
        self.nuke_database()

    # nuke and re-create db for restore
    def nukeAndCreate(self):
        self.db.close()
        self.nuke_database()
        self.db = self.module.Database(config, 'admin')
        setupSchema(self.db, 0, self.module)


class memorydbDBTest(memorydbOpener, DBTest, unittest.TestCase):
    pass


class memorydbROTest(memorydbOpener, ROTest, unittest.TestCase):
    def setUp(self):
        self.db = self.module.Database(config)
        setupSchema(self.db, 0, self.module)


class memorydbSchemaTest(memorydbOpener, SchemaTest, unittest.TestCase):
    pass


from .session_common import SessionTest
class memorydbSessionTest(memorydbOpener, SessionTest, unittest.TestCase):
    s2b = lambda x,y: strings.s2b(y)

    def setUp(self):
        self.db = self.module.Database(config, 'admin')
        setupSchema(self.db, 1, self.module)
        self.sessions = self.db.sessions
        self.db.Session = self.sessions
        self.otks = self.db.otks
        self.db.Otk = self.otks

    def get_ts(self):
        return (self.sessions.get('random_session', '__timestamp'),)

    def testDbType(self):
        self.assertIn("memorydb", repr(self.db))
        self.assertIn("{}", repr(self.db.Session))

# vim: set filetype=python ts=4 sw=4 et si