| changeset | 1f8492d68aca |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | bug: using 'null' value for attributes causes error. In rest.py, filter out any attributes that are set to 'None'. GET on an endpoint can return 'null' values when the attribute is unset. E.G. for a user: { "address": "baddy@example.com", "alternate_addresses": null, "last_login": "2026-03-18.05:57:09", "organisation": null, "password": null, "phone": null, "queries": [], "realname": "Fred Jones", "roles": "User", "timezone": null, "username": "badeggs" } But this json can not be submitted to a PUT or POST endpoint. The validators for passwords, strings, integers etc. don't expect a None value. This change handles attributes with "null" (None) values in json objects by filtering them from the python object before processing. The null value can't be used to unset an attribute via PUT or POST. The 'remove' action using the PATCH verb can unset the value. Also there appears to be some missing checks in the back_anydbm and rdbms_common files for the password type. All the other types have a check: value is not None and not isinstance(.....) but passwords only have the 'not isinstance(....)' part. Not sure why this was the case. Looking at commit history didn't make me think it was intentional. |
| files |
| changeset | fed0f839c260 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix: replace except: with except Exception: (by haosenwang1018@github) Remove bare 'except:' statements replace with 'except Exception'. In roundup_xmlrpc_server.py I changed Exception to BaseException so the database gets closed on signals as well. The rest of the changes do not look like they affect data integrity and are commited as supplied. |
| files |
| changeset | 370689471a08 |
|---|---|
| branch | issue2550923_computed_property |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | merge from default branch accumulated changes since Nov 2023 |
| files |
| changeset | a81a3cd067fa |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Generate savepoint only if necessary Now some methods got an additional 'allow_abort' parameter. By default this is True. When False the postgres backend generates a savepoint. The methods are called with allow_abort=False from some of the cgi methods which can produce a traceback when called with data from the web-interface. |
| files |
| changeset | 24549122f9b1 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Factor common code to hyperdb/roundupdb There was some common copied code in backends/back_anydbm.py and backends/rdbms_common.py. This is now moved to hyperdb.py and roundupdb.py, respectively (the FileClass lives in hyperdb.py while the IssueClass is in roundupdb.py) |
| files |
| changeset | 82a26ea1afdf |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | issue2551376: Fix tracebacks in item templates |
| files |
| changeset | 741ea8a86012 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix: issue2551374. Error handling for filter expressions. Errors in filter expressions are now reported. The UI needs some work but even the current code is helpful when debugging filter expressions. mlink_expr: defines/raises ExpressionError(error string template, context=dict()) raises ExpressionError when it detects errors when popping arguments off stack raises ExpressionError when more than one element left on the stack before returning also ruff fix to group boolean expression with parens back_anydbm.py, rdbms_common.py: catches ExpressionError, augments context with class and attribute being searched. raises the exception for both link and multilink relations client.py catches ExpressionError returning a basic error page. The page is a dead end. There are no links or anything for the user to move forward. The user has to go back, possibly refresh the page (because the submit button may be disalbled) re-enter the query and try again. This needs to be improved. test_liveserver.py test the error page generated by client.py db_test_base unit tests for filter with too few arguments, too many arguments, check all repr and str formats. |
| files |
| changeset | a9b136565838 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | feat: issue1525113 - notation to filter by logged-in user At long last (almost 18 years) this patch lands. It allows sharing of queries that want to use the currently logged in user (i.e. I or me). By replacing an id number for the user by '@current_user' in the query you can share the query for "my issues" where 'my' is the logged in user not the person who created the query. Updated the templates to use this. Updated upgrading.py for directions on using it. RDBMS and anydbm both work. Also expressions using it (e.g. not @current_user) work and are tested. Test code done. I am not sure what the change to templating.py does. I am following the original patch and have built a test case to hit the if clause. But the rest of the test doesn't actualy provide the props I need. If I knew what that code was supposed to do there I would create a real test. |
| files |
| changeset | 8b31893f5930 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551115/issue2551282 - utf8mb4 support in roundup Fix issues with utf8 support in Roundup. By default using: utf8mb4 charset utf8mb4_unicode_ci collation (case insensitive) utf8mb4_0900_ci collation (case sensitive) which are settable from config.ini. Sadly I couldn't come up with a way to mange these from one parameter. Doing a compatibility lookup table would have increased the maintenance burden and have me chasing MySQL changes. So I opted for the easy path and have the admins (with more MySQL experience) make the choices. Conversion directions added to upgrading.txt. I don't have any good testing for this. I was able to generate utf8/utf8mb3 tables and load a little data and convert. However this is a poor substitute for a conversion on a working tracker 8-(. |
| files |
| changeset | 216662fbaaee |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix(i18n): fix incorrect lookup of some translations The code had: _("some term %s here" % term) this extracts the template, but looks up the string with %s replaced. So the translation is broken. Changed to: _("some term %s here") % term which looks up the template and substitutes in the translation of the template. Found by ruff INT ruleset. |
| files |
| changeset | 4af0d235b570 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | feat(db): support using postgresql service connection file Add new service rdbms config option to set the service name to be used with a postgresql service connection file. This can be done using the PGSERVICE environment variable for a single instance tracker server. For a multi-instance server this per-tracker config option is needed. Note that settings (host, user, (db)name...) in config.ini file will override the service connection file setting. Also setting PGSERVICE and service will use the service setting. |
| files |
| changeset | 78585199552a |
|---|---|
| branch | issue2550923_computed_property |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | merge default trunk into branch |
| files |
| changeset | 6f841a55eabf |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | fix: database error handling during import of a non-user item The code to handle the case of retired and active users causing errors during import can get called on other db errors as well (e.g. out of memory in postgresql). The code that trys to detect/repair the out of order user case hard coded the db.user class. This crashed when a db error occurred when loading another data object (e.g. msg). Fix the crash by using the proper db object class for the object class being loaded. Credit to Norbert Schlemmer for finding this. |
| files |
| changeset | 82bbb95e5690 |
|---|---|
| branch | issue2550923_computed_property |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | merge from tip into issue2550923_computed_property |
| files |
| changeset | 506c86823abb |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Add config argument to more password.Password invocations. The work done to allow password_pbkdf2_default_rounds to be overridden for testing requires that calls to password.Password include a config argument. This was needed because using the real value more than quadrupled testing runtime. However there are still a few places where config was not being set when Password was called. I think this fixes all of the ones that are called from a function that have access to a db.config object. The remaining ones all call Password(encrypted=x). This results in Password.unpack() being called. If x is not a propertly formatted password string ("{scheme}...", it calls encodePassword. It then should end up raising the ConfigNotSet exception. This is probably what we want as it means the shape of "x" is not correct. I don't understand why Password.unpack() attempts to encrypt the value of encrypted if it doesn't match the right form. According to codecov, this encryption branch is being used, so somewhere x is of the wrong form. Hmmm.... |
| files |
| changeset | ca90f7270cd4 |
|---|---|
| branch | issue2550923_computed_property |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | merge from main tip. This should fix test failure in Markdown2TestCase.test_string_markdown_code_block_attribute by merging html diff method used in tests. |
| files |
| changeset | f524ddc27af8 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | replace for a,b in x.items() with for a in x.keys() where the value from items is not used. Recommended refactor from Tom Ekberg after seeing flake8 fixes for unused loop variables. |
| files |
| changeset | 05d81de0d92d |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | in clear() clear journal tables as well. Match clear in back_anydbm. |
| files |
| changeset | 98d72d4bb489 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | flake8 fixes Added str type as alias for missing unicode type in python3. Replace type(x) == type(y) with isinstance(x, (type_of_y, or other type)) which is usually str or unicode (hence need for unicode def). Used De Morgan's to convert things like: type(value) != type('') and type(value) != type(u'') into not isinstance(value, (str, unicode)) a couple of variable name replacements. |
| files |
| changeset | cb2ed1e8c852 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Change method for settin indexer; have test_livetest for pg cleanup Add code to defer opening the indexer only if indexer is native-fts. See if this fixes the sqlite OperationalError. Also under python 2.7 (only), the db from test_livetracker when using postgres FTS didn't empty the db. This caused the following test_postgres.py test to fail. Why it only showed up on 2.7 and not any of the 3.x releases is a mystery. |
| files |
| changeset | 9ff091537f43 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | postgresql native-fts; more indexer tests 1) Make postgresql native-fts actually work. 2) Add simple stopword filtering to sqlite native-fts indexer. 3) Add more tests for indexer_common get_indexer Details: 1) roundup/backends/indexer_postgresql_fts.py: ignore ValueError raised if we try to index a string with a null character in it. This could happen due to an incorrect text/ mime type on a file that has nulls in it. Replace ValueError raised by postgresql with customized IndexerQueryError if a search string has a null in it. roundup/backends/rdbms_common.py: Make postgresql native-fts work. When specified it was using using whatever was returned from get_indexer(). However loading the native-fts indexer backend failed because there was no connection to the postgresql database when this call was made. Simple solution, move the call after the open_connection call in Database::__init__(). However the open_connection call creates the schema for the database if it is not there. The schema builds tables for indexer=native type indexing. As part of the build it looks at the indexer to see the min/max size of the indexed tokens. No indexer define, we get a crash. So it's a a chicken/egg issue. I solved it by setting the indexer to the Indexer from indexer_common which has the min/max token size info. I also added a no-op save_indexer to this Indexer class. I claim save_indexer() isn't needed as a commit() on the db does all the saving required. Then after open_connection is called, I call get_indexer to retrieve the correct indexer and indexer_postgresql_fts woks since the conn connection property is defined. roundup/backends/indexer_common.py: add save_index() method for indexer. It does nothing but is needed in rdbms backends during schema initialization. 2) roundup/backends/indexer_sqlite_fts.py: when this indexer is used, the indexer test in DBTest on the word "the" fail. This is due to missing stopword filtering. Implement basic stopword filtering for bare stopwords (like 'the') to make the test pass. Note: this indexer is not currently automatically run by the CI suite, it was found during manual testing. However there is a FIXME to extract the indexer tests from DBTest and run it using this backend. roundup/configuration.py, roundup/doc/admin_guide.txt: update doc on stopword use for sqlite native-fts. test/db_test_base.py: DBTest::testStringBinary creates a file with nulls in it. It was breaking postgresql with native-fts indexer. Changed test to assign mime type application/octet-stream that prevents it from being processed by any text search indexer. add test to exclude indexer searching in specific props. This code path was untested before. test/test_indexer.py: add test to call find with no words. Untested code path. add test to index and find a string with a null \x00 byte. it was tested inadvertently by testStringBinary but this makes it explicit and moves it to indexer testing. (one version each for: generic, postgresql and mysql) Renamed Get_IndexerAutoSelectTest to Get_IndexerTest and renamed autoselect tests to include autoselect. Added tests for an invalid indexer and using native-fts with anydbm (unsupported combo) to make sure the code does something useful if the validation in configuration.py is broken. test/test_liveserver.py: add test to load an issue add test using text search (fts) to find the issue add tests to find issue using postgresql native-fts test/test_postgresql.py, test/test_sqlite.py: added explanation on how to setup integration test using native-fts. added code to clean up test environment if native-fts test is run. |
| files |
| changeset | bdd28b244839 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | - issue2551223 - fix timestamp truncation in mysql and postgresql The data types used to represent timestamps in pg and mysql for ephemeral tables: sessions and otks don't have enough signifcant digits to work. As a result the timestamps are rounduped (up/down) rsuling in the stored timestamp being 2 minutes (pg) or 2-3 hours(mysql) off from what it should be. Modify db schema to use a numeric type that preserves more significant figures. Implement schema upgrade. Document need for upgrade in upgrading.txt. Write tests for schema upgrade. Implement test for updateTimestamp method on BasicDatabase that showed this issue in the first place. Write overrides for test for anydbm/memorydb which store timestamp properly or not at all. |
| files |
| changeset | 408fd477761f |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Add i18n object to roundupdb.Database This makes the i18n object accessible everywhere (including in detectors where localized error messages were impossible). See issue2551184 |
| files |
| changeset | e1588ae185dc |
|---|---|
| branch | issue2550923_computed_property |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | merge from default branch. Fix travis.ci so CI builds don't error out |
| files |
| changeset | 39189dd94f2c |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551189 - increase size of words in full text index. Increased indexed word maxlength to 50 DB migration code is written and tests work. Restructured some tests to allow for code reuse. Docs. If this passes CI without errors 2551189 should be done. However, testing on my system generates errors. Encoding (indexer unicode russian unicode string invalid) and collation errors (utf8_bin not valid) when running under python2. No issues with python3 and I haven't changed code that should cause these since the last successful build in CI. So if this fails in CI we will have more checkins. |
| files |
| changeset | 91ab3e0ffcd0 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Summary: Add test cases for sqlite fts Add support for using the FTS5 full text query engine for sqlite. Also stubbed out some sections for adding postgresql FTS support as well. Added nee indexer type native-fts. It is not selected by default. The indexer=native is used if no indexer is set. This prevents an upgrade from seeming to wipe out the native index if upgraded and indexer=native is not explicitly set. Docs updated. Also changed section headers to sentence case for the current release notes. Indexing backend can control if the full text search phrase is broken into a list of words or passed intact. For backends with query languages (sqlite and can be enabled for whoosh and xapian) we do not want the phrase "tokenized" on whitespace. This also updates the rdbms database version to version 7 to add FTS table. I will be using the same version when I add postgresql. If somebody runs this version on postgresql, they will have to manually add the fts tables for postgresql if they want to use it. Added a new renderError method to client. This allows errors to be reported still using page.html rather than raw html. It also supports templates for any error code. If no template for the error code (e.g. 400) is found, the error in raw html with no page frame is shown. New IndexerQueryError exception to pass back message about query syntax errors. |
| files |
| changeset | 30358e334232 |
|---|---|
| branch | issue2550923_computed_property |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | merge trunk changes into this branch |
| files |
| changeset | c1d3fbcdbfbd |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551142 - Import of retired node ... unique constraint failure. Title: Import of retired node with username after active node fails with unique constraint failure. More fixes needed for mysql and postgresql. mysql: add unique constraint for (keyvalue, __retired__) when creating class in the database. On schema change if class is changed, remove the unique constraint too. upgrade version of rdbms database from 5 to 6 to add constraint to all version 5 databases that were created as version 5 and didn't get the unique constraint. Make no changes on version 5 databases upgraded from version 4, the upgrade process to 5 added the constraint. Make no changes to other databases (sqlite, postgres) during upgrade from version 5 to 6. postgres: Handle the exception raised on unique constraint violation. The exception invalidates the database connection so it can't be used to recover from the exception. Added two new database methods: checkpoint_data - performs a db.commit under postgres does nothing on other backends restore_connection_on_error - does a db.rollback on postgres, does nothing on other backends with the rollback() done on the connection I can use the database connection to fixup the import that failed on the unique constraint. This makes postgres slower but without the commit after every imported object, the rollback will delete all the entries done up to this point. Trying to figure out how to make the caller do_import batch and recover from this failure is beyond me. Also dismissed having to process the export csv file before importing. Pushing that onto a user just seems wrong. Also since import/export isn't frequently done the lack of surprise on having a failing import and reduced load/frustration for the user seems worth it. Also the import can be run in verbose mode where it prints out a row as it is processed, so it may take a while, ut the user can get feedback. db_test-base.py: add test for upgrade from 5 to 6. |
| files |
| changeset | ada1edcc9132 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | issue2551142 - Import ... unique constraint failure. Full title: Import of retired node with username after active node fails with unique constraint failure. Fix this in two ways: 1) sort export on keyname, retired status so that retired nodes for a given keyname are before the acive node in the export file. This stops generating a broken export. 2) handle importing a broken export by deactivating/fixing up/clearing the active record's unique index entry temporarily. Redo the import of the retired node and resetting the active record to active. The fixup changes the unique index (keyvalue, __retired__) from (keyvalue, 0) to (keyvalue, -1). Then it retries the failed import of a retired record with keyvalue. I use -1 in case something goes wrong, It makes the record stand out in the database allowing hand recovery if needed. Rather than using -1 I could just use the id of the record like a normal retirement does. If the retry of the import fails (raises exception), reset the active record from -1 back to 0 and raise the exception. If it succeeds, reset the active record from -1 back to 0 and continue the import process. Reset __retired__ from -1 to 0 on every import. I don't think the performance loss from resetting on every exception matters as there should be very few exceptions. Also this makes the code more understandable. There is no reason to leave the -1 value in place and do a bulk rest of -1 to 0 after the class csv file is loaded. Also if a fixup is needed it is logged at level info with the rest of the database logging. Also success of the fixup is logged. Fixup failure generates a propagated exception. |
| files |
| changeset | dbacf6bf2a2f |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Implement expressions for Link properties |
| files |
| changeset | 7b1b6dffc7ed |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Fix searching+sorting for Link properties Note that this turns off recursive pre-populating the cache in filter_iter, we only pre-populate the top-level element. Otherwise this interacts with searching producing duplicates. |
| files |
| changeset | 9957d8d10783 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Tests and bug-fix for issue2551119 .. and some other failing tests I came up with when trying to reproduce the problem of the issue. |
| files |
| changeset | 619807d9a2df |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Make rev multilink for Link work Fix searching of retired items. Fix implementation for anydbm. |
| files |
| changeset | 8bc5faeb7677 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Make rev multilink expressions work for anydbm |
| files |
| changeset | 5ce995c33eee |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Make reverse multilink expressions work for mysql |
| files |
| changeset | f3fcd6628c0c |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Allow '-1' (empty) in multilink expression |
| files |
| changeset | 75a53956cf13 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Multilink expressions with simple "or" .. also when searching for empty multilinks with '-1' as part of the list of IDs. |
| files |
| changeset | 8baf81d1cfc1 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Make multilink expressions work with mysql |
| files |
| changeset | 99455aeec1ae |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Fix dropping of multilinks on schema change Special case when a previously materialized multilink (one with an existing table) is replaced with a rev_multilink property from another table. In that case the existing multilink tables was not dropped. |
| files |
| changeset | 6a6b4651be1f |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Use server-side cursor for postgres in some cases In filter, filter_iter, and _materialize_multilinks, use named cursor with postgresql. This turns of client-side cursor handling and avoids *large* roundup process (or wsgi process) in case of large results. Fixes issue2551114. |
| files |
| changeset | 1e5ed659e8ca |
|---|---|
| branch | issue2550923_computed_property |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Initial implementation of Computed property It supports query/display in html, rest and xml interfaces. You can specify a cache parameter, but using it raises NotImplementedError. It does not support: search, sort or grouping by the computed field. Checking in on a branch to get more eyeballs on it and maybe some people to help. |
| files |
| changeset | 6834bb5473da |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Summary: Constrain format of classname and document Define the format of a valid classname. Updated design doc, function doc and glossary to document format. Error message for case where we have a redefinition of a classname now says why ValueError is raised since we are raising ValueError for incorrect classname format as well. Tests for all cases including verification of cause for ValueError exceptions. |
| files |
| changeset | 2492e2e17371 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Change text from no such <class> node <node id> to drop "node" This brings it in line with how same error is expressed in anydbm. Allows me to test more easily with one pattern to search for regardless of backend. |
| files |
| changeset | a701c9c81597 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Fix rev_multilink properties search/retrieval The code now only returns live (non-retired) items. Since for reverse multilinks the Link/Multilink property in the retired item cannot be changed, we now only return non-retired items in search (filter) and retrieve (get). |
| files |
| changeset | 3c52093d7490 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Fix string conversion from database The sql_to_hyperdb_value table would convert strings coming from the database using "str". This is now changed to "us2s", converting unicode strings coming from the database to utf-8. This happens only for python2, for python3 the database interface uses unicode (aka str in python3). This fixes the behaviour for mysql if utf8 encoding is chosen explicitly, it seems in that case the database interface returns values as python2 unicode objects. |
| files |
| changeset | ba0cfc1a87c9 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Fix Database.find for rev_multilink |
| files |
| changeset | ff059afae50a |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Make 'find' work for rev_multilink properties |
| files |
| changeset | 8497bf3f23a1 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Allow to define reverse Multilinks Now it's possible to specify a rev_multilink parameter when creating Link or Multilink properties. The parameter takes a property name to be inserted into the linked-to class. It allows to navigate from the other side of the link as if it where a forward Multilink using the existing data structures. |
| files |
| changeset | e7cb0147e6fe |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Make format strings use named placeholders rather than %s/%r Fixing some issues reported while updating roundup.pot to prep for 2.0.0 beta releases. Still more to do but.... |
| files |
| changeset | e6073c2291c6 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Better Date filtering Allow filtering by multiple date ranges or empty date. Date ranges are separated by comma, an empty date is represented by '-'. |
| files |
| changeset | 1ec4aa670b0c |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Flake8 whitespace; remove imports; type()!=type() -> isinstance() Removed imports sys errno. Replaced type(value) != type(''): isinstance(value, type('')) multiple instances. |
| files |
| changeset | 3175bb92ca28 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Cleanups for bandit Set bandit to ignore use of md5. Treat schema from database as trusted to eval. We wrote it based on the schema.py file. Replace some bare except: with a proper exception. mode 775 for index directory is correct. Allows indices to be written by email and web interfaces that run as different users but can have the roundup group in common. |
| files |
| changeset | acc4a128ab9b |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Backwards-compatible signature of _proptree method |
| files |
| changeset | 16e1255b16cf |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Implement limit and offset for filter These map to the corresponding SQL parameters for the SQL backends. |
| files |
| changeset | ee2e8f8d6648 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Ralf Schlatterbeck <rsc@runtux.com> |
| description | Implement exact string search .. in the 'filter' method of hyperdb.Class (and the corresponding backend implementations). |
| files |
| changeset | 936275dfe1fa |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Try to fix: DeprecationWarning: invalid escape sequence \d DeprecationWarning: invalid escape sequence \s DeprecationWarning: invalid escape sequence \) Strings under python 3 are unicode strings rather then "regular" strings as under python 2. So all regexps need to be raw strings. We will see how many I fixed and if I broke any. |
| files |
| changeset | 6923225fd781 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | John Rouillard <rouilj@ieee.org> |
| description | Handle UnicodeDecodeError in file class when file contents are not text (e.g. jpg). This shows up in python3 due to conversion from bytes to string. This allows binary file retrieval via the rest interface to complete. |
| files |
| changeset | 1a0498c1ed90 |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Joseph Myers <jsm@polyomino.org.uk> |
| description | Avoid errors indexing binary uploads with Python 3. If you upload a binary file for a FileClass whose content property is set to be indexed (the default), an error of the form "'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte" can occur when the code attempts to index the content of that file. (This error is after the creation of the file, and any issue etc. created at the same time, has been committed; the page returned gives the impression that the creation failed, but that's not the case.) The indexing itself only happens for text/plain files, but that check is in the indexers themselves, after this error occurs (and it's entirely possible that a text/plain upload could actually have some binary or non-UTF-8 content). bytes objects for the binary contents get converted to str, with resulting errors when they are not in fact UTF-8 text. This patch makes the places that might try indexing binary content do the conversion to strings, for Python 3, with errors='ignore', so that at least no such exception occurs (and if the file is not text/plain, the results of the conversion will then get discarded in the indexers). |
| files |
| changeset | bb7865241f8a |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Joseph Myers <jsm@polyomino.org.uk> |
| description | Make CSV import/export compatible across Python versions (also RDBMS journals) (issue 2550976, issue 2550975). The roundup-admin export and import commands are used for migrating between different database backends. It is desirable that they should be usable also for migrations between Python 2 and Python 3, and in some cases (e.g. with the anydbm backend) this may be required. To be usable for such migrations, the format of the generated CSV files needs to be stable, meaning the same as currently used with Python 2. The export process uses repr() to produce the fields in the CSV files and eval() to convert them back to Python data structures. repr() of strings with non-ASCII characters produces different results for Python 2 and Python 3. This patch adds repr_export and eval_import functions to roundup/anypy/strings.py which provide the required operations that are just repr() and eval() in Python 2, but are more complicated in Python 3 to use data representations compatible with Python 2. These functions are then used in the required places for export and import. repr() and eval() are also used in storing the dict of changed values in the journal for the RDBMS backends. It is similarly desirable that the database be compatible between Python 2 and Python 3, so that export and import do not need to be used for a migration between Python versions for non-anydbm back ends. Thus, this patch changes rdbms_common.py in the places involved in storing journals in the database, not just in those involved in import/export. Given this patch, import/export with non-ASCII characters appear based on some limited testing to work across Python versions, and an instance using the sqlite backend appears to be compatible between Python versions without needing import/export, *if* the sessions/otks databases (which use anydbm) are deleted when changing Python version. |
| files |
| changeset | 6b0c542642be |
|---|---|
| branch | |
| bookmark | |
| tag | |
| user | Christof Meerwald <cmeerw@cmeerw.org> |
| description | blobfiles now always stores/returns bytes any conversation is done in the backend layer added a special "binary_content" property to read the data as bytes changed history generation to read property data on demand |
| files |