http://hg.code.sf.net:8000/p/roundup/code/atom-log/tip/roundup/admin.py Mercurial Repository: p/roundup/code: roundup/admin.py history 2026-04-08T21:39:40-04:00 chore: remove __future print_funcion from code. http://hg.code.sf.net:8000/p/roundup/code/#changeset-9c3ec0a5c7fc88acb8a65632ecc13b2d52380314 John Rouillard rouilj@ieee.org 2026-04-08T21:39:40-04:00 2026-04-08T21:39:40-04:00
changeset 9c3ec0a5c7fc
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description chore: remove __future print_funcion from code.

Not needed as of Python 3.
files
feat: improve store_trace_reason with extract parameter http://hg.code.sf.net:8000/p/roundup/code/#changeset-f80c566f5726c55c8654ec24a9f73a138b2c8a6f John Rouillard rouilj@ieee.org 2026-04-06T22:10:23-04:00 2026-04-06T22:10:23-04:00
changeset f80c566f5726
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description feat: improve store_trace_reason with extract parameter

store_trace_reason() used embedded code to extract reason based on the
location passed to the function.

This change adds support for extract keyword that is a Python
expression eval'ed when the underlying function/method is called. All
callers now set the extract parameter. The prior embedded code has
been removed from store_trace_reason().

Failure to eval the expression results in an roundup.logcontext error
severity log. Also updated docs.

Also replaced env['REQUEST_URI'] with env['PATH_INFO'] for web based
entry points as REQUEST_URI isn't documented as a required key and
some other front end (e.g. zope, cgi) might not supply this.
files
bug: import/importables fail to set newid correctly. http://hg.code.sf.net:8000/p/roundup/code/#changeset-7a7f6ee0a09ed8860cdf63bbeea9b00fa88e6db0 John Rouillard rouilj@ieee.org 2026-03-23T22:22:24-04:00 2026-03-23T22:22:24-04:00
changeset 7a7f6ee0a09e
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description bug: import/importables fail to set newid correctly.

Looks like I broke import when I was fixing a scandir
issue. Overindented a couple of lines. This prevented the next used id
from being set properly.

It also resulted in wierd output when importing (bunches of newlines).

Also fixed the two export/import tests to verify the new nextid for
all the classes is equal or higher than the old one. Ideally they
should be equal, but there is a bug somewhere where we get a double
increment in the export/import cycle dropping a number or two.
files
bug: fix traceback in roundup-admin init with bad config values http://hg.code.sf.net:8000/p/roundup/code/#changeset-e8d1da6e3571e5bef15fdb757f6a8d68f5a1db7d John Rouillard rouilj@ieee.org 2026-03-23T13:18:41-04:00 2026-03-23T13:18:41-04:00
changeset e8d1da6e3571
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description bug: fix traceback in roundup-admin init with bad config values

initialize accepts setting values for config.ini file settings. If
they are not valid, you got a python traceback.


ConfigurationError exceptions are now trapped. The admin.py's
usageError_feedback method is used to inform the user. Also the
feedback message now starts with a newline making it easier to read by
separating it from command that caused the issue.
files
feat: add nanoid pkg trace_id gen and decorator for setting processName http://hg.code.sf.net:8000/p/roundup/code/#changeset-b09ef85f0da62646efe031ee76934ee9b158ae07 John Rouillard rouilj@ieee.org 2025-12-08T00:23:14-05:00 2025-12-08T00:23:14-05:00
changeset b09ef85f0da6
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description feat: add nanoid pkg trace_id gen and decorator for setting processName

nanoid is a shorter unique id generator and faster than uuid.
I truncate nanoid id's to 12 chars to make it more readable.

Also added decorator to allow setting the default processName
definition in the logging module. admin.py and wsgi_handler now set
processName. configuration.py knows how to overide the processName if
set to the default MainProcess.

Updated install docs to add nanoid as optional, how to switch to
different trace_id output. pydoc generated docs include logcontext
module and are referenced from admin.py.
files
refactor: change some classes to use __slots__ http://hg.code.sf.net:8000/p/roundup/code/#changeset-224ccb8b49ca54569350edb5905b19185a97b092 John Rouillard rouilj@ieee.org 2025-11-03T00:13:04-05:00 2025-11-03T00:13:04-05:00
changeset 224ccb8b49ca
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description refactor: change some classes to use __slots__

Speed up access to and reduce size of some low level classes. A few
classes in security.py, rest.py are heavily used. But for all, it
prevents adding random properties to lower level classes that people
shouldn't be mucking with. While doing this I found some test cases
accessing an invalid property name and this change caused the cases to
crash.

admin.py:

Use new method Role.props_dict() and Permission.props_dict() where
original code just referenced __dict__ when printing Role/Permission.

mlink_expr.py:

Add slots to multiple classes.

Classes Binary and Unary set real properties/attributes. Classes that
inherit from them (Equals, Empty, Not, Or, And) define empty slots
tuple to eliminate need for __dict__.

Class Expression also gets a slot.

rate_limit.py:

RateLimit and Gcra classes get slots.

A couple of pep8 fixes: sort imports, remove trailing spaces on a
line, remove unused noqa comment.

rest.py:

Add slots to class SimulateFieldStorageFromJson and FsValue
classes. The memory savings from this could be useful as well as
speedier access to the attributes.

security.py:

Add slots to Permission class. To prevent conflict between slot
limit_perm_to_props_only and the class variable of the same name,
rename the class variable to limit_perm_to_props_only_default.
Also define method props_dict() to allow other code to get a dict to
iterate over when checking permissions.

Add slots to class Role along with props_dict() method.

Add slots to class Security. Also have to add explicit __dict__ slot
to support test override of the hasPermission() method. Add
props_dict() method, currently unused, but added for symmetry.

support.py:

TruthDict and PrioList gets slots.

test/test_cgi.py:

Fix incorrect setting of permission property. Was setting
permissions. So testing may not have been doing what we thought it
was. Multiple places found with this typo.

Remove setting of permissions in some places where it should
have no effect on the test and looks like it was just copypasta.

test/test_xmlrpc.py

Remove setting of permissions in some places where it should
have no effect on the test and looks like it was just copypasta.
files
feature: add thread local trace_id and trace_reason to logging. http://hg.code.sf.net:8000/p/roundup/code/#changeset-14c7c07b32d846e405260147a3112e8c8b15642a John Rouillard rouilj@ieee.org 2025-09-16T22:53:00-04:00 2025-09-16T22:53:00-04:00
changeset 14c7c07b32d8
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description feature: add thread local trace_id and trace_reason to logging.

Added trace_id to default logging so that all logs for a given request
share the same trace_id.

This allows correlation of logs across a request.

admin_guide.txt, upgrading.txt:
add docs
update sample configs to include trace_id.
rewrite logging docs in admin_guide. Hopefully they are clearer now.
clean up some stuff in the logging config file docs.

admin.py:

add decorators to run_command to enable trace_id.
change calls to db.commit() to use run_command to get trace_id.

configuration.py:

clean up imports.
update docstrings, comments and inline docs.
add trace_id to default log format.
add function for testing decorated with trace_id.
add support for dumping stack trace in logging.
add check for pytest in sys.modules to enable log propagation when
pytest is running. Otherwise tests fail as the caplog logger doesn't
see the roundup logs.

logcontext.py:

new file to handle thread local contextvar mangement.

mailgw.py:

add decorators for trace_id etc.

scripts/roundup_xlmrpc_server.py:

add decorators for trace_id etc.
fix encoding bug turning bytes into a string.
fix command line issue where we can't set encoding. (not sure if
changing encoding via command line even works)

cgi/client.py

decorate two entry points for trace_id etc.

cgi/wsgi_handler.py:

decorate entry point for trace_id etc.

test/test_config.py:

add test for trace_id in new log format.
test various cases for sinfo and errors in formating msg.
files
bug, refactor, test: make pragma history_length work interactively http://hg.code.sf.net:8000/p/roundup/code/#changeset-254f70dfc5856d142ecbc7a7c31c59d75c625f08 John Rouillard rouilj@ieee.org 2025-08-31T20:59:04-04:00 2025-08-31T20:59:04-04:00
changeset 254f70dfc585
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description bug, refactor, test: make pragma history_length work interactively

history_length could be set interactively, but it was never used to set
readline/pyreadline3's internal state. Using the pragma setting on the
roundup-admin command line did set readline's state.

Also refactored 2 calls to self.readline.get_current_history_length()
into one call and storing in a variable. Also changed method for
creating history strings for printing.

Tests added for history_length pragma on cli and interactive use.

Added test for exiting roundup-admin with EOF on input.

Added test for 'readline nosuchdirective' error case.

Added test to readline with a command directive to set an internal
variable. This last one has no real test to see if it was successful
because I can't emulate a real keyboard/tty which is needed to test.
files
feat: add support for ! history and readline command in roundup-admin http://hg.code.sf.net:8000/p/roundup/code/#changeset-3bdae15252c65b87f524942af3e1d019896295e1 John Rouillard rouilj@ieee.org 2025-08-31T16:54:17-04:00 2025-08-31T16:54:17-04:00
changeset 3bdae15252c6
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description feat: add support for ! history and readline command in roundup-admin

Ad support to change input mode emacs/vi using new 'readline'
roundup-admin command. Also bind keys to command/input strings, List
numbered history and allow rerunning a command with !<number> or allow
user to edit it using !<number>:p.

admin_guide.txt:
Added docs.

admin.py:

add functionality. Reconcile import commands to standard. Replace
IOError with FileNotFoundError no that we have removed python 2.7
support. Add support for identifying backend used to supply line
editing/history functions. Add support for saving commands sent on
stdin to history to allow preloading of history.

test_admin.py:

Test code. Can't test mode changes as lack of pty when driving command
line turns off line editing in readline/pyreadline3. Similarly can't
test key bindings/settings.

Some refactoring of test conditions that had to change because of
additional output reporting backend library.
files
feat: add 'q' as alias to quit to exit interactive roundup-admin http://hg.code.sf.net:8000/p/roundup/code/#changeset-1a93dc58f97529a96a14e3973a1589870a82b726 John Rouillard rouilj@ieee.org 2025-08-26T23:37:42-04:00 2025-08-26T23:37:42-04:00
changeset 1a93dc58f975
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description feat: add 'q' as alias to quit to exit interactive roundup-admin

Also require no arguments to 'q', 'quit' or 'exit' before exiting.
Now typing 'quit a' will get an unknown command error.

Add to admin-guide how to get out of interactive mode.

Also test 'q' and 'exit' commands.

No upgrading docs added. Not that big a feature. Just noted in
CHANGES. Reporting error if argument provided is unlikely to be an
issue IMO, so no upgrading.txt entry.
files
build: remove extra <p></p> generated by blank lines. http://hg.code.sf.net:8000/p/roundup/code/#changeset-e5d07fac02496c9b5a2cede5c67a96c0281c8abc John Rouillard rouilj@ieee.org 2025-07-12T22:55:39-04:00 2025-07-12T22:55:39-04:00
changeset e5d07fac0249
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description build: remove extra <p></p> generated by blank lines.
files
fix: issue1895197 - translated help texts in admin.py not displayed correctly. http://hg.code.sf.net:8000/p/roundup/code/#changeset-4d3b371ed5439cd5c4b5b9d402ab45d8c573070a John Rouillard rouilj@ieee.org 2025-01-20T15:21:56-05:00 2025-01-20T15:21:56-05:00
changeset 4d3b371ed543
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix: issue1895197 - translated help texts in admin.py not displayed correctly.

Adjust the technique used to remove the translated "Usage:" prefix
from the printed line. Support Chinese (zh) as well.

Add coding marker so I don't have to modify the tests to skip the ones
that touch the dmin module.
files
fix: crash roundup-admin perftest password without rounds= argument. http://hg.code.sf.net:8000/p/roundup/code/#changeset-1189c742e4b3e5b39675efa30542566ec6195874 John Rouillard rouilj@ieee.org 2024-12-30T02:59:27-05:00 2024-12-30T02:59:27-05:00
changeset 1189c742e4b3
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix: crash roundup-admin perftest password without rounds= argument.

If the rounds value is taken from the config file, it doesn't need to
be parsed into an int.
files
refactor: extract code as method from do_import http://hg.code.sf.net:8000/p/roundup/code/#changeset-5913ec1673c25ebf9c213ddf38ef61a881872c7a John Rouillard rouilj@ieee.org 2024-12-18T21:49:48-05:00 2024-12-18T21:49:48-05:00
changeset 5913ec1673c2
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description refactor: extract code as method from do_import

This started by trying to fix a resource error where the os.scandir
iterator was not closed for some reason (it should have been exhausted
and close). While the ResourceWarning is fixed with:

with os.scandir() as ...:
indented existing code

it breaks under python2. Trying a python2 friendly option:

dirscan = os.scandir()
...

if hasattr(dirscan,'close'):
dirscan.close()

doesn't fix the resource warning.

Using the 'with os.scandir()...' indented a nested block too far so I
extracted it into another method.

It is only this method that survives 8-(.
files
[permission-performance] Change permission representation http://hg.code.sf.net:8000/p/roundup/code/#changeset-c12377fb4144e63ba869fd0d85ea37b269af1b7a Ralf Schlatterbeck rsc@runtux.com 2024-10-18T16:52:42+02:00 2024-10-18T16:52:42+02:00
changeset c12377fb4144
branch permission-performance
bookmark
tag
user Ralf Schlatterbeck <rsc@runtux.com>
description Change permission representation

Now permissions are checked in different order. Permissions without a
check method (which are cheap to check) are checked first. Only if no
permission is found do we check permissions with check methods.
files
fix: broken translatable error message template http://hg.code.sf.net:8000/p/roundup/code/#changeset-e579aef218aa65903a13cc6bb01d2c7ea7f86bf0 John Rouillard rouilj@ieee.org 2024-08-07T21:05:39-04:00 2024-08-07T21:05:39-04:00
changeset e579aef218aa
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix: broken translatable error message template
files
docs: normalize all help strings and make html output look better. http://hg.code.sf.net:8000/p/roundup/code/#changeset-5aed4911836b67e96f5fded6a5a2a497c9ff6f59 John Rouillard rouilj@ieee.org 2024-07-16T21:45:10-04:00 2024-07-16T21:45:10-04:00
changeset 5aed4911836b
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description docs: normalize all help strings and make html output look better.
files
fix: keep python2 working a little longer. http://hg.code.sf.net:8000/p/roundup/code/#changeset-586f76eb33e880272bf46663f139a35deb23fc69 John Rouillard rouilj@ieee.org 2024-07-16T02:10:25-04:00 2024-07-16T02:10:25-04:00
changeset 586f76eb33e8
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix: keep python2 working a little longer.
files
refactor!: replace os.listdir() with os.scandir() http://hg.code.sf.net:8000/p/roundup/code/#changeset-1045425c23b2a30a8648a346d8f68b22a6e1a9de John Rouillard rouilj@ieee.org 2024-07-16T01:05:49-04:00 2024-07-16T01:05:49-04:00
changeset 1045425c23b2
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description refactor!: replace os.listdir() with os.scandir()

In many places we did a listdir() then a stat to see if it's a file or
directory. This change removes the need for the stat call. Also for
larger directories, scandir() is an iterator, so less memory use.

There is one remnant of listdir used in an error handler. That
requires a stat on each element in the directory, so there is no
benefit to using scandir() other than a slight memory saving on a
rarely used piece of code.

BREAKING CHANGE:

Python 2 requires installation of scandir pip package after this
commit.
files
issue2550983 - help_commands_html unused http://hg.code.sf.net:8000/p/roundup/code/#changeset-851ddd72f9ce871cf302d6a305f3838bf1039ee8 John Rouillard rouilj@ieee.org 2024-05-26T00:33:43-04:00 2024-05-26T00:33:43-04:00
changeset 851ddd72f9ce
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2550983 - help_commands_html unused

Make the command htmlhelp trigger the command.

The output is ugly however. But clean this up and we could generate
html docs for the current admin.py version replacing existing docs
similar to how the doc is now done for config.ini.
files
fix: import/export under windows. http://hg.code.sf.net:8000/p/roundup/code/#changeset-77c109725a7e326c55b8313b41c9748324d3e53e John Rouillard rouilj@ieee.org 2024-04-18T19:36:32-04:00 2024-04-18T19:36:32-04:00
changeset 77c109725a7e
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix: import/export under windows.

Export used native \r\n line endings on windows. This results in
blank lines when read and Roundup crashes on import.

Use \n line endings when writing due to the hard coded \n or \r (but
not \r\n) line terminator expected by csv.reader().

Also updates CHANGES.txt to cover this and a the fix for
{Otk,Session}.clear() when backed by dumb dbm on windows.
files
fix: roundup-admin updateconfig warn on mysql_{charset/collation} mismatch http://hg.code.sf.net:8000/p/roundup/code/#changeset-eb1fbbd53b6c6dc6fa50e0cc412e917cfce1e094 John Rouillard rouilj@ieee.org 2024-04-08T11:55:55-04:00 2024-04-08T11:55:55-04:00
changeset eb1fbbd53b6c
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix: roundup-admin updateconfig warn on mysql_{charset/collation} mismatch

I added a simple test that requires the mysql_charset with a '_'
appended to be a prefix of the mysql_collation.

The intent is to detect an incompatible charset and collation since
collation is now settable in config.ini.

This may not be sufficient but...
files
feat: can use escaped tokens inside quotes including quotes. http://hg.code.sf.net:8000/p/roundup/code/#changeset-9a74dfeb862057d0dcf2b433aeb045a8a8150baf John Rouillard rouilj@ieee.org 2024-04-06T20:37:45-04:00 2024-04-06T20:37:45-04:00
changeset 9a74dfeb8620
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description feat: can use escaped tokens inside quotes including quotes.

Change tokenizer to support:

cmd with arg "string with embedded \" double quote"

works for single quotes too. Mixed quotes can skip the \" or \'
escaping.

Also:

quoted cmds args "can include \n newline, \t tab and \r return"

Added a doc example, also tests for new feature.
files
feat: add Progress output for admin export/exporttables http://hg.code.sf.net:8000/p/roundup/code/#changeset-d523d5c648e9e4adf4ca4a61ab27b797de53923c John Rouillard rouilj@ieee.org 2024-03-30T23:29:02-04:00 2024-03-30T23:29:02-04:00
changeset d523d5c648e9
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description feat: add Progress output for admin export/exporttables

This provides feedback on how far along the export is.

Detailed per node output with -V (for use when identifying exactly
what node is hanging the output) is kind of ugly when used with
Progress. But it's functional and not the expected operation mode.
files
chore(lint): formatting whitespace consolidate nested if ... http://hg.code.sf.net:8000/p/roundup/code/#changeset-b530069f3120d9cb94a2cb070270985e43f3eb17 John Rouillard rouilj@ieee.org 2024-03-24T15:22:50-04:00 2024-03-24T15:22:50-04:00
changeset b530069f3120
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description chore(lint): formatting whitespace consolidate nested if ...
files
fix(i18n): clean up translator hint http://hg.code.sf.net:8000/p/roundup/code/#changeset-aa1fd8704469631a37e5337f46a1f03023e87c37 John Rouillard rouilj@ieee.org 2024-03-21T15:26:29-04:00 2024-03-21T15:26:29-04:00
changeset aa1fd8704469
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix(i18n): clean up translator hint
files
feat: add support for controlling readline history features http://hg.code.sf.net:8000/p/roundup/code/#changeset-7c0a8088b053891acfee65ecbbd9b7c151ebc604 John Rouillard rouilj@ieee.org 2024-03-13T12:18:36-04:00 2024-03-13T12:18:36-04:00
changeset 7c0a8088b053
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description feat: add support for controlling readline history features

You can disable loading or saving the history file and disable loading
the readline rc file. This is controlled by pragma history_features
which is a bit string. Setting a bit disables the feature.

Also cleaned up a python traceback if an invalid pragma was specified
on the command line. Just displays an error message and drops into
interactive mode.

I had a hand when running regression tests with
tet-admin.py::testBasicInteractive. It was loading the history file
and saving the commands making the file huge. So disable all
history_features by default when testing by setting pragma to 7.
files
fix: enhancement to history command output and % template fix. http://hg.code.sf.net:8000/p/roundup/code/#changeset-2d4684e4702d2aecb2e8b907755a5cffcd4f5270 John Rouillard rouilj@ieee.org 2024-03-12T11:52:17-04:00 2024-03-12T11:52:17-04:00
changeset 2d4684e4702d
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix: enhancement to history command output and % template fix.

Rather than using the key field, use the label field for descriptions.
Call cls.labelprop(default_to_id=True) so it returns id rather than
the first sorted property name.

If labelprop() returns 'id' or 'title', we return nothing. 'id' means
there is no label set and no properties named 'name' or 'title'. So
have the caller do whatever it wants (prepend classname for example)
when there is no human readable name. This prevents %(name)s%(key)s
from producing: 23(23).

Also don't accept the 'title' property. Titles can be too
long. Arguably we could: '%(name)20s' to limit the title
length. However without ellipses or something truncating the title
might be confusing. So again pretend there is no human readable name.
files
fix: replace FileNotFoundError with IOError http://hg.code.sf.net:8000/p/roundup/code/#changeset-10da9e12c10fb15ba6bfe1f84036b527f587194b John Rouillard rouilj@ieee.org 2024-03-10T23:12:54-04:00 2024-03-10T23:12:54-04:00
changeset 10da9e12c10f
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix: replace FileNotFoundError with IOError

Python2 doesn't have the FileNotFoundError which is a subclass of
IOError.

So use IOError.
files
fix: handle case where readline init file is missing http://hg.code.sf.net:8000/p/roundup/code/#changeset-f84f7879c768f945690cfac3551839b78aed7f02 John Rouillard rouilj@ieee.org 2024-03-10T22:47:58-04:00 2024-03-10T22:47:58-04:00
changeset f84f7879c768
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix: handle case where readline init file is missing

Ignore the FileNotFoundError. File is optional.
files
Summary: feat: roundup-admin history command has human interpretable output http://hg.code.sf.net:8000/p/roundup/code/#changeset-8bdf0484215cf7beebd02bc43e754856c6b0a685 John Rouillard rouilj@ieee.org 2024-03-10T21:59:20-04:00 2024-03-10T21:59:20-04:00
changeset 8bdf0484215c
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Summary: feat: roundup-admin history command has human interpretable output

Reformat journal entries and make try to make readable sentences out
of them.

Set up translation markers and added hints for the tanslators by
marking translator comments with;

# .Hint text for translator

on the line before _() markers.

Doc'ed changes in roundup-admin docs and added info to upgrading.txt.

If the user wants old format, they can call

history designator raw
files
add missing help text for history_length pragma. Also don't crash report missing text if text is missing http://hg.code.sf.net:8000/p/roundup/code/#changeset-5f3b49bb7742b87b66278e75c5e0e173a25c8dcb John Rouillard rouilj@ieee.org 2024-03-10T17:41:16-04:00 2024-03-10T17:41:16-04:00
changeset 5f3b49bb7742
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description add missing help text for history_length pragma. Also don't crash report missing text if text is missing
files
save roundup-admin history between sessions. http://hg.code.sf.net:8000/p/roundup/code/#changeset-d7d68da9878f1e0ddd7f9bc746ec8cc1c56a2235 John Rouillard rouilj@ieee.org 2024-03-10T13:44:13-04:00 2024-03-10T13:44:13-04:00
changeset d7d68da9878f
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description save roundup-admin history between sessions.

Also load ~/.roundup_admin_rlrc (rlrc readline run commands) file to
set history-size persistently, allow user to use vi mode etc.
files
chore: reduce nesting if/elif/else blocks; uneeded var assignment http://hg.code.sf.net:8000/p/roundup/code/#changeset-ac2c034542e6b7c0a87f8264ba5d37995200b869 John Rouillard rouilj@ieee.org 2024-03-02T13:48:40-05:00 2024-03-02T13:48:40-05:00
changeset ac2c034542e6
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description chore: reduce nesting if/elif/else blocks; uneeded var assignment

also add lint hint

Leave chained if/elif/else blocks after returns if it doesn't reduce
nesting.
files
fix: undo last minute edit and try to get make checkin clean. http://hg.code.sf.net:8000/p/roundup/code/#changeset-417c8ddc98ac2d2c2c70b602f5fb529db81d6fd4 John Rouillard rouilj@ieee.org 2024-03-01T15:42:24-05:00 2024-03-01T15:42:24-05:00
changeset 417c8ddc98ac
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix: undo last minute edit and try to get make checkin clean.

Fixed admin.py.

Change # noqa: 821 to # noqa: F821 to make ruff ignore undefined
variable in the python 2 branch of the code. Hopefully the flake8
run in CI will accept the change from 821 to F821 and not error with
an undefined variable.
files
fix: overeager cleanup http://hg.code.sf.net:8000/p/roundup/code/#changeset-09b216591db564069f6f3f400decb9fed51af208 John Rouillard rouilj@ieee.org 2024-03-01T15:13:36-05:00 2024-03-01T15:13:36-05:00
changeset 09b216591db5
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix: overeager cleanup

I was cleaning some dregs and removed a required line after I
completed testing 8-(.


Added a 'make check' that runs ruff with the test that finds the
undefined variable.
files
fix(refactor): multiple fixups using ruff linter; more testing. http://hg.code.sf.net:8000/p/roundup/code/#changeset-b2dbab2b34bce7223c4369f85f2f358a10959d3f John Rouillard rouilj@ieee.org 2024-03-01T14:53:18-05:00 2024-03-01T14:53:18-05:00
changeset b2dbab2b34bc
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix(refactor): multiple fixups using ruff linter; more testing.

Converting to using the ruff linter and its rulesets. Fixed a number
of issues.

admin.py:

sort imports

use immutable tuples as default value markers for parameters where a
None value is valid.

reduced some loops to list comprehensions for performance

used ternary to simplify some if statements

named some variables to make them less magic
(e.g. _default_savepoint_setting = 1000)

fixed some tests for argument counts < 2 becomes != 2 so 3 is an
error.

moved exception handlers outside of loops for performance where
exception handler will abort loop anyway.

renamed variables called 'id' or 'dir' as they shadow builtin
commands.

fix translations of form _("string %s" % value) -> _("string %s") %
value so translation will be looked up with the key before
substitution.

end dicts, tuples with a trailing comma to reduce missing comma
errors if modified

simplified sorted(list(self.setting.keys())) to
sorted(self.setting.keys()) as sorted consumes whole list.

in if conditions put compared variable on left and threshold condition
on right. (no yoda conditions)

multiple noqa: suppression

removed unneeded noqa as lint rulesets are a bit different

do_get - refactor output printing logic: Use fast return if not
special formatting is requested; use isinstance with a tuple
rather than two isinstance calls; cleaned up flow and removed
comments on algorithm as it can be easily read from the code.

do_filter, do_find - refactor output printing logic. Reduce
duplicate code.

do_find - renamed variable 'value' that was set inside a loop. The
loop index variable was also named 'value'.

do_pragma - added hint to use list subcommand if setting was not
found. Replaced condition 'type(x) is bool' with 'isinstance(x,
bool)' for various types.

test_admin.py

added testing for do_list

better test coverage for do_get includes: -S and -d for multilinks,
error case for -d with non-link.

better testing for do_find including all output modes

better testing for do_filter including all output modes

fixed expected output for do_pragma that now includes hint to use
pragma list if setting not found.
files
doc: add comment on method to set savepoint_limit dynamically http://hg.code.sf.net:8000/p/roundup/code/#changeset-4dda4a9dfe0bf396f39c73e0887e4707bdd2e429 John Rouillard rouilj@ieee.org 2023-10-29T13:18:36-04:00 2023-10-29T13:18:36-04:00
changeset 4dda4a9dfe0b
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description doc: add comment on method to set savepoint_limit dynamically

savepoint_limit isn't a real config option as in config.ini. I
generate one on the fly for use in the backend during import.

While this is arguably a valid config option, it is also something
that is niche (import a tracker to postgres) and adds clutter to
config.ini.
files
fix: out of memory error when importing under postgresql http://hg.code.sf.net:8000/p/roundup/code/#changeset-5b41018617f2984f188ec7f1516cfcd970a8b9ba John Rouillard rouilj@ieee.org 2023-10-19T16:11:25-04:00 2023-10-19T16:11:25-04:00
changeset 5b41018617f2
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix: out of memory error when importing under postgresql

If you try importing more than 20k items under postgresql you can run
out of memory:

psycopg2.errors.OutOfMemory: out of shared memory
HINT: You might need to increase max_locks_per_transaction.

Tuning memory may help, it's unknown at this point.

This checkin forces a commit to the postgres database after 10,000
rows have been added. This clears out the savepoints for each row and
starts a new transaction.

back_postgresql.py:

Implement commit mechanism in checkpoint_data(). Add two class level
attributes for tracking the number of savepoints and the limit when
the commit should happen.

roundup_admin.py:

implement pragma and dynamically create the config item
RDBMS_SAVEPOINT_LIMIT used by checkpoint_data.

Also fixed formatting of descriptions when using pragma list in
verbose mode.

admin_guide.txt, upgrading.txt:

Document change and use of pragma savepoint_limit in roundup-admin
for changing the default of 10,000.

test/db_test_base.py:

add some more asserts. In existing testAdminImportExport, set the
savepoint limit to 5 to test setting method and so that the commit
code will be run by existing tests. This provides coverage, but
does not actually test that the commit is done every 5 savepoints
8-(. The verification of every 5 savepoints was done manually
using a pdb breakpoint just before the commit.

acknowledgements.txt:

Added 2.4.0 section mentioning Norbert as he has done a ton of
testing with much larger datasets than I can test with.
files
bug: Fix roundup-admin security command. Lowercase optionalarg. http://hg.code.sf.net:8000/p/roundup/code/#changeset-4de48eadf5f4eb792ef84b1188800b273fe33f49 John Rouillard rouilj@ieee.org 2023-10-06T09:53:22-04:00 2023-10-06T09:53:22-04:00
changeset 4de48eadf5f4
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description bug: Fix roundup-admin security command. Lowercase optionalarg.

Roles are indexed by lower case role name. So 'security User' and
'security user' should generate the same output. Also add testing for
this case.

Thread:

https://sourceforge.net/p/roundup/mailman/roundup-users/thread/CAH-41398iTPhze7D_pZB8tqTBHF%3Dq6HYonbcG%2B%2BYN-ioDssXBw%40mail.gmail.com/#msg41557225

starting from:
https://sourceforge.net/p/roundup/mailman/message/41557225/
files
fix: restore roundup-admin display output format w/o pragmas http://hg.code.sf.net:8000/p/roundup/code/#changeset-8329b2227adb9891b2ea070ce4377152b96d136f John Rouillard rouilj@ieee.org 2023-07-26T11:05:10-04:00 2023-07-26T11:05:10-04:00
changeset 8329b2227adb
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix: restore roundup-admin display output format w/o pragmas

Do not indent the display output unless user requested protected
fields or headers. This returns the format to what it was prior to
2.3.0.

Parsing roundup-admin output programmaticly was never a use case, but
don't break the output format without a reason.
files
flake8 fix indents. http://hg.code.sf.net:8000/p/roundup/code/#changeset-df87f02c2930452138dd3f5480c68c7b76e86fb2 John Rouillard rouilj@ieee.org 2023-07-23T23:18:26-04:00 2023-07-23T23:18:26-04:00
changeset df87f02c2930
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description flake8 fix indents.
files
issue685275 - show retired/unretired items in roundup-admin http://hg.code.sf.net:8000/p/roundup/code/#changeset-73dfa9df9fb08618e72479ec013d48ffac0427c7 John Rouillard rouilj@ieee.org 2023-07-14T22:07:23-04:00 2023-07-14T22:07:23-04:00
changeset 73dfa9df9fb0
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue685275 - show retired/unretired items in roundup-admin

add pragma display_header to print headers for display command. Header
displays designator and retired/active status.

Add doc of pragma to affected commands.

Add test for code paths.
files
issue685275 - show retired/unretire commands http://hg.code.sf.net:8000/p/roundup/code/#changeset-c8c4514f4c3e809991bc419a330900a5e9d5ddd5 John Rouillard rouilj@ieee.org 2023-07-14T20:34:30-04:00 2023-07-14T20:34:30-04:00
changeset c8c4514f4c3e
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue685275 - show retired/unretire commands

For roundup-admin, add pragma show_retired [no, only, both] to control
showing retired items in list and table commands:

no - do not show retired
only - only show retired
both - show retired and unretired (active) items

Also sort results of Class::getnodeids() in back_anydbm.py. Anydbm
Class::list() expicitly sorts the returned values and a test depends
on the order of the returned items.

I can't find any docs that say Class::list() sorts and there is no
explicit sort in the rdbms_common.py implementation. It looks like
the natural order returned in the rdbms case for these methods is
sorted.

However the test fails for the anydbm case if I don't sort the results
of back_anydbm.py:Class::getnodeids() to match
back_anydbm.py:Class::list().

This also fixes a spelling error in comment.
files
Add -P pragma=value command line option to roundup-admin. http://hg.code.sf.net:8000/p/roundup/code/#changeset-534f8bdb8f943a61056d14d62e53c681b50f2204 John Rouillard rouilj@ieee.org 2023-07-14T00:30:44-04:00 2023-07-14T00:30:44-04:00
changeset 534f8bdb8f94
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Add -P pragma=value command line option to roundup-admin.

To set pragmas when using non-interactive mode, or set on command line
when going into interactive mode.

Also changed specification test to use command line pragma setting
rather than interactive. This tests the -P option without having to
run an extra test.

Docs updated as well.
files
update help text on display_protected pragma http://hg.code.sf.net:8000/p/roundup/code/#changeset-939fcfdfa3703f08bbdf8cc15a5f08a5f87d175b John Rouillard rouilj@ieee.org 2023-07-14T00:10:42-04:00 2023-07-14T00:10:42-04:00
changeset 939fcfdfa370
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description update help text on display_protected pragma

mention specification command.
files
initialize indexer_backend pragma from config. http://hg.code.sf.net:8000/p/roundup/code/#changeset-4c90a57c89e3a05dc6d13d9a60d29ef45dfd1067 John Rouillard rouilj@ieee.org 2023-07-14T00:09:47-04:00 2023-07-14T00:09:47-04:00
changeset 4c90a57c89e3
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description initialize indexer_backend pragma from config.

Prep for someday allowing roundup-admin to reindex using a different
indexer than the running roundup installation. This allows the
existing install to use one indexer while preparing to move to another
indexer using roundup-admin.
files
- issue2551103 - add pragma 'display_protected' to roundup-admin. http://hg.code.sf.net:8000/p/roundup/code/#changeset-fc9daba984c02230c8b5f91eda776cdb1f3086a3 John Rouillard rouilj@ieee.org 2023-07-13T23:47:43-04:00 2023-07-13T23:47:43-04:00
changeset fc9daba984c0
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description - issue2551103 - add pragma 'display_protected' to roundup-admin.

If setting is true, print protected attributes like id, activity,
actor... when using display or specification subcommands.
files
fix typo in comment. http://hg.code.sf.net:8000/p/roundup/code/#changeset-f7dc37df61b729445d8b2e3a6fe1c2ad233f2c8b John Rouillard rouilj@ieee.org 2023-05-30T15:15:35-04:00 2023-05-30T15:15:35-04:00
changeset f7dc37df61b7
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description fix typo in comment.
files
Handle traceback when OptionValueError raised during init. http://hg.code.sf.net:8000/p/roundup/code/#changeset-36916abe36e9f24bf00e6d97419db6fd871a3810 John Rouillard rouilj@ieee.org 2023-05-29T18:12:50-04:00 2023-05-29T18:12:50-04:00
changeset 36916abe36e9
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Handle traceback when OptionValueError raised during init.

Turn it into a UsageError for now. This causes it to report the
failing config option and print the help text for init. Printing the
help text may not be needed, but it works for now.
files
Remove pdb invocation. http://hg.code.sf.net:8000/p/roundup/code/#changeset-6795bd38411509167db391d1795e65177cef2129 John Rouillard rouilj@ieee.org 2023-05-24T12:37:06-04:00 2023-05-24T12:37:06-04:00
changeset 6795bd384115
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Remove pdb invocation.
files
- issue2551190 - Allow roundup-admin reindex to work in batches. http://hg.code.sf.net:8000/p/roundup/code/#changeset-312d52305583653240391f70b8ceeb72963cb95f John Rouillard rouilj@ieee.org 2023-05-24T12:13:05-04:00 2023-05-24T12:13:05-04:00
changeset 312d52305583
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description - issue2551190 - Allow roundup-admin reindex to work in batches.

Running:

roundup-admin -i ... reindex issue:1-1000

will reindex the first 1000 issues while reporting any missing issues
in the range.

Also completion progress is reported when indexing a specific class.

Note this require a chnge that makes an invalid command like:

reindex issue23f

to error with no such class issue23f. It used to reindex
issue23 which I consider a bug.

Updates to man page admin_guide.py.

Tests added.
files
Fix test failure caused by making genconfig trackerless http://hg.code.sf.net:8000/p/roundup/code/#changeset-bd6523c84a95c83e9079c84fdebc0c4ea34a6bf6 John Rouillard rouilj@ieee.org 2023-05-24T11:25:47-04:00 2023-05-24T11:25:47-04:00
changeset bd6523c84a95
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Fix test failure caused by making genconfig trackerless

When I removed the need to have a tracker for genconfig, I broke a
couple of tests. The breakage was hidden by a bad workflow.

This change allows the tests to pass and refactors reporting of
UsageError exceptions in admin.py.

It also adds a test for failure in install command as there is a test
case for it and this test tests the refactor.
files
set items can take 1 or more property=value http://hg.code.sf.net:8000/p/roundup/code/#changeset-6480443c26075ac9bf24ba49a43dc062f038bed8 John Rouillard rouilj@ieee.org 2023-05-22T11:19:38-04:00 2023-05-22T11:19:38-04:00
changeset 6480443c2607
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description set items can take 1 or more property=value

original usage string reported 2 or more.
files
Document that security subcommand reports invalid properties. http://hg.code.sf.net:8000/p/roundup/code/#changeset-40cab56a97f1967d3c046b295b035f44491aeafe John Rouillard rouilj@ieee.org 2023-05-22T11:17:17-04:00 2023-05-22T11:17:17-04:00
changeset 40cab56a97f1
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Document that security subcommand reports invalid properties.
files
issue2551253 - Modify password PBKDF2 method to use SHA512 http://hg.code.sf.net:8000/p/roundup/code/#changeset-9bd7ed918121ce5a8b5e03dac1cde046531e5c74 John Rouillard rouilj@ieee.org 2023-05-20T15:34:13-04:00 2023-05-20T15:34:13-04:00
changeset 9bd7ed918121
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description issue2551253 - Modify password PBKDF2 method to use SHA512

Implement new PBKDF2-SHA512 more secure password hash function.
Intended to be future default once I get more feedback.

upgrading.txt:
added doc for using it now.

reference.py:
Added enabling future features to use case for interfaces.py.

admin.py:
Changed perftest password command to use the default password
function. Can be used to verify that new hash is enabled properly.
files
Incomplete work to generate config doc from config.ini http://hg.code.sf.net:8000/p/roundup/code/#changeset-a210f4437b49548b399f115d93d12e288b4d08cf John Rouillard rouilj@ieee.org 2023-05-17T13:34:36-04:00 2023-05-17T13:34:36-04:00
changeset a210f4437b49
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Incomplete work to generate config doc from config.ini

This is an incomplete attempt to allow generation of the config.ini
documentation in reference.txt. It reformats the output of
'roundup_admin.py genconfig'. So it now includes all of the
settings. Using a Makefile rule like:

tracker_config.txt: ../roundup/configuration.py
python3 ../roundup/scripts/roundup_admin.py \
genconfig _temp_config.txt
sed -e '1,8d' \
-e 's/^\[\([a-z]*\)\]/\n.. index:: config.ini; sections \1\n\n.. code:: ini\n\n [\1]/' \
-e 's/^\([^[]\)/ \1/' \
_temp_config.txt > tracker_config.txt
rm -f _temp_config.txt

results in the config.ini split by section and index links being put
in place. However some sections have a comment before the [section]
marker. This comment is orphaned at the end of the prior section
rather than starting the new section. A simple sed won't allow the
lookahead needed to target the [section] marker and include the prior
comment block. Also there are still have some long lines generated (>
65 characters). Maybe a python script can import configuration.py and
output proper restructured text output?


reference.txt:
add a commented out include:: tracker_config.txt directive

roundup/admin.py:
don't require a tracker home for genconfig. So user can generate a
clean config.ini on demand. Tracker home is still required for
updateconfig.

roundup/configuration.py:
wrap lines better. A number of them are generating comments > 65
characters which is the targeted line length. This cleans up
config.ini too, so is a good thing.

website/www/conf.py:
ignore doc/tracker_config.ini when processing.
files
templates command should not prompt for a tracker home. http://hg.code.sf.net:8000/p/roundup/code/#changeset-209e94ed0277f177a971ef5dd4937d5c4ea4fc01 John Rouillard rouilj@ieee.org 2023-05-11T11:25:58-04:00 2023-05-11T11:25:58-04:00
changeset 209e94ed0277
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description templates command should not prompt for a tracker home.
files
flake8 fixes http://hg.code.sf.net:8000/p/roundup/code/#changeset-57f34b0b912cf5dea9cd93e5536878ca0ce39fb3 John Rouillard rouilj@ieee.org 2023-05-07T09:10:10-04:00 2023-05-07T09:10:10-04:00
changeset 57f34b0b912c
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description flake8 fixes

move import to top, indentation, remove trailing whitespace ...
files
Cache result of tracker open in roundup-admin http://hg.code.sf.net:8000/p/roundup/code/#changeset-393ae88bf7b1bc469414ca1cd4c94906d6783998 John Rouillard rouilj@ieee.org 2023-04-12T20:50:28-04:00 2023-04-12T20:50:28-04:00
changeset 393ae88bf7b1
branch
bookmark
tag
user John Rouillard <rouilj@ieee.org>
description Cache result of tracker open in roundup-admin

This change caches the tracker open command by default. To restore
the original action where every command run re-opened the tracker,
use:

pragma _reopen_tracker=true
files