Mercurial > p > roundup > code
changeset 8423:94eed885e958
feat: add support for using dictConfig to configure logging.
Basic logging config (one level and one output file non-rotating) was
always possible from config.ini. However the LOGGING_CONFIG setting
could be used to load an ini fileConfig style file to set various
channels (e.g. roundup.hyperdb) (also called qualname or tags) with
their own logging level, destination (rotating file, socket,
/dev/null) and log format.
This is now a deprecated method in newer logging modules. The
dictConfig format is preferred and allows disabiling other loggers as
well as invoking new loggers in local code. This commit adds support
for it reading the dict from a .json file. It also implements a
comment convention so you can document the dictConfig.
configuration.py:
new code
test_config.py:
test added for the new code.
admin_guide.txt, upgrading.txt CHANGES.txt:
docs added upgrading references the section in admin_guid.
| author | John Rouillard <rouilj@ieee.org> |
|---|---|
| date | Tue, 19 Aug 2025 22:32:46 -0400 |
| parents | e97cae093746 |
| children | 4a948ad46579 |
| files | CHANGES.txt doc/admin_guide.txt doc/upgrading.txt roundup/configuration.py test/test_config.py |
| diffstat | 5 files changed, 573 insertions(+), 15 deletions(-) [+] |
line wrap: on
line diff
--- a/CHANGES.txt Sun Aug 17 16:47:21 2025 -0400 +++ b/CHANGES.txt Tue Aug 19 22:32:46 2025 -0400 @@ -32,6 +32,8 @@ - add support for authorized changes. User can be prompted to enter their password to authorize a change. If the user's password is properly entered, the change is committed. (John Rouillard) +- add support for dictConfig style logging configuration. Ini/File + style configs will still be supported. (John Rouillard) 2025-07-13 2.5.0
--- a/doc/admin_guide.txt Sun Aug 17 16:47:21 2025 -0400 +++ b/doc/admin_guide.txt Tue Aug 19 22:32:46 2025 -0400 @@ -47,31 +47,284 @@ in the tracker's config.ini. -Configuring Roundup's Logging of Messages For Sysadmins -======================================================= - -You may configure where Roundup logs messages in your tracker's config.ini -file. Roundup will use the standard Python (2.3+) logging implementation. - -Configuration for standard "logging" module: - - tracker configuration file specifies the location of a logging - configration file as ``logging`` -> ``config`` - - ``roundup-server`` specifies the location of a logging configuration - file on the command line +Configuring Roundup Message Logging +=================================== + +You can control how Roundup logs messages using your tracker's +config.ini file. Roundup uses the standard Python (2.3+) logging +implementation. The config file and ``roundup-server`` provide very +basic control over logging. + Configuration for "BasicLogging" implementation: - tracker configuration file specifies the location of a log file ``logging`` -> ``filename`` - tracker configuration file specifies the level to log to as ``logging`` -> ``level`` + - tracker configuration file lets you disable other loggers + (e.g. when running under a wsgi framework) with + ``logging`` -> ``disable_loggers``. - ``roundup-server`` specifies the location of a log file on the command line - - ``roundup-server`` specifies the level to log to on the command line - -(``roundup-mailgw`` always logs to the tracker's log file) + - ``roundup-server`` enable using the standard python logger with + the tag/channel ``roundup.http`` on the command line + +By supplying a standard log config file in ini or json (dictionary) +format, you get more control over the logs. You can set different +levels for logs (e.g. roundup.hyperdb can be set to WARNING while +other Roundup log channels are set to INFO and roundup.mailgw logs at +DEBUG level). You can also send the logs for roundup.mailgw to syslog, +and other roundup logs go to an automatically rotating log file, or +are submitted to your log aggregator over https. + +Configuration for standard "logging" module: + - tracker configuration file specifies the location of a logging + configuration file as ``logging`` -> ``config``. In both cases, if no logfile is specified then logging will simply be sent to sys.stderr with only logging of ERROR messages. +Standard Logging Setup +---------------------- + +You can specify your log configs in one of two formats: + + * `fileConfig format + <https://docs.python.org/3/library/logging.config.html#logging.config.fileConfig>`_ + in ini style + * `dictConfig format + <https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig>`_ + using json with comment support + +The dictConfig allows more control over configuration including +loading your own log handlers and disabling existing handlers. If you +use the fileConfig format, the ``logging`` -> ``disable_loggers`` flag +in the tracker's config is used to enable/disable pre-existing loggers +as there is no way to do this in the logging config file. + +.. _`dictLogConfig`: + +dictConfig Based Logging Config +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +dictConfigs are specified in JSON format with support for comments. +The file name in the tracker's config for the ``logging`` -> ``config`` +setting must end with ``.json`` to choose the correct processing. + +Comments have to be in one of two forms: + +1. A ``#`` with preceding white space is considered a comment and is + stripped from the file before being passed to the json parser. This + is a "block comment". + +2. A ``#`` preceded by at least three + white space characters is stripped from the end of the line before + begin passed to the json parser. This is an "inline comment". + +Other than this the file is a standard json file that matches the +`Configuration dictionary schema +<https://docs.python.org/3/library/logging.config.html#configuration-dictionary-schema>`_ +defined in the Python documentation. + + +Example dictConfig Logging Config +................................. + +Note that this file is not actually JSON format as it include comments. +So you can not use tools that expect JSON (linters, formatters) to +work with it. + +The config below works with the `Waitress wsgi server +<https://github.com/Pylons/waitress>`_ configured to use the +roundup.wsgi channel. It also controls the `TransLogger middleware +<https://github.com/pasteorg/paste>`_ configured to use +roundup.wsgi.translogger, to produce httpd style combined logs. The +log file is specified relative to the current working directory not +the tracker home. The tracker home is the subdirectory demo under the +current working directory. The commented config is:: + + { + "version": 1, # only supported version + "disable_existing_loggers": false, # keep the wsgi loggers + + "formatters": { + # standard format for Roundup messages + "standard": { + "format": "%(asctime)s %(levelname)s %(name)s:%(module)s %(msg)s" + }, + # used for waitress wsgi server to produce httpd style logs + "http": { + "format": "%(message)s" + } + }, + "handlers": { + # create an access.log style http log file + "access": { + "level": "INFO", + "formatter": "http", + "class": "logging.FileHandler", + "filename": "demo/access.log" + }, + # logging for roundup.* loggers + "roundup": { + "level": "DEBUG", + "formatter": "standard", + "class": "logging.FileHandler", + "filename": "demo/roundup.log" + }, + # print to stdout - fall through for other logging + "default": { + "level": "DEBUG", + "formatter": "standard", + "class": "logging.StreamHandler", + "stream": "ext://sys.stdout" + } + }, + "loggers": { + "": { + "handlers": [ + "default" + ], + "level": "DEBUG", + "propagate": false + }, + # used by roundup.* loggers + "roundup": { + "handlers": [ + "roundup" + ], + "level": "DEBUG", + "propagate": false # note pytest testing with caplog requires + # this to be true + }, + "roundup.hyperdb": { + "handlers": [ + "roundup" + ], + "level": "INFO", # can be a little noisy use INFO for production + "propagate": false + }, + "roundup.wsgi": { # using the waitress framework + "handlers": [ + "roundup" + ], + "level": "DEBUG", + "propagate": false + }, + "roundup.wsgi.translogger": { # httpd style logging + "handlers": [ + "access" + ], + "level": "DEBUG", + "propagate": false + }, + "root": { + "handlers": [ + "default" + ], + "level": "DEBUG", + "propagate": false + } + } + } + +fileConfig Based Logging Config +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The file config is an older and more limited method of configuring +logging. It is described by the `Configuration file format +<https://docs.python.org/3/library/logging.config.html#configuration-file-format>`_ +in the Python documentation. The file name in the tracker's config for +the ``logging`` -> ``config`` setting must end with ``.ini`` to choose +the correct processing. + +Example fileConfig LoggingConfig +................................ + +This is an example .ini used with roundup-server configured to use +``roundup.http`` channel. It also includes some custom logging +qualnames/tags/channels for logging schema/permission detector and +extension output:: + + [loggers] + #other keys: roundup.hyperdb.backend + keys=root,roundup,roundup.http,roundup.hyperdb,actions,schema,extension,detector + + [logger_root] + #also for root where channlel is not set (NOTSET) aka all + level=DEBUG + handlers=rotate + + [logger_roundup] + # logger for all roundup.* not otherwise configured + level=DEBUG + handlers=rotate + qualname=roundup + propagate=0 + + [logger_roundup.http] + level=INFO + handlers=rotate_weblog + qualname=roundup.http + propagate=0 + + [logger_roundup.hyperdb] + level=WARNING + handlers=rotate + qualname=roundup.hyperdb + propagate=0 + + [logger_actions] + level=INFO + handlers=rotate + qualname=actions + propagate=0 + + [logger_detector] + level=INFO + handlers=rotate + qualname=detector + propagate=0 + + [logger_schema] + level=DEBUG + handlers=rotate + qualname=schema + propagate=0 + + [logger_extension] + level=INFO + handlers=rotate + qualname=extension + propagate=0 + + [handlers] + keys=basic,rotate,rotate_weblog + + [handler_basic] + class=StreamHandler + args=(sys.stderr,) + formatter=basic + + [handler_rotate] + class=logging.handlers.RotatingFileHandler + args=('roundup.log','a', 5120000, 2) + formatter=basic + + [handler_rotate_weblog] + class=logging.handlers.RotatingFileHandler + args=('httpd.log','a', 1024000, 2) + formatter=plain + + [formatters] + keys=basic,plain + + [formatter_basic] + format=%(asctime)s %(process)d %(name)s:%(module)s.%(funcName)s,%(levelname)s: %(message)s + datefmt=%Y-%m-%d %H:%M:%S + + [formatter_plain] + format=%(process)d %(message)s + Configuring roundup-server ==========================
--- a/doc/upgrading.txt Sun Aug 17 16:47:21 2025 -0400 +++ b/doc/upgrading.txt Tue Aug 19 22:32:46 2025 -0400 @@ -133,6 +133,21 @@ See :ref:`Confirming the User` in the reference manual for details. +Support for dictConfig Logging Configuration (optional) +------------------------------------------------------- + +Roundup's basic log configuration via config.ini has always had the +ability to use an ini style logging configuration to set levels per +log channel, control output file rotation etc. + +With Roundup 2.6 you can use a JSON like file to configure logging +using `dictConfig +<https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig>`_. The +JSON file format as been enhanced to support comments that are +stripped before being processed by the logging system. + +You can read about the details in the :ref:`admin manual <dictLogConfig>`. + .. index:: Upgrading; 2.4.0 to 2.5.0 Migrating from 2.4.0 to 2.5.0
--- a/roundup/configuration.py Sun Aug 17 16:47:21 2025 -0400 +++ b/roundup/configuration.py Tue Aug 19 22:32:46 2025 -0400 @@ -43,6 +43,16 @@ return self.args[0] +class LoggingConfigError(ConfigurationError): + def __init__(self, message, **attrs): + super().__init__(message) + for key, value in attrs.items(): + self.__setattr__(key, value) + + def __str__(self): + return self.args[0] + + class NoConfigError(ConfigurationError): """Raised when configuration loading fails @@ -2330,14 +2340,97 @@ self.detectors.reset() self.init_logging() + def load_config_dict_from_json_file(self, filename): + import json + comment_re = re.compile( + r"""^\s*\#.* # comment at beginning of line possibly indented. + | # or + ^(.*)\s\s\s\#.* # comment char preceeded by at least three spaces. + """, re.VERBOSE) + + config_list = [] + with open(filename) as config_file: + for line in config_file: + match = comment_re.search(line) + if match: + if match.lastindex: + config_list.append(match.group(1) + "\n") + else: + # insert blank line for comment line to + # keep line numbers in sync. + config_list.append("\n") + continue + config_list.append(line) + + try: + config_dict = json.loads("".join(config_list)) + except json.decoder.JSONDecodeError as e: + error_at_doc_line = e.lineno + # subtract 1 - zero index on config_list + # remove '\n' for display + line = config_list[error_at_doc_line - 1][:-1] + + hint = "" + if line.find('#') != -1: + hint = "\nMaybe bad inline comment, 3 spaces needed before #." + + raise LoggingConfigError( + 'Error parsing json logging dict (%(file)s) ' + 'near \n\n %(line)s\n\n' + '%(msg)s: line %(lineno)s column %(colno)s.%(hint)s' % + {"file": filename, + "line": line, + "msg": e.msg, + "lineno": error_at_doc_line, + "colno": e.colno, + "hint": hint}, + config_file=self.filepath, + source="json.loads" + ) + + return config_dict + def init_logging(self): _file = self["LOGGING_CONFIG"] - if _file and os.path.isfile(_file): + if _file and os.path.isfile(_file) and _file.endswith(".ini"): logging.config.fileConfig( _file, disable_existing_loggers=self["LOGGING_DISABLE_LOGGERS"]) return + if _file and os.path.isfile(_file) and _file.endswith(".json"): + config_dict = self.load_config_dict_from_json_file(_file) + try: + logging.config.dictConfig(config_dict) + except ValueError as e: + # docs say these exceptions: + # ValueError, TypeError, AttributeError, ImportError + # could be raised, but + # looking through the code, it looks like + # configure() maps all exceptions (including + # ImportError, TypeError) raised by functions to + # ValueError. + context = "No additional information available." + if hasattr(e, '__context__') and e.__context__: + # get additional error info. E.G. if INFO + # is replaced by MANGO, context is: + # ValueError("Unknown level: 'MANGO'") + # while str(e) is "Unable to configure handler 'access'" + context = e.__context__ + + raise LoggingConfigError( + 'Error loading logging dict from %(file)s.\n' + '%(msg)s\n%(context)s\n' % { + "file": _file, + "msg": type(e).__name__ + ": " + str(e), + "context": context + }, + config_file=self.filepath, + source="dictConfig" + ) + + return + _file = self["LOGGING_FILENAME"] # set file & level on the roundup logger logger = logging.getLogger('roundup')
--- a/test/test_config.py Sun Aug 17 16:47:21 2025 -0400 +++ b/test/test_config.py Tue Aug 19 22:32:46 2025 -0400 @@ -25,6 +25,7 @@ import unittest from os.path import normpath +from textwrap import dedent from roundup import configuration from roundup.backends import get_backend, have_backend @@ -1046,3 +1047,197 @@ print(string_rep) self.assertIn("nati", string_rep) self.assertIn("'whoosh'", string_rep) + + def testDictLoggerConfigViaJson(self): + + # test case broken, comment on version line misformatted + config1 = dedent(""" + { + "version": 1, # only supported version + "disable_existing_loggers": false, # keep the wsgi loggers + + "formatters": { + # standard Roundup formatter including context id. + "standard": { + "format": "%(asctime)s %(levelname)s %(name)s:%(module)s %(msg)s" + }, + # used for waitress wsgi server to produce httpd style logs + "http": { + "format": "%(message)s" + } + }, + "handlers": { + # create an access.log style http log file + "access": { + "level": "INFO", + "formatter": "http", + "class": "logging.FileHandler", + "filename": "demo/access.log" + }, + # logging for roundup.* loggers + "roundup": { + "level": "DEBUG", + "formatter": "standard", + "class": "logging.FileHandler", + "filename": "demo/roundup.log" + }, + # print to stdout - fall through for other logging + "default": { + "level": "DEBUG", + "formatter": "standard", + "class": "logging.StreamHandler", + "stream": "ext://sys.stdout" + } + }, + "loggers": { + "": { + "handlers": [ + "default" # used by wsgi/usgi + ], + "level": "DEBUG", + "propagate": false + }, + # used by roundup.* loggers + "roundup": { + "handlers": [ + "roundup" + ], + "level": "DEBUG", + "propagate": false # note pytest testing with caplog requires + # this to be true + }, + "roundup.hyperdb": { + "handlers": [ + "roundup" + ], + "level": "INFO", # can be a little noisy INFO for production + "propagate": false + }, + "roundup.wsgi": { # using the waitress framework + "handlers": [ + "roundup" + ], + "level": "DEBUG", + "propagate": false + }, + "roundup.wsgi.translogger": { # httpd style logging + "handlers": [ + "access" + ], + "level": "DEBUG", + "propagate": false + }, + "root": { + "handlers": [ + "default" + ], + "level": "DEBUG", + "propagate": false + } + } + } + """) + + log_config_filename = self.instance.tracker_home \ + + "/_test_log_config.json" + + # happy path + with open(log_config_filename, "w") as log_config_file: + log_config_file.write(config1) + + config = self.db.config.load_config_dict_from_json_file( + log_config_filename) + self.assertIn("version", config) + self.assertEqual(config['version'], 1) + + # broken inline comment misformatted + test_config = config1.replace(": 1, #", ": 1, #") + with open(log_config_filename, "w") as log_config_file: + log_config_file.write(test_config) + + with self.assertRaises(configuration.LoggingConfigError) as cm: + config = self.db.config.load_config_dict_from_json_file( + log_config_filename) + self.assertEqual( + cm.exception.args[0], + ('Error parsing json logging dict ' + '(_test_instance/_test_log_config.json) near \n\n ' + '"version": 1, # only supported version\n\nExpecting ' + 'property name enclosed in double quotes: line 3 column 18.\n' + 'Maybe bad inline comment, 3 spaces needed before #.') + ) + + # broken trailing , on last dict element + test_config = config1.replace(' "ext://sys.stdout"', + ' "ext://sys.stdout",' + ) + with open(log_config_filename, "w") as log_config_file: + log_config_file.write(test_config) + + with self.assertRaises(configuration.LoggingConfigError) as cm: + config = self.db.config.load_config_dict_from_json_file( + log_config_filename) + self.assertEqual( + cm.exception.args[0], + ('Error parsing json logging dict ' + '(_test_instance/_test_log_config.json) near \n\n' + ' }\n\nExpecting property name enclosed in double ' + 'quotes: line 37 column 6.') + ) + + # happy path for init_logging() + + # verify preconditions + logger = logging.getLogger("roundup") + self.assertEqual(logger.level, 40) # error default from config.ini + self.assertEqual(logger.filters, []) + + with open(log_config_filename, "w") as log_config_file: + log_config_file.write(config1) + + # file is made relative to tracker dir. + self.db.config["LOGGING_CONFIG"] = '_test_log_config.json' + config = self.db.config.init_logging() + self.assertIs(config, None) + + logger = logging.getLogger("roundup") + self.assertEqual(logger.level, 10) # debug + self.assertEqual(logger.filters, []) + + # broken invalid format type (int not str) + test_config = config1.replace('"format": "%(message)s"', + '"format": 1234',) + with open(log_config_filename, "w") as log_config_file: + log_config_file.write(test_config) + + # file is made relative to tracker dir. + self.db.config["LOGGING_CONFIG"] = '_test_log_config.json' + with self.assertRaises(configuration.LoggingConfigError) as cm: + config = self.db.config.init_logging() + self.assertEqual( + cm.exception.args[0], + ('Error loading logging dict from ' + '_test_instance/_test_log_config.json.\n' + "ValueError: Unable to configure formatter 'http'\n" + 'expected string or bytes-like object\n') + ) + + # broken invalid level MANGO + test_config = config1.replace( + ': "INFO", # can', + ': "MANGO", # can') + with open(log_config_filename, "w") as log_config_file: + log_config_file.write(test_config) + + # file is made relative to tracker dir. + self.db.config["LOGGING_CONFIG"] = '_test_log_config.json' + with self.assertRaises(configuration.LoggingConfigError) as cm: + config = self.db.config.init_logging() + self.assertEqual( + cm.exception.args[0], + ("Error loading logging dict from " + "_test_instance/_test_log_config.json.\nValueError: " + "Unable to configure logger 'roundup.hyperdb'\nUnknown level: " + "'MANGO'\n") + + )