[TODO]: Revise logging integrations #4718

Recently a PR was merged for a pflogsumm daily cron job task, which addressed a concern with piping the active log and all rotated copies into the command.

The concern was old logs from over a year were being reported as part of the daily filtered logs as the pflogsumm task invoked uses -d yesterday. The solution was to filter the logs to pipe into pflogsumm with a 180 days age filter on the log files that'd be piped to the command, but this duration was not clearly justified nor raised as a concern during review?

I looked into it and reproduced the issue, where the actual problem was from logs prior to DMS v14, where RFC 3164 timestamp format was used which did not include a year for pflogsumm to filter by, hence the old logs qualified for -d yesterday if the day+month matched.

The fix only resolved it for the PR author, but this concern would also apply to logwatch? (EDIT: logwatch is fine and selects by mtime of archived logs based on the --range query given)

Historically there's been some issues and decisions/discussions on these features, which I don't presently have the time to detail as much as I'd like to cover.

• A concern was expressed with a race condition of the cron job timing between logrotate and logwatch or pflogsumm tasks, the log reporting commands shouldn't be invoked until after log rotation has completed.
• When that is ensured, we should have a rather reliable /var/mail/mail.log.1 that could be aligned with the interval (daily/weekly)?

The daily cron job pflogsumm was introduced specifically to not be dependent upon logrotate (and to introduce the -d yesterday report to avoid any overlap between runs related to processing /var/log/mail/mail.log, whereas with logrotate and the specific newly rotated log file, this was not a concern). Presumably there was a desire to have disjoint intervals between log rotation and reporting, but that is unclear.

All this aside, the only notable issues we've had that I can recall regarding logs as of late is:

• A single user raising a PR to resolve an issue with non-standard DMS config, due to a change from approx 2 years ago prior to the PR, where our logs didn't include a year in the timestamps. Specifically pflogsumm.
• A single user raising a PR to ignore noisy Dovecot FTS indexing logs from being reported by logwatch.

Thus this isn't a major concern to resolve, but I'm not fond of arbitrary workarounds being introduced, these accumulate and make maintaining such things a mess. Lax review that doesn't take this into consideration is also how we get these problems (of which so many were introduced so long ago).

Tasks:

• Identify why the REPORT_ sender/recipient ENV were deemed deprecated (these are still used as fallbacks/default in our startup scripts, the PR that introduced logstash in Sep 2019(?) introduced the discrete service variations instead of unifying). Possibly worth unifying on again and removing the service specific vars instead.
• Consider locking the log rotation interval to the log reporter(s) interval? That or a better thought out approach to exclude old logs than an arbitrary mtime filter of 180 days. (Done: #4709 comment)
• Determine how useful it is to support pflogsumm vs logwatch. We only seem to have docs about logging vaguely covered in ENV section?
• Confirm if the logwatch weekly range syntax is chosen instead of 7 days ago due to skipping the active day? (overlap concerns?)
• logwatch config is more layered, there are paths configured by Debian that are effectively symlinks now (in addition to our own supplied maillog.conf for DMS /var/log/mail/ services), this seems prone to producing reports doubling statistics, which is obviously a bug that should be resolved (unset Debian rules via LogFile = (no value assigned), followed by LogFile assignments for relevant entries only). The DMS maillog.conf could be redundant if instead adding that directory to LogDir in logwatch.conf instead? (Then any relative paths like mail.log configured by Debian would be sorted?) Presently the /var/log/mail.log symlink does result in double counts for metrics reported as /var/log/mail/mail.log gets processed twice, only relevant if this log file matches the range query.
• Probably more concerns?
  • When I was producing the reproduction script for the problem caused by RFC 3164 timestamps, I recall logwatch logging a warning about duplicate cron.log files in it's config?
  • Some of the config changes for DMS in the log.sh setup script or Dockerfile probably also need review. Along with ensuring that any assumptions to modifications or removal of files remain relevant with latest Debian 13.

Besides what has been detailed above, I have noticed other concerns with logging in other components of DMS, some of which may be better resolved if adopting Vector (which may make rsyslog redundant).