Presently DMS implements multiple supervisord services for Fetchmail when the DMS ENV FETCHMAIL_PARALLEL=1 is used. This is to enable support for the IDLE extension in the IMAP protocol and requires running multiple instances of Fetchmail, 1 per mailbox account (with only a single mailbox/folder to retrieve from, default INBOX).
Reference - Fetchmail compose.yaml
For context so there's no need to piece yourself a config to experiment with:
Docker Compose example (fetchmail + idle)
services:
dms-local:
image: ghcr.io/docker-mailserver/docker-mailserver:15.1
hostname: mail.example.test
environment:
ENABLE_FETCHMAIL: 1
# This setting splits the fetchmail config by `poll` lines,
# each with their own supervisord service created.
FETCHMAIL_PARALLEL: 1
# Lowered polling from default 300s for quick testing feedback:
# NOTE: Polling is irrelevant for any configs using `idle`.
FETCHMAIL_POLL: 10
# Only required if the sender address used at the remote cannot resolve:
# This is because fetchmail is attempting to deliver the mail through standard
# SMTP, however Postfix security checks are still applied (without this workaround).
PERMIT_DOCKER: container
configs:
- source: dms-accounts
target: /tmp/docker-mailserver/postfix-accounts.cf
- source: fetchmail
target: /tmp/docker-mailserver/fetchmail.cf
# This would represent an external mail service like Gmail:
dms-remote:
image: ghcr.io/docker-mailserver/docker-mailserver:15.1
hostname: mail.remote.test
environment:
# Allows for us send a test mail easily by trusting any mail client run within this container (`swaks`):
PERMIT_DOCKER: container
# Required to enable (if testing spam delivery to/from the `Junk` mailbox).
# `dms-local` will pull mail from Alice's junk mail and then store a copy
# into John's junk mailbox (due to default sieve scripts paired with headers added by SpamAssassin).
ENABLE_SPAMASSASSIN: 1
configs:
- source: dms-accounts-remote
target: /tmp/docker-mailserver/postfix-accounts.cf
# This file is copied over to `/var/mail/remote.test/alice/home/.dovecot.sieve` during container setup:
- source: sieve
target: /tmp/docker-mailserver/alice@remote.test.dovecot.sieve
# Using the Docker Compose `configs.content` feature instead of volume mounting separate files.
# NOTE: This feature requires Docker Compose v2.23.1 (Nov 2023) or newer:
# https://github.com/compose-spec/compose-spec/pull/446
configs:
# Fetchmail config docs:
# https://www.fetchmail.info/fetchmail-man.html
# https://docker-mailserver.github.io/docker-mailserver/v15.1/config/advanced/mail-fetchmail/
#
# Config to remotely monitor the two accounts and copy remote mail into the local `john.doe` mailbox.
# NOTE: `no sslcertck` used as this compose example has no TLS certs configured for DMS.
#
# Default delivery is via SMTP (via default: `smtphost localhost/25`_) to the `is` recipient address.
# Alternatives require either unix socket write access (LMTP) or execute permissions (CLI),
# these would require modifications in DMS or running Fetchmail as root.
# Consult upstream docs before deciding to pursue either alternative:
# - LMTP protocol (skip Postfix, direct to Dovecot): `smtphost /var/run/dovecot/lmtp`
# - MDA/LDA command (CLI submission to Dovecot): `mda "/usr/lib/dovecot/deliver -d %T"`
# Another option would be SMTP (Postfix) with trust established to relax security,
# be mindful of Anti-(virus/spam) or similar integrations impact on delivery.
fetchmail:
content: |
poll 'mail.remote.test' proto imap
user 'jane.doe@remote.test' pass 'secret'
is 'john.doe@example.test'
idle
no sslcertck
# The `folders` setting (default: INBOX) can monitor multiple remote mailboxes:
# This is necessary if other mailboxes of interest exist at the remote.
# NOTE: Mailbox hierarchy does not result in recursively pulling from child mailboxes,
# they must be explicitly configured instead.
# NOTE: When using `idle`, the `folders` setting must only specify at most 1 mailbox,
# otherwise the fetchmail command will fail to start (validation failure).
poll 'mail.remote.test' proto imap
user 'alice@remote.test' pass 'secret'
folders 'INBOX', 'Junk', 'INBOX.nested-mailbox'
is 'john.doe@example.test'
no sslcertck
# When the subaddress `+sieve-example` is used, store mail in a mailbox (folder) nested under `INBOX` mailbox:
sieve:
content: |
require ["envelope", "fileinto", "mailbox", "subaddress", "variables"];
# Check if the mail recipient address uses the expected subaddress tag (sourced from `:detail`)
if envelope :detail :matches "to" "sieve-example" {
if mailboxexists "INBOX.nested-mailbox" {
fileinto "INBOX.nested-mailbox";
} else {
fileinto :create "INBOX.nested-mailbox";
}
}
# DMS requires an account to complete setup, configure at least one account for each DMS container:
# NOTE: All accounts are configured with the same example password (`secret`, SHA512-CRYPT hashed)
# NOTE: `$$` is an escaped `$` (opt-out of pre-processing for environment variable interpolation)
dms-accounts:
content: |
john.doe@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
dms-accounts-remote:
content: |
jane.doe@remote.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
alice@remote.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
Mail test examples to trigger fetchmail functionality:
# Start example and shell into remote container to send test mails:
docker compose up -d --force-recreate
docker compose exec dms-remote bash
# Test against IMAP idle monitored mailbox:
swaks --server localhost --from hello@somewhere.test --to jane.doe@remote.test
# Junk mail example:
swaks --server localhost --from hello@somewhere.test --to alice@remote.test \
--body "XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X."
# Sieve script delivers mail into `INBOX.nested-mailbox` folder:
swaks --server localhost --from hello@somewhere.test --to alice+sieve-example@remote.test
supervisord support for multiple service instances
The supervisord side has separate service configs created by this script logic, which is slightly out of sync from the default fetchmail service config (single instance).
We can avoid that script logic in favor of supervisord's own parallel support (described in it's docs):
-
numprocs (default: 1) can be increased to opt-in to the parallel service support. This has an associated expression %(process_num)s that each instance of the config will use for it's own process number (incremented from 0 or value of numprocs_start), which must be used within the process_name setting (defaults to %(program_name)s).
-
As such we make small change to the supervisord config (add suffix: -%(process_num)s and define a few extra settings):
+ numprocs=2
+ numprocs_start=1
+ process_name=%(program_name)s-%(process_num)s
# NOTE: Multi-line values are valid with white-space indentation (like Postfix config)
command=/usr/bin/fetchmail --verbose --nodetach
--idfile /var/lib/fetchmail/.fetchids
- --fetchmailrc /etc/fetchmailrc.d/%(program_name)s.rc
+ --fetchmailrc /etc/fetchmailrc.d/%(program_name)s-%(process_num)s.rc
- --pidfile /var/run/fetchmail/%(program_name)s.pid
+ --pidfile /var/run/fetchmail/%(program_name)s-%(process_num)s.pid
NOTE: I excluded the --daemon %(ENV_FETCHMAIL_POLL)s arg here as there's a base config in DMS already setting daemon 300, it'd be wiser for DMS to just modify that value instead than enforce polling time in the service command args.
Usage via supervisorctl:
# Inform supervisord of updated config (affected running services are stopped)
# NOTE: Don't use `supervisorctl reload` when supervisor is effectively PID1, it'll crash the container.
$ supervisorctl update
# Start the process group, running all child services via appending `:*`:
# As the output shows, there is no longer a plain `fetchmail` service anymore.
$ supervisorctl start fetchmail:*
fetchmail:fetchmail-1: started
fetchmail:fetchmail-2: started
$ supervisorctl status fetchmail
fetchmail: ERROR (no such process)
$ supervisorctl status 'fetchmail:*'
fetchmail:fetchmail-1 RUNNING pid 5715, uptime 0:06:12
fetchmail:fetchmail-2 RUNNING pid 5716, uptime 0:06:12
-
This switch to leveraging supervisord's own support for running multiple instances of a service will also simplify our related startup logic as shown above to just _default_start_daemon 'fetchmail:*' 🥳 (also related change to dms-healthcheck for status checks and supervisord support refactor)
NOTE: While Fetchmail does support an --idle CLI option, it's easier for the user to just opt-in via idle in their individual account poll configs where that is needed.
All this requires for FETCHMAIL_PARALLEL=1 support is to adjust numprocs as necessary once the config split processing is done. If wanting to retain the fetchmail service without any numbered suffix involved, a simpler approach would be to have single vs multi config and just use the one based on FETCHMAIL_PARALLEL enabled status.
Presently DMS implements multiple supervisord services for Fetchmail when the DMS ENV
FETCHMAIL_PARALLEL=1is used. This is to enable support for the IDLE extension in the IMAP protocol and requires running multiple instances of Fetchmail, 1 per mailbox account (with only a single mailbox/folder to retrieve from, defaultINBOX).Reference - Fetchmail
compose.yamlFor context so there's no need to piece yourself a config to experiment with:
Docker Compose example (fetchmail + idle)
Mail test examples to trigger fetchmail functionality:
supervisordsupport for multiple service instancesThe supervisord side has separate service configs created by this script logic, which is slightly out of sync from the default
fetchmailservice config (single instance).We can avoid that script logic in favor of supervisord's own parallel support (described in it's docs):
numprocs(default:1) can be increased to opt-in to the parallel service support. This has an associated expression%(process_num)sthat each instance of the config will use for it's own process number (incremented from0or value ofnumprocs_start), which must be used within theprocess_namesetting (defaults to%(program_name)s).As such we make small change to the supervisord config (add suffix:
-%(process_num)sand define a few extra settings):NOTE: I excluded the
--daemon %(ENV_FETCHMAIL_POLL)sarg here as there's a base config in DMS already settingdaemon 300, it'd be wiser for DMS to just modify that value instead than enforce polling time in the service command args.Usage via
supervisorctl:This switch to leveraging supervisord's own support for running multiple instances of a service will also simplify our related startup logic as shown above to just
_default_start_daemon 'fetchmail:*'🥳 (also related change todms-healthcheckfor status checks andsupervisordsupport refactor)NOTE: While Fetchmail does support an
--idleCLI option, it's easier for the user to just opt-in viaidlein their individual accountpollconfigs where that is needed.All this requires for
FETCHMAIL_PARALLEL=1support is to adjustnumprocsas necessary once the config split processing is done. If wanting to retain thefetchmailservice without any numbered suffix involved, a simpler approach would be to have single vs multi config and just use the one based onFETCHMAIL_PARALLELenabled status.