Skip to content

[TODO]: Refactor parallel feature support for Fetchmail / Getmail #4713

Description

@polarathene

Presently DMS implements multiple supervisord services for Fetchmail when the DMS ENV FETCHMAIL_PARALLEL=1 is used. This is to enable support for the IDLE extension in the IMAP protocol and requires running multiple instances of Fetchmail, 1 per mailbox account (with only a single mailbox/folder to retrieve from, default INBOX).

Reference - Fetchmail compose.yaml

For context so there's no need to piece yourself a config to experiment with:

Docker Compose example (fetchmail + idle)
services:
  dms-local:
    image: ghcr.io/docker-mailserver/docker-mailserver:15.1
    hostname: mail.example.test
    environment:
      ENABLE_FETCHMAIL: 1
      # This setting splits the fetchmail config by `poll` lines,
      # each with their own supervisord service created.
      FETCHMAIL_PARALLEL: 1
      # Lowered polling from default 300s for quick testing feedback:
      # NOTE: Polling is irrelevant for any configs using `idle`.
      FETCHMAIL_POLL: 10
      # Only required if the sender address used at the remote cannot resolve:
      # This is because fetchmail is attempting to deliver the mail through standard
      # SMTP, however Postfix security checks are still applied (without this workaround).
      PERMIT_DOCKER: container
    configs:
      - source: dms-accounts
        target: /tmp/docker-mailserver/postfix-accounts.cf
      - source: fetchmail
        target: /tmp/docker-mailserver/fetchmail.cf

  # This would represent an external mail service like Gmail:
  dms-remote:
    image: ghcr.io/docker-mailserver/docker-mailserver:15.1
    hostname: mail.remote.test
    environment:
      # Allows for us send a test mail easily by trusting any mail client run within this container (`swaks`):
      PERMIT_DOCKER: container
      # Required to enable (if testing spam delivery to/from the `Junk` mailbox).
      # `dms-local` will pull mail from Alice's junk mail and then store a copy
      # into John's junk mailbox (due to default sieve scripts paired with headers added by SpamAssassin).
      ENABLE_SPAMASSASSIN: 1
    configs:
      - source: dms-accounts-remote
        target: /tmp/docker-mailserver/postfix-accounts.cf
      # This file is copied over to `/var/mail/remote.test/alice/home/.dovecot.sieve` during container setup:
      - source: sieve
        target: /tmp/docker-mailserver/alice@remote.test.dovecot.sieve


# Using the Docker Compose `configs.content` feature instead of volume mounting separate files.
# NOTE: This feature requires Docker Compose v2.23.1 (Nov 2023) or newer:
# https://github.com/compose-spec/compose-spec/pull/446
configs:
  # Fetchmail config docs:
  # https://www.fetchmail.info/fetchmail-man.html
  # https://docker-mailserver.github.io/docker-mailserver/v15.1/config/advanced/mail-fetchmail/
  #
  # Config to remotely monitor the two accounts and copy remote mail into the local `john.doe` mailbox.
  # NOTE: `no sslcertck` used as this compose example has no TLS certs configured for DMS.
  #
  # Default delivery is via SMTP (via default: `smtphost localhost/25`_) to the `is` recipient address.
  # Alternatives require either unix socket write access (LMTP) or execute permissions (CLI),
  # these would require modifications in DMS or running Fetchmail as root.
  # Consult upstream docs before deciding to pursue either alternative:
  # - LMTP protocol (skip Postfix, direct to Dovecot): `smtphost /var/run/dovecot/lmtp`
  # - MDA/LDA command (CLI submission to Dovecot): `mda "/usr/lib/dovecot/deliver -d %T"`
  # Another option would be SMTP (Postfix) with trust established to relax security,
  # be mindful of Anti-(virus/spam) or similar integrations impact on delivery.
  fetchmail:
    content: |
      poll 'mail.remote.test' proto imap
        user 'jane.doe@remote.test' pass 'secret'
        is 'john.doe@example.test'
        idle
        no sslcertck

      # The `folders` setting (default: INBOX) can monitor multiple remote mailboxes:
      # This is necessary if other mailboxes of interest exist at the remote.
      # NOTE: Mailbox hierarchy does not result in recursively pulling from child mailboxes,
      #       they must be explicitly configured instead.
      # NOTE: When using `idle`, the `folders` setting must only specify at most 1 mailbox,
      #       otherwise the fetchmail command will fail to start (validation failure).
      poll 'mail.remote.test' proto imap
        user 'alice@remote.test' pass 'secret'
        folders 'INBOX', 'Junk', 'INBOX.nested-mailbox'
        is 'john.doe@example.test'
        no sslcertck

  # When the subaddress `+sieve-example` is used, store mail in a mailbox (folder) nested under `INBOX` mailbox:
  sieve:
    content: |
      require ["envelope", "fileinto", "mailbox", "subaddress", "variables"];

      # Check if the mail recipient address uses the expected subaddress tag (sourced from `:detail`)
      if envelope :detail :matches "to" "sieve-example" {

        if mailboxexists "INBOX.nested-mailbox" {
          fileinto "INBOX.nested-mailbox";
        } else {
          fileinto :create "INBOX.nested-mailbox";
        }
      }

  # DMS requires an account to complete setup, configure at least one account for each DMS container:
  # NOTE: All accounts are configured with the same example password (`secret`, SHA512-CRYPT hashed)
  # NOTE: `$$` is an escaped `$` (opt-out of pre-processing for environment variable interpolation)
  dms-accounts:
    content: |
      john.doe@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.

  dms-accounts-remote:
    content: |
      jane.doe@remote.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
      alice@remote.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.

Mail test examples to trigger fetchmail functionality:

# Start example and shell into remote container to send test mails:
docker compose up -d --force-recreate
docker compose exec dms-remote bash

# Test against IMAP idle monitored mailbox:
swaks --server localhost --from hello@somewhere.test --to jane.doe@remote.test

# Junk mail example:
swaks --server localhost --from hello@somewhere.test --to alice@remote.test \
  --body "XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X."

# Sieve script delivers mail into `INBOX.nested-mailbox` folder:
swaks --server localhost --from hello@somewhere.test --to alice+sieve-example@remote.test

supervisord support for multiple service instances

The supervisord side has separate service configs created by this script logic, which is slightly out of sync from the default fetchmail service config (single instance).

We can avoid that script logic in favor of supervisord's own parallel support (described in it's docs):

  • numprocs (default: 1) can be increased to opt-in to the parallel service support. This has an associated expression %(process_num)s that each instance of the config will use for it's own process number (incremented from 0 or value of numprocs_start), which must be used within the process_name setting (defaults to %(program_name)s).

  • As such we make small change to the supervisord config (add suffix: -%(process_num)s and define a few extra settings):

    + numprocs=2
    + numprocs_start=1
    + process_name=%(program_name)s-%(process_num)s
    # NOTE: Multi-line values are valid with white-space indentation (like Postfix config)
    command=/usr/bin/fetchmail --verbose --nodetach
      --idfile /var/lib/fetchmail/.fetchids
    - --fetchmailrc /etc/fetchmailrc.d/%(program_name)s.rc
    + --fetchmailrc /etc/fetchmailrc.d/%(program_name)s-%(process_num)s.rc
    - --pidfile /var/run/fetchmail/%(program_name)s.pid
    + --pidfile /var/run/fetchmail/%(program_name)s-%(process_num)s.pid

    NOTE: I excluded the --daemon %(ENV_FETCHMAIL_POLL)s arg here as there's a base config in DMS already setting daemon 300, it'd be wiser for DMS to just modify that value instead than enforce polling time in the service command args.

    Usage via supervisorctl:

    # Inform supervisord of updated config (affected running services are stopped)
    # NOTE: Don't use `supervisorctl reload` when supervisor is effectively PID1, it'll crash the container.
    $ supervisorctl update
    
    # Start the process group, running all child services via appending `:*`:
    # As the output shows, there is no longer a plain `fetchmail` service anymore.
    $ supervisorctl start fetchmail:*
    fetchmail:fetchmail-1: started
    fetchmail:fetchmail-2: started
    
    $ supervisorctl status fetchmail
    fetchmail: ERROR (no such process)
    
    $ supervisorctl status 'fetchmail:*'
    fetchmail:fetchmail-1            RUNNING   pid 5715, uptime 0:06:12
    fetchmail:fetchmail-2            RUNNING   pid 5716, uptime 0:06:12
  • This switch to leveraging supervisord's own support for running multiple instances of a service will also simplify our related startup logic as shown above to just _default_start_daemon 'fetchmail:*' 🥳 (also related change to dms-healthcheck for status checks and supervisord support refactor)

NOTE: While Fetchmail does support an --idle CLI option, it's easier for the user to just opt-in via idle in their individual account poll configs where that is needed.

All this requires for FETCHMAIL_PARALLEL=1 support is to adjust numprocs as necessary once the config split processing is done. If wanting to retain the fetchmail service without any numbered suffix involved, a simpler approach would be to have single vs multi config and just use the one based on FETCHMAIL_PARALLEL enabled status.

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions