startsecs=0 (default is 1 second) has been used in DMS since supervisord was added into the project back in Aug 2017. It determines how long a service needs to be running before it's moved from the STARTING to RUNNING state.startretries=3 (default) is the number of retry attempts before the service is marked with the status FATAL and is no longer restarted. This only applies during the STARTING status prior to being assigned RUNNING (unless the service fails in this state, where supervisord will restart it, placing it back into STARTING status).
Due to the above mismatch, a service immediately enters RUNNING status, and if it then crashes with an uptime of 0 seconds the startsecs value will immediately switch from STARTING to RUNNING, there is no opportunity to increment the restart counter so it is always reset once the service is marked as RUNNING again.
For a reliable healthcheck script (as is being pursued here), we should correct this.
Known examples of this behaviour observed in DMS:
A side-effect of changing to startsecs = 1 is supervisorctl start (and potentially other sub-commands) are blocking with that delay. Any usage within DMS scripts would be affected by such, most notably during container startup where we sequentially invoke the start command per service individually. In the startup case we should probably collect the services to start and provide them all to a single supervisorctl start ... command, that should keep the impact low. Some container deployments may have a healthcheck policy that kills/restarts the container under an unhealthy state, so let's ensure the downtime is low 😅
Locations for startsecs=0 => startsecs=1 (or deletion):
Refactor invoking supervisorctl start to single use in scripts/startup/daemons-stack.sh:
Either that or a more extensive refactor. This change may also impact the test suite if any test cases are aggressive with timing expectations.
startsecs=0(default is1second) has been used in DMS sincesupervisordwas added into the project back in Aug 2017. It determines how long a service needs to be running before it's moved from theSTARTINGtoRUNNINGstate.startretries=3(default) is the number of retry attempts before the service is marked with the statusFATALand is no longer restarted. This only applies during theSTARTINGstatus prior to being assignedRUNNING(unless the service fails in this state, where supervisord will restart it, placing it back intoSTARTINGstatus).Due to the above mismatch, a service immediately enters
RUNNINGstatus, and if it then crashes with an uptime of 0 seconds thestartsecsvalue will immediately switch fromSTARTINGtoRUNNING, there is no opportunity to increment the restart counter so it is always reset once the service is marked asRUNNINGagain.For a reliable healthcheck script (as is being pursued here), we should correct this.
Known examples of this behaviour observed in DMS:
setup config dkim#4579 (rspamd)FETCHMAIL_PARALLEL=1) #4586 (fetchmail)A side-effect of changing to
startsecs = 1issupervisorctl start(and potentially other sub-commands) are blocking with that delay. Any usage within DMS scripts would be affected by such, most notably during container startup where we sequentially invoke the start command per service individually. In the startup case we should probably collect the services to start and provide them all to a singlesupervisorctl start ...command, that should keep the impact low. Some container deployments may have a healthcheck policy that kills/restarts the container under an unhealthy state, so let's ensure the downtime is low 😅Locations for
startsecs=0=>startsecs=1(or deletion):supervisor/conf.d/dms-services.confsupervisor/conf.d/saslauth.confscripts/startup/setup.d/fetchmail.shexamples/use-cases/ios-mail-push-support.mdRefactor invoking
supervisorctl startto single use inscripts/startup/daemons-stack.sh:supervisorctl startand carry over relevant logic from prior linked method.Either that or a more extensive refactor. This change may also impact the test suite if any test cases are aggressive with timing expectations.