Puma 5.6.5 with SystemD - #52240
Conversation
…estart things with pumactl
Hamms
added
the
Ruby Update
|
|
||
| # We always want to restart the web server process whenever port/socket | ||
| # listener configuration is changed, so create a file with that information | ||
| # as its contents and invoke the restart whenever those contents change. |
There was a problem hiding this comment.
We specify listener via code and config that in practice only changes with a release and an invocation of our build process (which stops/starts the service). Do we need to support this use case?
There was a problem hiding this comment.
It might not be strictly necessary, but building in protections for execution scenarios other than a full ci_build is still good practice. We don't want our Chef scripts to break unless they're run in a particular way by a particular external build process; they should be internally consistent.
|
Is it important to test the persistent server use case? Provision an adhoc from a feature branch off of the staging branch and then merge this change into that feature branch to ensure the adhoc rebuilds itself smoothly with the new SystemD service? Or maybe you already ran this test. I wasn’t sure based on the details in the Testing Story. |
sureshc
left a comment
sureshc
left a comment
There was a problem hiding this comment.
Assuming we have a good plan for how this change gets applied to existing managed servers, this looks great!
|
Ah, good callout; I haven't tested that scenario yet, and it does seem worth doing. I'll do that before merging. Although, I will point out that so far that approach hasn't been sufficient to catch many of the issues that arise in practice with upgrading our persistent servers. Drift and accumulated one-off operations that have been applied to the servers outside the context of Chef are rampant. Still worth doing just in case, though! |
3 participants
Reverts #52045, restoring #51929
Daemonization was removed in Puma 5 without replacement, so in addition to updating the Puma gem to 5.6.5 we also refactor our process monitoring. Before, we were using a SystemD service to manage a SysV init script which executes
pumain daemon mode and handles things like sending a restart signal manually. Now, we simply use SystemD to directly manage a non-daemonized puma process.The main piece of functionality we lose out on with this approach is Puma's "hot restart", which we had previously implemented manually in the SysV init script with a
SIGUSR2signal. We could replace our existing functionality with an invocation ofpumactl restart, but after talking it through with @sureshc I'm convinced we don't actually need any fancy restart support here so we instead simply perform a hard restart of the entire SystemD service whenever we need to update. Note that this does mean that any requests made to the server while it is restarting will now immediately fail with a 502 error; previously, those requests would hang until they either time out or the restart finishes, whichever comes first. This is probably a less-desirable behavior overall, but because it only impacts our content and build servers and not production, we've decided it's an acceptable loss.Links
Testing story
Tested on an adhoc to verify that everything starts up as expected, and can be successfully updated with subsequent builds.
Comparative analysis of
chef-bootstrap-debug.logindicates that this approach does result in us doing a few more restarts and spending a bit more time waiting than in our current implementation, but at the cost of only 1-3 minutes of time over the course of a 2+ hour build process.Deployment strategy
This update should apply cleanly to our existing persistent managed servers, but we will need to manually remove any artifacts of the old implementation that we no longer want. In particular, the
/etc/init.d/{dashboard,pegasus}scripts.