Improve aborting daemon upon start timeout

- Upon start timeout, abort daemon in two stages: first abort gracefully (SIGTERM); if that times out (stop_abort_timeout), then abort forcefully (SIGKILL).
  Previously, we aborted the daemon by sending SIGTERM but without checking whether it actually terminates.
- Make timeout handling more robust. Instead of timing out at any point in the code (which could leave behind inconsistent state), only allow timing out in well-defined interruption points.
- Much improve test coverage.

Author: FooBarWidget

.github/workflows/ci.yaml

@@ -26,7 +26,8 @@ jobs:
           bundler-cache: true
 
       - name: Run RSpec tests
-        run: bundle exec rake test
+        run: /usr/bin/timeout -s QUIT 120 bundle exec rake test
+        timeout-minutes: 3
         env:
           MRI_RUBY: "env RUBYOPT= RUBYLIB= /usr/bin/ruby"
 

README.md

@@ -500,4 +500,5 @@ API documentation
 
 Detailed API documentation is available here:
 - [Configuration options](doc/OPTIONS.md)
+- [Stop flow](doc/STOP_FLOW.md)
 - Inline comments in `lib/daemon_controller.rb`.

doc/OPTIONS.md

@@ -71,7 +71,8 @@ Lock file to use for serializing concurrent daemon management operations.
 Command to stop the daemon with, e.g. "/etc/rc.d/nginx stop".
 
 If no stop command is given (i.e., `nil`), then will stop the daemon
-by sending signals to the PID written in the PID file.
+by sending signals to the PID written in the PID file. See
+[Stop flow](STOP-FLOW.md) for how that works.
 
 ### restart_command (default: nil)
 
@@ -89,7 +90,15 @@ The Proc call is not subject to the start timeout.
 Maximum amount of time (seconds) that #start may take to start
 the daemon. Since #start also waits until the daemon can be connected to,
 that wait time is counted as well. If the daemon does not start in time,
-then #start will raise an exception and also stop the daemon.
+then #start will raise an exception and also stop the daemon. See also
+[Stop flow](STOP-FLOW.md) for how that works.
+
+### start_abort_timeout (default: 10)
+
+Maximum amount of time (seconds) to wait for the daemon to terminate after
+sending SIGTERM during the start timeout flow. If the daemon does not terminate
+within this time, it will be forcefully terminated with SIGKILL. See
+[Stop flow](STOP-FLOW.md) for more details.
 
 ### stop_timeout (default: 30)
 
@@ -98,6 +107,8 @@ the daemon. Since #stop also waits until the daemon is no longer running,
 that wait time is counted as well. If the daemon does not stop in time,
 then #stop will raise an exception and force stop the daemon.
 
+See [Stop flow](STOP-FLOW.md) for al overview of the entire stopping flow.
+
 ### log_file_activity_timeout (default: 10)
 
 Once a daemon has gone into the background, it will become difficult to
@@ -117,6 +128,12 @@ given by this option, then the daemon is assumed to have terminated with an erro
 Time interval (seconds) between pinging attempts (see `ping_command`) when waiting
 for the daemon to start.
 
+### stop_graceful_signal (default: "TERM")
+
+Signal to send to the daemon when attempting to stop it gracefully during `#stop`
+(regular stop flow). This is only used when no `stop_command` is provided.
+See [Stop flow](STOP-FLOW.md) for more details.
+
 ### dont_stop_if_pid_file_invalid (default: false)
 
 If the stop_command option is given, then normally daemon_controller will

doc/STOP-FLOW.md

@@ -0,0 +1,19 @@
+# Stop flow
+
+## Regular stop flow
+
+This flow is triggered upon `#stop` calling stop.
+
+1. Graceful termination request: either run the stop command, or send `stop_graceful_signal` to the PID. Then wait until daemon is gone.
+   - Stop command is only invoked, or signal is only sent, if the PID file is valid or `dont_stop_if_pid_file_invalid` is true.
+2. Force termination upon timeout (`stop_timeout`): send SIGKILL to the PID. Then wait until daemon is gone, then delete the PID file.
+   - No timeout here: we assume the OS processes it quickly enough.
+
+## Start timeout stop flow
+
+This flow is triggered when `#stop` times out.
+
+1. Graceful termination request: send SIGTERM to the PID and wait until it's gone.
+   - No possibility to customize signal here. Rationale: this is an abnormal stop so we don't use the stop command.
+2. Force termination upon timeout (`start_abort_timeout`) of step 1: send SIGKILL to the PID and wait until it's gone.
+   - No timeout here: we assume the OS processes a SIGKILL quickly enough.