fix: spot interruption detection

[saikonen]

fix for sending SIGTERM to the main process when a spot interruption is detected.

os.ppid() is not reliable inside a Sidecar due to nesting of subprocesses, so we need to instead pass the main process id via some other means. This PR does it with an environment variable.

Also noteworthy is that because tasks running on Kubernetes have termination_grace_period_seconds=0, this makes graceful handling of SIGTERM impossible as the pod goes down immediately upon receiving it.

Flow for testing with AWS FIS:

import signal
from time import sleep
from metaflow import step, FlowSpec, retry, current, kubernetes, batch

class RetryFlow(FlowSpec):

    @kubernetes
    # @batch
    @retry(times=3)
    @step
    def start(self):
        print("Starting 👋")
        
        def _debug_signal(*args, **kwargs):
            print("SIGTERM RECEIVED")
        
        signal.signal(signal.SIGTERM, _debug_signal)
        
        print("sleeping for 300")
        sleep(300)
        if current.retry_count<1:
            raise Exception("You shall not pass (to the next step!)")
        
        self.next(self.end)

    @step
    def end(self):
        print("Done! 🏁")

if __name__ == "__main__":
    RetryFlow()

[nflx-mf-bot]

Netflix internal testing[1084] @ fd872fd

[nflx-mf-bot]

Testing for OSS PR [1084] @ commit fd872fd had 10 FAILUREs.