How can we wait for the exit code of a subprocess asynchronously?

[Atry]

Hi all, I am working on a new HSL API for creating subprocesses. One of the goals is to make the new API completely asynchronous. Unfortunately, currently we don't have a good mechanism to wait for the exit code of a subprocess asynchronously.

I think we could expose a Hack function calling LightProcess::SetLostChildHandler to solve the problem if LightProcess is enabled, and provide a fallback implementation to wait for the exit code synchronously when LightProcess is not enabled.

The proposed native function would like this:

function add_lost_child_handler(int $pid, (function():void{}) $handler):void;
function remove_lost_child_handler(int $pid):void;

What do you think?

[fredemmott]

This will need modifying LightProcess, but it would be much better to natively expose an Awaitable; we generally consider wrapping callbacks to awaitables promise-style to be an antipattern, and it's intentionally awkward to do.

[fredemmott]

We don't need extensions, or to get an FD/awaitable for the process itself; the flow (not the code) of the lightprocess worker currently looks something like this:

cmd_and_args = receive_from_hhvm();
pid = fork { exec ( foo) };
send_to_hhvm(pid);
while (next = recv_from_hhvm()) {
  switch(next) {
     // ... other cases ...
     case "waitpid": { // existing blocking thing
        int pid = recv_from_hhvm();
        auto result = waitpid(pid);
        auto saved_errno = errno;
        send_to_hhvm(result);
        if (result < 0) {
          send_to_hhvm(saved_errno);
        }
     }
  }
}

I'm suggesting adding something like:

case "waitpid_async":
  auto pid = recv_from_hhvm();
  auto [r, w] = pipe();
  // initial function call in C++ returns the FD or awaitable here
  send_fd_to_hhvm(r);
  // hhvm is not waiting for anything below until an `await` happens
  close(r);
  auto result = waitpid(pid);
  auto saved_errno = errno;
  // awaitable maybe resolves here...
  dprintf(w, "%d\n", result);
  // ... and here... 
  if (result < 0) {
    dprintf(w, "%d\n", saved_errno);
  }
  //... and here; or, only here if we wrap it natively rather than returning an OS\FileDescriptor to hack code
  close(w);

[Atry]

Thank you. I supposed that waitpid_async could be named pidfd_open if it returns an fd, and we can implement it with the help of a SIGCHLD handler instead of blocking a lightprocess worker process.

[fredemmott]

I'd rather not name it the same as a syscall (even if linux-specific), especially when it doesn't do quite the same thing: pidfd_open attaches to an existing FD, this can only work from the parent of the process that spawned it - which does mean we still need to do it in the lightprocess worker that originally spawned it, like we do for blocking waitpid

[Atry]

I didn't find that pidfd_open is limited to child processes. Any way we cannot depend on it right now because it is Linux specific and too new to be available in our supported distributions.

[fredemmott]

pidfd_open is not limited to child processes

The fd approach I suggested is, so shouldn't be called pidfd_open, and needs to be in the original worker thread

[Atry]

Shall we expose some functions of LightProcess to Hack, like:

newtype LightProcess = mixed;

// General LightProcess functions
<<__Native>>
function select_read_async(LightProcess $proc): Awaitable<void>;
<<__Native>>
function release_light_process(LightProcess $proc): void;
<<__Native>>
function acquire_light_process(): LightProcess;


// waitpid functions
newtype stat_loc = int;
<<__Native>>
function write_waitpid(LightProcess $proc, pid_t $pid, int $options = 0): void;
<<__Native>>
function read_waitpid_result(LightProcess $proc): stat_loc; 

The current implementation of LightProcess::waitpid can be split into write_waitpid and read_waitpid_result, where write_waitpid includes the lwp::write part and read_waitpid_result includes the lwp::read part. With the help of general LightProcess functions to select the afdt_fd, the waitpid_async can be implemented roughly as following:

async function waitpid_async(pid_t $pid, int $options = 0): stat_loc {
  $proc = acquire_light_process();
  try {
    write_waitpid($proc, $pid, $options);
    await select_read_async($proc);
    return read_waitpid_result($proc);
  } finally {
    release_light_process($proc);
  }
}