Should we map ENOBUFS on macOS to `WouldBlock`?

[thomaseizinger]

We are seeing the following errors from the UDP socket on some installations on Mac OS:

Tunnel error: UDP socket failed: Failed to send datagram-batch with segment_size 1312 and total length 7872 to X.X.X.X:57689: No buffer space available (os error 55)

This appears to be the ENOBUFS error mentioned in the man page:

[ENOBUFS]         The system is unable to allocate an internal buffer.
                  The operation may succeed when buffers become avail-able. available.
                  able.

libc for Apple systems maps the error code 55 to this error: https://github.com/rust-lang/libc/blob/8e6f36c6ba5c91c150f2d79a06ffdc3d0b83a0f6/src/unix/bsd/apple/mod.rs#L2998

According to the Apple developer forum, this is a form of back-pressure and we should retry sending the packet when there are buffers available again.

I figured, we could do this in quinn-udp directly and map this error code on macOS to WouldBlock? Thoughts?

[Ralith]

Will a socket that previously returned ENOBUFS be marked as writable for kqueue purposes once buffer space becomes available?

[thomaseizinger]

I've found this NetBSD discussion that suggests that this is an acceptable pattern: https://mail-index.netbsd.org/netbsd-users/2012/09/20/msg011462.html

UDP packets are sent using sendto() directly
in loops similar to:

tries = 0;
do {
  if tries > 0
     sleep
  rc = sendto(sock, ...);
  tries++;
} while((rc is EAGAIN || EWOULDBLOCK || ENOBUFS) && tries < 8)

[thomaseizinger]

Actually, just noticed the sleep in there. That is not what we want!

[thomaseizinger]

libuv appears to be treating ENOBUFS identical to EWOULDBLOCK: https://github.com/libuv/libuv/blob/aac866f39923670d9c136851f866531f5f6cd889/src/unix/stream.c#L820

[thomaseizinger]

The manpage for kqueue says:

Takes a descriptor as the identifier, and returns whenever it is possible to write to the descriptor. For sockets, pipes and fifos, data will contain the amount of space remaining in the write buffer.

It is a bit of a long-shot but if it wants to return the size of the write-buffer, presumably it is to monitor for more size becoming available?

[thomaseizinger]

I am implementing a feature-flag in Firezone to test out this behaviour: firezone/firezone#9798

Once this is shipped, I can report back to how well this is working.