On the implications of not having keep-alive in the default way of starting a server

[IvanUkhov]

Summary

In an earlier discussion, it was noted that Axum does not enabled keep-alive as part of axum::serve, which is arguably the main method people use to start a server. To understand the situation better, is it then the case that the server is losing one file descriptor each time a connection is not closed properly? So over time, the server is expected to run out of file descriptors. And it applies to all servers using Axum (without any auxiliary tweaks, such as the one above). Is this correct?

axum version

0.8

[mladedav]

I wouldn't say it's expected to run out, but yes, someone could try to starve you out.

This is not an axum thing though as you're providing the impl Listener or specifically I assume the TcpListener. You can tweak it any way you see fit before passing it to axum::serve.

We also generally recommend to put a proxy like nginx in front of your axum server for hardening.

[mladedav]

Because what you're describing is not normal operations. Or it is if you expect you're going to be a target of an attacker, in which case I would again recommend to use a reverse proxy in front of your server.

In this case I don't think TCP keep alives would even really help that much as anyone can just open 1024 connections to your server and not write anything (just sending keep alives) and you would run out of fd's again.

As for out-of-the-box axum behavior, again, there is nothing it can do. You provide the TcpListener which if you use tokio defaults does not have keep alives but you can create it with TcpSocket instead, setting the options before binding it. The only thing axum can reasonably do is to provide guidance on how to set up the listener differently (and maybe provide a newtype around TcpListener with different defaults).

[IvanUkhov]

Well, I was more thinking about the asymptotic behavior. As time goes to infinity, there will be more and more dead but not closed connections for one reason or another. Software is not perfect; the client might crash. There might be some firewall and NAT dropping idle connections. And yes, there could be attackers, too. But I was more thinking about normal operations, as you put it.

From the previous discussion, I inferred that it was not possible to continue using axum::serve but with keep-alive enabled; it was suggested to go down to the hyper level. However, your response seems to suggest that I can just customize the listener, which would be sufficient. It just, as you say, has to be mentioned somewhere where people usually read, like in axum::serve perhaps.

[mladedav]

Right, I guess that could theoretically happen, but I would expect it to be pretty rare. If the server tries to send anything after the client dies, the connection will be cleaned up. But yeah, fair. Though if that's what you're worried about, I'd also suggest to just enable TCP keep alive globally in kernel.

You can use all settings on the listener. You currently cannot use custom settings on hyper.

The original discussion seems to have asked about http keep alives and for some reason I thought you were talking about tcp keep alives. The former is set on hyper so you'd couldn't use axum::serve while the latter is set on TcpSocket so you can just build a listener out of that and pass it to axum::serve.

[IvanUkhov]

Uh, I was entirely oblivious to this aspect that keep-alive can be done on different levels, including transport and application. So the one in hyper is HTTP, and the one here is TCP. Interesting. Thank you for pointing this out!

[IvanUkhov]

Regarding your point about enabling in the kernel, after reading around, the default value seems to be two hours, and it is difficult to imagine Google disabling it in their container-optimized OS, which is what is used. So this breaks my argument that the number of dead connections will only be increasing over time. It is then effectively pruned.