NIOFileSystem performance crawls to a halt with many small ops

[Joannis]

I've been working on an application for a client where we perform a large volume (50-100k/sec) of small reads & writes using NIOFileSystem. As the system gets under stress, around 3000 IOPS, things start stalling and the NIOThreadPool starts building up a massive backlog. It would be great to have a better solution here where we can use NIOFileSystem, but without worrying about massive backlogs in the NIOThreadPool.

[Lukasa]

This sounds like what you want is some form of backpressure management, where we can slow you down once we get a large enough queue depth.

[Lukasa]

Note, though, that slowing you down isn't going to be substantially cheaper than enqueueing: do you want to be given errors instead?

[glbrntt]

Is each operation on a different file? If so I imagine you're suffering a lot of overhead from each operation being submitted to the thread pool e.g., open, read, and close are all separate operations. This is necessary to allow users to do async things with a file handle.

I can imagine a lower-level API where you're given a blocking file handle whereby you'd effectively await getting a turn in the thread pool, but once there you could then where you can then do blocking work.

[hassila]

If this is on Linux and you are targeting sustained 100k IOPS rates I'd consider rolling something with the new io-uring support in the swift system package as an alternative that may be better suited for this purpose. If not, carry on 🙂

[Joannis]

@Lukasa I'd expect either errors or backpressure when the system is creating a backlog. But in addition to that, I would expect a mechanism through which I could expect a bit better performance too. It's not ideal, but right now we do our read and write calls through swift-system instead, performing an infinite retry on retryable failures. It's definitely suboptimal, I understand that, but through this change alone we've gone from 3-5k IOPS to about 100k+.

@glbrntt we've got a directory of files, each file contains many pages, each page is a ringbuffer of tiny (128-byte) entities. Every second one process writes data (received over UDP/DTLS) into the ringbuffer. The ringbuffer is then emptied out and handled by a different process round-robin style. If a ringbuffer has a backlog, all entities that weren't processed yet are handled in a batch. And all pages in the ringbuffer can and should be grouped as I/O too.

Right now, NIOFileSystem doesn't allow us to group those I/O calls for as single file. That alone would likely enable much more throughput. In addition, once a backlog occurs the whole system grinds to a halt, with very many threads spending the majority of their time spent in the NIOThreadPool.

@hassila this would definitely help us with our current approach! While it's definitely a better fit, it would be nice if NIOFileSystem supported this use case too in some form. Even if by handling a batch of I/O requests.

[Lukasa]

If this is on Linux and you are targeting sustained 100k IOPS rates I'd consider rolling something with the new io-uring support in the swift system package as an alternative that may be better suited for this purpose.

The intention of the design of NIOFileSystem is that the API should support dropping that iouring API in underneath.

@Joannis This sounds a lot like the thread pool isn't big enough for your needs. Have you tried making it wider?

[Joannis]

I did try making it wider. I can work around it infinitely stalling that way, but my throughput is still much slower than if I just do blocking reads and writes through swift system itself. Aside from applying back pressure, it would be nice if there was API that enables more efficient access to these files. It's not a must have, but not having it would mean I have to build something myself on swift-system.