I/O scheduler is task-heavy and inefficient for random access

[westonpace]

The current I/O scheduler is very task heavy. For example:

• The scheduler thread submits a request by pushing the task into a channel
• The I/O loop pulls this task from the channel and spawns a task to run the I/O
• The I/O task calls into the object store which is often spawning yet another task

There can be 10's of microseconds between notifying a thread and that thread starting up. So in the above example, even if the object store isn't using tasks, we might introduce 20-30 microseconds of latency for each value.

To put this into context, when I was testing a random-access heavy workload, 60% of the latency was due to I/O scheduling inefficiency. This is probably the single biggest performance bottleneck for both 2.0 and 2.1 random access workloads.

Fortunately, fixing it should be a non-breaking change

[Xuanwo]

Hi @westonpace, I'm doing some exploration on this issue. Could you share how you got the 60% number here?

[westonpace]

Tracing. Everything in wait_for_page is us waiting on I/O but the actual work that needs to be done is super lightweight (see the narrow pink lines) since we are using an in-memory object store in the benchmark and a read is just an Arc clone (Bytes) at the end of the day:

[westonpace]

(note: I added a lot of #[instrument] lines to get the above picture)

[Xuanwo]

(note: I added a lot of #[instrument] lines to get the above picture)

Perhaps you can commit that in a branch so I can continue working on this 😆

[westonpace]

Perhaps you can commit that in a branch so I can continue working on this 😆

Sure, let me clean it up a bit and I'll push it today