Why can't RefCells be used inside (non-local) tasks?

[oliver-giersch]

This is something I've been pondering over for some time and have not been able to find an answer to.

Assume we have a generic async I/O server working roughly like this:

1. initialize some (local) Shared state (e.g. a map for pending requests)
2. split connection into read and write halves
3. handle reads and writes concurrently, but both require mutable access to the state from 1.

So basically I want to avoid synchronization (so no spawning two sub tasks and wrapping Arc<Mutex<Shared>> for each half) and I also don't want writes blocking reads, which means no single select loop with both reads and writes on an unsplit connection.

I tried spawning two local tasks and wrapping the shared state in a Rc<RefCell<Shared>>, but LocalSets can not be easily used on regular spawned tasks.
I also tried using just a RefCell<SharedState> and using two separate read and write loops, each implemented in their own future, which are just joined by the "main" (server) task, but, you RefCell can't even be used exclusively inside a task it seems.

Is there some other option I for this problem? And also, is there actually a safety reason why RefCell can not be used inside a task (future), or is that only a limitation of the type system, which may, e.g. not be able to differentiate between !Sync types passed into the future and such types that are both created & dropped entirely within a future?

[Darksonn]

You are correct that having two halves of a tokio::join! access the same RefCell would be fine, because even if the task gets moved across threads, both shared references to the RefCell are moved together. You never end up with two different threads both having a reference to the same RefCell at the same time.

Unfortunately, there's no safe way to do this because the Rust language has no concept of "safe to send as long as you send both halves together". There's only "any kind of sending is safe" and "no kind of sending is safe".

[oliver-giersch]

Isn't is really about what can "escape" a future, rather than what could be sent where? I have to add that I don't have a really good grasp on when exactly a future is or is not Send, but the way I understand it, futures are only Send if they are also Sync, which means RefCell (which is Send but !Sync) can not be used in Send futures, regardless of whether it can escape. It would be nice if the compiler would do some kind of escape analysis in the future, like it already takes into consideration what is held across await points right now.

Anyways, I get that this is not a tokio limitation but one of the language itself, but is there any safe workaround for this issue? I guess I could implement Sync for the Shared state, but that is rather ugly, of course.

[Darksonn]

Isn't is really about what can "escape" a future, rather than what could be sent where?

In some sense yes. I imagine it would be possible to define a trait similar to Send and Sync that means "moving a future containing me is ok, but don't let me leave the future". However, such a trait does not exist.

The way I understand it, futures are only Send if they are also Sync, which means RefCell (which is Send but !Sync) can not be used in Send futures, regardless of whether it can escape.

Not quite. Storing a RefCell inside a future does not make the future non-Send on its own. What makes your future non-Send is the fact that you are also storing a &RefCell somewhere, and immutable references to RefCell are neither Send nor Sync.

is there any safe workaround for this issue?

You can use a mutex instead of a RefCell, or you can restructure your code to not need the RefCell.

[oliver-giersch]

In some sense yes. I imagine it would be possible to define a trait similar to Send and Sync that means "moving a future containing me is ok, but don't let me leave the future". However, such a trait does not exist.

These requirements for futures are directly implemented in the compiler, no? At least I don't see any of them encoded in regular trait bound terms in the Future docs, so perhaps some kind of escape analysis could likewise be done. Do you know of any work being done in that direction?

Not quite. Storing a RefCell inside a future does not make the future non-Send on its own. What makes your future non-Send is the fact that you are also storing a &RefCell somewhere, and immutable references to RefCell are neither Send nor Sync.

Ah yes, now I remember that there are some funky restrictions placed specifically on shared references.

You can use a mutex instead of a RefCell, or you can restructure your code to not need the RefCell.

Can you give some hint how I would restructure the code to get there? Currently, it roughly looks like this:

async fn run_server(mut conn: Connection, rx: Receiver<Request>) {
    let shared = Shared {
        pending: RefCell::new(HashMap::new()),
    };

    // run both loops concurrently & abort the other once when one finishes
    let (reader, writer) = stream.split();
    let res = tokio::select! {
        res = shared.handle_reads(reader) => res,
        res = shared.handle_writes(writer, rx) => res,
    };

    if let Err(e) = res {
        tracing::error!(err = ?e, "exiting server due to error");
    }
}

I guess I could go the route of a single select loop within a single task that handles both reads and writes (without split), but does not "block" (await) on the write call in the respective branch, but I have no idea how, so far. Maybe by using the readiness API somehow..

[Darksonn]

These requirements for futures are directly implemented in the compiler, no?

All the compiler does is check if any values you are storing in the future are non-Send, and if so, the future becomes non-Send. The type &RefCell is neither Send nor Sync, so as far as the compiler knows, there are no circumstances whatsoever where sending a &RefCell to another thread is safe. You really do need another trait so the compiler can understand the nuances of when it can be moved across threads.

Ah yes, now I remember that there are some funky restrictions placed specifically on shared references.

The meaning of Sync is literally "immutable references to this type are Send", so I don't know if I would consider it that funky.

Can you give some hint how I would restructure the code to get there?

Basically you would need to write a select loop that looks like this:

loop {
    tokio::select! {
        msg = read_next_msg() => { /* ... */ },
        _ = write_next_msg() => { /* ... */ },
    }
}

then only access Shared in the select arms. It's often a massive pain to do it because you need both methods to be cancel safe, but in some cases the tokio-util::codec utility can be utilized for this, as all of the encoder and decoder methods are cancel safe.

[oliver-giersch]

Well, at least to me it's funky, but perhaps only in the sense that I still have trouble wrapping my head around this stuff, regardless of often I've read up on it in the past ;)

Anyways, thanks for the hint and thanks for the extremely quick and helpful responses in general! I think I'll try storing something like an Option<Request> outside of the loop selectively disable the write and channel-receive branches based on its current content. I think that might work.