When awaiting handles in a loop vs join_all: thread 'tokio-runtime-worker' has overflowed its stack

[gitmalong]

Hi guys!

I am facing an issue I cannot really understand. I have a bunch of handles (only 11):

let mut handles: Vec<Timeout<impl Future<Output = Result<LoadAndStoreBalanceOutput, Report>>>> = vec![];
for x in y {
 handles.push(x.doSomeAsyncWork());
}
// Added just 11 handles

After creating the futures I would like to await the results. First I was awaiting all of them at once but I then decided to await them sequentially.

But when replacing

let output = futures::future::join_all(handles).await;

with

let mut output = vec![]; 
for handle in handles {
  output.push(handle.await);
}

my app crashes with

thread 'tokio-runtime-worker' has overflowed its stack
fatal runtime error: stack overflow

This is something I cannot understand. tokio-console didn't help me in identifying the root cause. Do you have any ideas?

Thanks

[Darksonn]

The cause of the issue is not visible in the code you have shared with us, unfortunately.

That said, I will add this: The difference between the two snippets is that using join_all runs all of the handles concurrently, whereas the loop runs them one-after-the-other. If you wanted the loop to run them concurrently, you would need to tokio::spawn them.

[gitmalong]

Hi @Darksonn. tokio-console has also not shown a high amount of async tasks or something I can make sense of regarding this.

I ended up using let stream = futures::stream::iter(handles).buffer_unordered(4); which has again a different behavior but does also not trigger the issue.

I will try to create a minimal reproducible example and play around a bit. Hope I find some useful insights to report back.

[Darksonn]

Note that join_all or buffer_unordered uses within-a-single-task concurrency, so they all show up as a single task in tokio-console.

[gitmalong]

Alright. I still can confirm that I only had 11 handles during my tests.

[Noah-Kennedy]

Something is going on here with how your futures within the loop are getting desugared and resulting in the future size blowing up. I don't know enough about how this happens in the compiler to say what's going on though.

Something to start with: can you check what the size of the resulting top-level future for this function is, both with the join_all approach and the loop approach? I'm curious how big the blowup is. It's entirely possible that this isn't the actual problem section of your code, and is merely "the straw that broke the camels back" so to speak, and I'd like to rule this possibility out.

[gitmalong]

Now I am facing the same error message when inserting into a DashMap: xacrimon/dashmap#244

@Noah-Kennedy do you want me to determine the size (not the length) of each handle also of the join_all's future (written above as output)? How can I do that?