stack overflow

[jb-alvarado]

Hello,
ich use your crate in a tokio runtime and some how I get on some machines a stack overflow with this code:

let mut m3u8_files = Vec::new();

let mut entries = WalkDir::new(path).filter(move |entry| async move {
    if entry.path().is_file() && entry.path().extension().is_some_and(|ext| ext == "m3u8") {
        return Filtering::Continue;
    }

    Filtering::Ignore
});

loop {
    match entries.next().await {
        Some(Ok(entry)) => {
            m3u8_files.push(entry.path().to_string_lossy().to_string());
        }
        Some(Err(e)) => {
            error!(target: Target::file_mail(), "error: {e}");
            break;
        }
        None => break,
    }
}

With this not:

let mut entries = WalkDir::new(path);

while let Some(Ok(entry)) = entries.next().await {
    if entry.path().is_file() && entry.path().extension().is_some_and(|ext| ext == "m3u8") {
        m3u8_files.push(entry.path().to_string_lossy().to_string());
    }
}

Is my code on top wrong? I though I get it correct from doc.

[ririsoft]

Hello,
The code looks good to me.
Do you have some more information : what do you mean by stack overflow ? what is the error returned by the terminal ? Do you have a rust stack trace to share ?

[jb-alvarado]

Thanks @ririsoft for your quick response!

This issue is hardly to reproduce, because when I create a minimal example, the code works just fine. When I run my original code in my target VM I get the error, but on my desktop fedora system not.

I only know, when I change that filter part with the second code example, I don't get this stack overflow.

The full message is:

thread 'main' has overflowed its stack
fatal runtime error: stack overflow
[1]    375660 IOT instruction  RUST_BACKTRACE=1 cargo run -- --channel 1 --foreground --log-to-console

I don't get any stack trace, supposedly this kind or errors don't produce stack traces.

[jb-alvarado]

Ok, now I can reproduce this issue with this code:

use async_walkdir::{Filtering, WalkDir};
use tokio::runtime::Builder;
use tokio_stream::StreamExt;

fn main() -> std::io::Result<()> {
    let runtime = Builder::new_multi_thread()
        .worker_threads(4)
        .thread_name("main-tokio-runtime")
        .thread_stack_size(2 * 1024)
        .enable_time()
        .build()
        .unwrap();

    runtime.block_on(async {
        tokio::spawn(async move {
            let mut m3u8_files = Vec::new();

            let mut entries =
                WalkDir::new("/home/user/Videos/hls/live").filter(move |entry| async move {
                    if entry.path().is_file()
                        && entry.path().extension().is_some_and(|ext| ext == "m3u8")
                    {
                        return Filtering::Continue;
                    }

                    Filtering::Ignore
                });

            loop {
                match entries.next().await {
                    Some(Ok(entry)) => {
                        m3u8_files.push(entry.path().to_string_lossy().to_string());
                    }
                    Some(Err(e)) => {
                        eprintln!("error: {e}");
                        break;
                    }
                    None => break,
                }
            }

            println!("{m3u8_files:?}");
        });

        tokio::time::sleep(tokio::time::Duration::from_secs(10)).await;
    });

    Ok(())
}

Here is the same, when I replace the filter with:

let mut entries = WalkDir::new("/home/user/Videos/hls/live");

while let Some(Ok(entry)) = entries.next().await {
    if entry.path().is_file()
        && entry.path().extension().is_some_and(|ext| ext == "m3u8")
    {
        m3u8_files.push(entry.path().to_string_lossy().to_string());
    }
}

I even can set thread_stack_size to 1 and it still runs fine.

Some how it is related to tokio::spawn, because when I remove that the WalkDir filtering works also.

[ririsoft]

I do not reproduce the test locally in a directory without any m3u files.
I suspect the stack overflow is due to your particular directory tree.
What is the result of the following command: find "/home/user/Videos/hls/live" | wc -l ?

[jb-alvarado]

Is 460, six of them are *.m3u8 files and the rest has other extensions. The directory has no sub-directories.

[ririsoft]

The directory I tested has 6487 files, so this is definitely not the issue :)

On which platform are you running : OS/CPU ?

[ririsoft]

Ok forget about that I reproduced the bug with your test. Let me some time to investigate.

[svenstaro]

@ririsoft Hey, sorry for the ping but did you have some success debugging this? I've hit this recently.

EDIT: The code change as suggested by @jb-alvarado does work though. It should still be fixed I believe.

[ririsoft]

Hello @svenstaro,

I am really struggling fixing the issue: the way the code is using closures and recursion makes the issue occurring when there are many files in the repository.

I am really sorry for the situation. I am considering rewriting it all thanks to new changes introduced in rust recently on the async closure side.

Any help here would be much appreciated.

[svenstaro]

Perhaps it might be nice to give this library an API that's more like walkdirs so it's more of a drop-in? The iterator-like usage feels more natural than the filter-and-then-iterate approach of async-walkdir.