File descriptor exhaustion when calling FilesystemWriter::write

[Lynnesbian]

When creating a FilesystemWriter, calling push_file with many (over a thousand) files, then calling write, the process will crash due to running out of file descriptors.

I've created a small reproducer below - it depends on backhand and anyhow, and you'll need to replace the path with something that points to a folder with many files on your system.

use std::fs::File;
use std::io::{BufReader, BufWriter};
use backhand::{FilesystemWriter, NodeHeader};

fn main() -> anyhow::Result<()> {
    let mut squash_fs = FilesystemWriter::default();
    let path = "/lots/of/files";
    let header = NodeHeader {
      permissions: 0o755,
      ..NodeHeader::default()
    };
    
    for entry in std::fs::read_dir(&path)? {
      let entry = entry?;
      let file = File::open(entry.path())?;
      let reader = BufReader::new(file);
      squash_fs.push_file(reader, entry.path().file_name().unwrap(), header)?
    }
    
    squash_fs.write(BufWriter::new(File::create("out.squashfs")?))?;

    Ok(())
}

Is there a way around this, beside writing and re-opening the SquashFS filesystem after every few hundred calls to push_file?

[wcampbell0x2a]

I think what I would do to solve this would be to pre-read files that aren't that large, as otherwise the File will be open until after write. Something like this:

        if len < 1024 {
            let mut buffer = Vec::new();
            file.read_to_end(&mut buffer)?;
            let reader = BufReader::new(Cursor::new(buffer));
            squash_fs.push_file(reader, entry.path().file_name().unwrap(), header)?
        } else {
            let reader = BufReader::new(file);
            squash_fs.push_file(reader, entry.path().file_name().unwrap(), header)?
        }

[aleasims]

I faced exactly the same issue: when I'm trying to create SquashFS image for a pretty big filesystem, I run into "Too many open files (os error 24)".

I solved it probably in the most ugliest way ever: I write image into a file after every single file handling and re-write it again and again. This way I always have only one source file opened at a time. However this solution introduces enormous performance issue (even with compression disabled) and becomes useless.

Your solution with pre-reading small files results in a huge memory increase, which is also not pretty.

I'm wondering, is it possible to extend the library with another input option: something like SquashfsFileWriter::UserDefinedPath(Path).

For now I'm thinking about another solution: implementing a specific reader which will open the file only on demand (when read() is called) and close it right after. Since the file is now opened and closed only once anyway, the number of syscalls should stay the same. What do you think about it? Can it work out? Any help appreciated.

[aleasims]

Looks like I was able to survive with lazy-open approach. It looks something like this:

struct LazyOpen {
    path: PathBuf,
    pos: SeekFrom,
}

impl io::Read for LazyOpen {
    fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
        let mut file = fs::File::open(&self.path)?;
        file.seek(self.pos)?;
        let count = file.read(buf)?;
        let current_pos = file.stream_position()?;
        self.pos = SeekFrom::Start(current_pos);
        Ok(count)
    }
}

fn write_squashfs() {
    // ...
    let reader = LazyOpen {
        path: local_path,
        pos: SeekFrom::Start(0),
    };
    fs_writer.push_file(reader, path_in_squashfs, header).unwrap();
}

I don't see any significant performance degradation with this.

[wcampbell0x2a]

oh cool!