Problems when `file://` storage runs out of space

[vi]

Steps to reproduce:

1. Mount tmpfs with a size limit
2. Create ZboxFS file:// storage on that size
3. Create a zbox::File there and start filling it with data (without finishing) ad infinum.

Expected:

Eventually std::io::Write::write returns IO error suggesting that allowed storage space is exhaused. Ideally the file may be finish()ed at that point, with the data previously accepted by std::io::Write::write without error being guaranteed to fit on remaining space.

Actual:

[2020-01-01T00:45:51Z DEBUG fuse_mt::fusemt] getxattr: "/file" "security.capability"
[2020-01-01T00:45:51Z DEBUG fuse_mt::fusemt] getxattr: error 61
[2020-01-01T00:45:51Z DEBUG fuse::request] WRITE(153376) ino 0x0000000000000002, fh 0, offset 314085376, size 4096, flags 0x0
[2020-01-01T00:45:51Z DEBUG fuse_mt::fusemt] write: "/file" 0x1000 @ 0x12b89000
[2020-01-01T00:45:51Z DEBUG fuse::request] GETXATTR(153377) ino 0x0000000000000002, name "security.capability", size 0
[2020-01-01T00:45:51Z DEBUG fuse_mt::fusemt] getxattr: "/file" "security.capability"
[2020-01-01T00:45:51Z DEBUG fuse_mt::fusemt] getxattr: error 61
[2020-01-01T00:45:51Z DEBUG fuse::request] WRITE(153378) ino 0x0000000000000002, fh 0, offset 314089472, size 4096, flags 0x0
[2020-01-01T00:45:51Z DEBUG fuse_mt::fusemt] write: "/file" 0x1000 @ 0x12b8a000
[2020-01-01T00:45:51Z DEBUG zbox::trans::txmgr] run tx failed: Io(Custom { kind: Other, error: "other os error" })
[2020-01-01T00:45:51Z DEBUG zbox::trans::txmgr] abort tx#3
[2020-01-01T00:45:51Z WARN  zbox::trans::txmgr] abort tx#3 failed: No space left on device (os error 28)
[2020-01-01T00:45:51Z ERROR zboxmount::zboxfuse] IO error: other os error
[2020-01-01T00:45:51Z WARN  fuse::request] Ignoring unknown FUSE operation 40
[2020-01-01T00:45:51Z DEBUG fuse::request] GETXATTR(153380) ino 0x0000000000000002, name "security.capability", size 0
[2020-01-01T00:45:51Z DEBUG fuse_mt::fusemt] getxattr: "/file" "security.capability"
[2020-01-01T00:45:51Z DEBUG fuse_mt::fusemt] getxattr: error 61
[2020-01-01T00:45:51Z DEBUG fuse::request] WRITE(153381) ino 0x0000000000000002, fh 0, offset 314089472, size 4096, flags 0x0
[2020-01-01T00:45:51Z DEBUG fuse_mt::fusemt] write: "/file" 0x1000 @ 0x12b8a000
[2020-01-01T00:45:51Z DEBUG zbox::trans::txmgr] run tx failed: Io(Custom { kind: Other, error: "other os error" })
[2020-01-01T00:45:51Z DEBUG zbox::trans::txmgr] abort tx#3
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /rustc/3eeb8d4f2fbae0bb1c587d00b5abeaf938da47f4/src/libcore/macros/mod.rs:15:40
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:84
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:61
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1025
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1426
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:65
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:50
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:193
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:210
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:471
  11: rust_begin_unwind
             at src/libstd/panicking.rs:375
  12: core::panicking::panic_fmt
             at src/libcore/panicking.rs:84
  13: core::panicking::panic
             at src/libcore/panicking.rs:51
  14: core::option::Option<T>::unwrap
             at /rustc/3eeb8d4f2fbae0bb1c587d00b5abeaf938da47f4/src/libcore/macros/mod.rs:15
  15: zbox::trans::txmgr::TxMgr::abort_trans
             at /home/vi/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.8/src/trans/txmgr.rs:157
  16: zbox::trans::txmgr::TxHandle::abort
             at /home/vi/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.8/src/trans/txmgr.rs:238
  17: zbox::trans::txmgr::TxHandle::run
             at /home/vi/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.8/src/trans/txmgr.rs:208
  18: <zbox::file::File as std::io::Write>::write
             at /home/vi/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.8/src/file.rs:586
...

Additionally, storage cannot be opened again after that:

[2020-01-01T00:52:25Z INFO  zbox::base] ZboxFS v0.8.8 - Zero-details, privacy-focused in-app file system
[2020-01-01T00:52:25Z INFO  zbox::fs::fs] open repo: file:///tmp/limit/q, read_only: false
[2020-01-01T00:52:25Z DEBUG zbox::volume::volume] volume opened: Storage(FileStorage { base: "/tmp/limit/q" })
[2020-01-01T00:52:25Z DEBUG zbox::trans::wal] cold redo abort tx#3
[2020-01-01T00:52:25Z DEBUG zbox::trans::wal] cold abort completed
[2020-01-01T00:52:25Z DEBUG zbox::trans::wal] WalQueue opened, seq: 6, watermarks: (txid: 3, block: 6)
Error: Io(Custom { kind: Other, error: "Decrypt error" })

[vi]

Similar panic happens in sqlite:// mode (with a "database is full" error instead). In that case however storage can be opened again (if I grow tmpfs a little bit). Unfinished information in the file is lost, obviously (how do I recover it by the way?).

[burmecia]

Thank you for issue, that's a good edge case.

The unfinished data won't be able to recover as writing is processed in a transaction. If writing doesn't call finish(), the transaction doesn't commit and all uncommitted data will be lost.

[vi]

If writing doesn't call finish(), the transaction doesn't commit and all uncommitted data will be lost.

After a crash with unfinished writes, storage is still full in megabytes. Are those files securely inaccessible (e.g. no decryption key available) or just unrefereced?

[vi]

With mem:// storage, running out of space is even nastier: memory allocation of 8192 bytes failed, then just plain abort.

---

I think ZboxFS should actively mind available room (in file://, in memory, in database) and reserve some space, so writes start failing while transaction can still be committed.

[burmecia]

If writing doesn't call finish(), the transaction doesn't commit and all uncommitted data will be lost.

After a crash with unfinished writes, storage is still full in megabytes. Are those files securely inaccessible (e.g. no decryption key available) or just unrefereced?

Yes, those files are still secured but they are inaccessible.

[burmecia]

With mem:// storage, running out of space is even nastier: memory allocation of 8192 bytes failed, then just plain abort.

I think ZboxFS should actively mind available room (in file://, in memory, in database) and reserve some space, so writes start failing while transaction can still be committed.

That could be hard as some storage are not able to retrieve free space at run time, even through it can be retrieved the cost to constantly monitor it is still high.

[vi]

That could be hard as some storage are not able to retrieve free space at run time, even through it can be retrieved the cost to constantly monitor it is still high.

ZboxFS can just pre-allocate some space needed to complete a transaction.

[vi]

Yes, those files are still secured but they are inaccessible.

Secured with the same key as the rest of files or secured with some session key that is supposed to be written to storage only on transaction commit?

If the former, there should be some recovery function that trades consistency for availability, allowing to read possibly broken data from outside normal transaction finish. Like git fsck + git cat-file for Git.

[burmecia]

That could be hard as some storage are not able to retrieve free space at run time, even through it can be retrieved the cost to constantly monitor it is still high.

ZboxFS can just pre-allocate some space needed to complete a transaction.

Accurately pre-allocate space is impossible because it has no idea how much data user will write in. The file writing is like streaming, until transaction was committed there is no way to know the actual size in advance.

[vi]

Accurately pre-allocate space is impossible because it has no idea how much data user will write in. The file writing is like streaming, until transaction was committed there is no way to know the actual size in advance.

I see megabytes used by file:// storage going up while writing a file even before committing a transaction. Does it mean only some constant additional space is needed to commit transaction when std::io::Write::write calls are already done?

Anyway, even a heuristical "pre-allocate X kilobytes for transaction commits" where X is configurable by user can be beneficial, to reduce probability of finish() also failing after write has failed.

---

In short: running out of space should not lead to big data loss, only maybe a limited one.

[burmecia]

Yes, those files are still secured but they are inaccessible.

Secured with the same key as the rest of files or secured with some session key that is supposed to be written to storage only on transaction commit?

If the former, there should be some recovery function that trades consistency for availability, allowing to read possibly broken data from outside normal transaction finish. Like git fsck + git cat-file for Git.

ZboxFS uses the former. When writing data to file, it firstly writes data to a staging area. When transaction is committing, those staging data will be transferred to content management system. That means any uncommitted data will not be in content management system, thus it cannot be accessed and it will discard when transaction aborted.

[vi]

When writing data to file, it firstly writes data to a staging area.

How do I recover data from that staging area if transaction was neither cleanly finished nor cleanly aborted?

Does that staging area contain all necessary information needed to complete a transaction or some things are only kept in RAM?

[burmecia]

Accurately pre-allocate space is impossible because it has no idea how much data user will write in. The file writing is like streaming, until transaction was committed there is no way to know the actual size in advance.

I see megabytes used by file:// storage going up while writing a file even before committing a transaction. Does it mean only some constant additional space is needed to commit transaction when std::io::Write::write calls are already done?

Anyway, even a heuristical "pre-allocate X kilobytes for transaction commits" where X is configurable by user can be beneficial, to reduce probability of finish() also failing after write has failed.

In short: running out of space should not lead to big data loss, only maybe a limited one.

You see the storage going up because of the staging data, the file content has been written to storage and committing tx is just to 'mark' it as permanent. Committing tx needs to write a WAL file, its size is not large but still depends on how large the file you are writing. Larger file produces more content nodes to be added to tx, thus the wal file size will increase. Heuristically estimate the wal file size in advance is impossible, but I agree pre-allocation some blocks for wal is a good idea, and it might be able to handle most cases.

[burmecia]

When writing data to file, it firstly writes data to a staging area.

How do I recover data from that staging area if transaction was neither cleanly finished nor cleanly aborted?

Does that staging area contain all necessary information needed to complete a transaction or some things are only kept in RAM?

The staging area only contains file content and it is referenced by a temporary content node which is only in memory. That content node is merged with pre-existing file content node and written to storage when committing. Thus, if tx is not completed, the temporary content is lost and staging data cannot be accessed.

[burmecia]

Would you mind share your code and setup, so I can reproduce this issue locally and that will be helpful? Thanks.