Reclaim disk space

[tbmale]

Hi,

I am inserting repeatedly 1_000_000 items with the same 1 million keys and a single byte array value and my db is keep growing,
I have searched for a method to reclaim disk space, but I haven't found one.
Is there a catch or is just me being ... me ? :)

[marvin-j97]

Ah! This is already on my todo list, this only happens when you never use a snapshot read or any transactions, because the GC watermark is never increased, so old versions are never below that watermark.
If you sometimes create and instantly close a snapshot (let _ = partition.snapshot();), you may see space being reclaimed as you continue writing more data. Though there is some weirdness going on with it, the SnapshotTracker needs to be improved a bit.

Tracking: #86

[marvin-j97]

Should be resolved by #87

[tbmale]

Thanks @marvin-j97 !

I build 2 test cases as stated in my question (with/without write transaction) that I ran 3 times for main and fix branches, in order to test the size growth of my db.

My results are as follows:

branch	transaction	run no	db size	time
main	yes	1	26,6 MB	20.04s
main	yes	2	36,3 MB	22.43s
main	yes	3	46,1 MB	26.03s
main	no	1	31,7 MB	12.75s
main	no	2	41,6 MB	17.26s
main	no	3	51,2 MB	22.20s
fix	yes	1	26,6 MB	20.57s
fix	yes	2	36,3 MB	22.75s
fix	yes	3	46,1 MB	26.28s
fix	no	1	31,7 MB	13.70s
fix	no	2	35,3 MB	18.72s
fix	no	3	36,5 MB	21.06s

I included the times because I got them anyway, but they include also a retrieval with prefix db.read_tx().prefix(&pt,"key".as_bytes()).count() .

Please note that I have deleted the db between each set of tests (therefore the run no).

There seams to be an improvement in the insert without transaction, although my test is rather crude.
Please tell me if I did something wrong.

[marvin-j97]

@tbmale If I understand correctly, I think the problem is that your process is very short-lived for each run, meaning that the database never gets a chance to run a compaction, which would get rid of old versions.

If you try this code, you should see that the space is in fact reclaimed:

fn main() -> fjall::Result<()> {
    println!("Hello, world!");

    let keyspace = fjall::Config::default().open()?;
    let p = keyspace.open_partition("default", Default::default())?;

    loop {
        for x in 0u32..1_000_000 {
            p.insert(x.to_be_bytes(), [5])?;
        }

        let len = p.len()?;

        eprintln!("{:?} MiB - {len} items", p.disk_space() / 1_024 / 1_024);
    }
}

while running it using the main will not reclaim space, unless opening and closing a read snapshot sometimes.

[tbmale]

Ok, I think get it.

I thought that persist will force compaction somehow :) .