Grok wrecked my relatively simple FASTER DB project - I can't solve it - please help

[user29A]

Hi - I was having Grok help me further develop this database application, and I don't know what happened but Grok changed something, and as far as I can tell the code looks fine, but I'm no longer getting a persistent database and cannot restore, and the log files are tiny. I'm a bit of a noob so I apologize, but at this point I would like some help with the code. Anyone willing to give me an hour of their time, I'll pay a good rate for 1 hr (100+/hr USD), we can do a screen-share on Zoom/Teams or something? It shouldn't take long.

I can show you some basic code here if you like. Environment is VS2022, C# 8.0, developing for Windows 11. This is the basic structure, seems like it's fine to me? Before exiting, it definitely has all good values held in memory...but it just never goes to disk it seems. If you can answer just from looking at this code, awesome, if not, I'm sure an hour with someone who knows better what they're doing would get me on my way. Thank you.

//defined in the class
 private readonly FasterKV<byte[], byte[]> _triangleStore;

In the constructor:

 // Initialize FASTER stores
 var logSettings = new LogSettings
 {
     LogDevice = new ManagedLocalStorageDevice(Path.Combine(databasePath, "fastdb_log")),
     ObjectLogDevice = new NullDevice(),
     PageSizeBits = 12, // 4KB pages
     MemorySizeBits = 20, // 1MB memory
     SegmentSizeBits = 24 // 16MB segments
 };

 var checkpointSettings = new CheckpointSettings
 {
     CheckpointDir = Path.Combine(databasePath, "Checkpoint"),
 };

 ReportProgress("Initializing _triangleStore...");
 _triangleStore = new FasterKV<byte[], byte[]>(1L << 20, new LogSettings
 {
     LogDevice = new ManagedLocalStorageDevice(Path.Combine(databasePath, "triangles_log")),
     ObjectLogDevice = new NullDevice(),
     PageSizeBits = logSettings.PageSizeBits,
     MemorySizeBits = logSettings.MemorySizeBits,
     SegmentSizeBits = logSettings.SegmentSizeBits
 }, checkpointSettings);
 string triangleLogPath = Path.Combine(databasePath, "triangles_log.0");
 if (File.Exists(triangleLogPath))
 {
     ReportProgress("Recovering _triangleStore...");
     _triangleStore.Recover();
     ReportProgress($"Recovered {_triangleStore.EntryCount} records from _triangleStore");
 }
 ReportProgress($"_triangleStore initialized. Initial EntryCount: {_triangleStore.EntryCount}");

In the builder:

using var triangleSession = _triangleStore.For(new SimpleFunctions<byte[], byte[]>()).NewSession<SimpleFunctions<byte[], byte[]>>("triangle_session");
///some work
triangleSession.Upsert(ref serializedKey, ref newValue);
triangleSession.CompletePending(true);
await _triangleStore.TakeFullCheckpointAsync(CheckpointType.FoldOver);
 _triangleStore.Log.Flush(true);

The database *.0 files used to be significant (MB's), now they are tiny (a few kB's), and when I try to recover, this code reports nothing is in the database, which is consistent with the .0 files being tiny:

public Dictionary<(ushort, ushort), HashSet<(ulong, ulong, ulong)>> LoadTrianglesIntoMemory(bool smallDatabase)
{
    if (!smallDatabase)
    {
        throw new InvalidOperationException("This method is only for small test databases.");
    }

    var triangles = new Dictionary<(ushort, ushort), HashSet<(ulong, ulong, ulong)>>();
    using var session = _triangleStore.For(new SimpleFunctions<byte[], byte[]>()).NewSession<SimpleFunctions<byte[], byte[]>>();
    using var iterator = session.Iterate();
    while (iterator.GetNext(out RecordInfo recordInfo))
    {
        byte[] key = iterator.GetKey();
        if (key.Length != 4) continue; // Skip invalid keys
        ushort angle1 = BitConverter.ToUInt16(key, 0);
        ushort angle2 = BitConverter.ToUInt16(key, 2);
        var angleKey = (angle1, angle2);

        byte[] value = iterator.GetValue();
        using var ms = new MemoryStream(value);
        using var reader = new BinaryReader(ms);
        int count = reader.ReadInt32();
        var tripletSet = new HashSet<(ulong, ulong, ulong)>();
        for (int i = 0; i < count; i++)
        {
            ulong id1 = reader.ReadUInt64();
            ulong id2 = reader.ReadUInt64();
            ulong id3 = reader.ReadUInt64();
            tripletSet.Add((id1, id2, id3));
        }
        triangles[angleKey] = tripletSet;
    }
    ReportProgress($"Loaded {triangles.Sum(kvp => kvp.Value.Count)} triangles into memory.");
    return triangles;
}

[TechnikEmpire]

1. You're configuring two different LogSettings - just use one or the other.
2. You're expecting two different files (due to point 1)
3. You're using SimpleFunctions for raw (I think var-length) bytes. SimpleFunctions are basic, "empty" functions. You need to use one of the functions for your type. That will either be MemoryFunctions or SpanByteFunctions. SpanByteFunctions will expect you to pin your memory, so you should probably use MemoryFunctions.
4. You're manually invoking Recover instead of passing tryRecoverLatest : true in the ctor of the store. Just let FASTER figure out recovery by passing this.
5. Every time I've pulled my hair out with recovery issues, it was a problem with how my keys were being hashed. Since you're using the incorrect store functions to hash your byte[] keys, well, there you have it.
6. Note that FASTER is unofficially abandoned and all the development in the last couple of years has been happening in a hard-fork here: https://github.com/microsoft/garnet - Specifically here: https://github.com/microsoft/garnet/tree/main/libs/storage/Tsavorite

Notes:

When you're struggling with recovery issues, especially when changing default memory / size constraints, have a look at this sample:
https://github.com/microsoft/FASTER/blob/main/cs/samples/CacheStore/Program.cs

See:
https://github.com/microsoft/FASTER/blob/main/cs/samples/StoreVarLenTypes/MemoryByteSample.cs

[user29A]

Thank you. Yes, I did solve the log settings problem and did get it working again. Good lesson in using LLM to help code....only do tiny little bits at once, and commit every single step so you can walk it back; but isn't that the lesson for anyone...

Thank you for the link to the fork, and also the samples link.