Trade Data Database

[sneilan]

Implementation Notes

• Use sqlite write ahead log to journal trades. Leverage SQLite for WAL and atomic commits instead of apache avro. Can tune settings for performance.
• Rotate trade journals by storing no more than 100k trades per file. Filename should be unique. Can put trade id / date etc in filename but rely on database because storing metadata in filenames is less scalable.
• Take checkpoints of market depth and event store in sqlite too.
• Track checkpoints in database to say which trade ID was latest when checkpoint taken.
• Restore checkpoints in sqlite to memory by deserialization.
• Do not save / restore checkpointed data by pure memory because the pointers to mmap buffers will misalign on reload. Plus saving memory data directly will also save any memory fragmentation.
• Replaying trade journals should result in correct system state.
• Add tests to make sure journals / replay work.
• Define data structures to store this.