v1.7.0

New Features

• SSTables now features different kind of indices and allows to implement your own
• New recordio v4 format that includes checksums and support for nil records
• SimpleDB now supports byte functions in addition to the string functions

What's Changed

• add null record counter to sstables by @thomasjungblut in #45
• Fix compaction with holes by @thomasjungblut in #44
• Compaction by tombstone ratio in simpledb by @thomasjungblut in #46
• refactor sstable index into a dedicated interface with a loader by @thomasjungblut in #48
• fix partial sstable writes by @thomasjungblut in #49
• add a slice based key index by @thomasjungblut in #50
• add a read benchmark for recordio by @thomasjungblut in #51
• use a faster buffer pool by @thomasjungblut in #53
• minor performance optimizations by @thomasjungblut in #55
• Add byte based functions to simpledb by @thomasjungblut in #56
• Add SeekNext to recordio mmap based readers by @thomasjungblut in #57
• add sstables benchmark for random reads by @thomasjungblut in #59
• Add offset to return of SeekNext by @thomasjungblut in #60
• fix seeking through magic number in data by @thomasjungblut in #61
• Add a disk-based index by @thomasjungblut in #58
• Add a map-based key index by @thomasjungblut in #62
• doc updates by @thomasjungblut in #63
• Add recordio v4 by @thomasjungblut in #65
• Add kaitai and tests for all recent versions by @thomasjungblut in #68
• Bump golang.org/x/text from 0.3.7 to 0.3.8 by @dependabot in #69
• set the slicekey index as the default in sstables by @thomasjungblut in #70
• increase disk index cache size by @thomasjungblut in #71

Again, I want to emphasize how much of a help @sebheitzmann has been and want to thank him for his contributions and ideas.

Full Changelog: v1.6.3...v1.7.0

v1.6.3

What's Changed

• Add a fonction to memstore to implement a fast Exists method on db by @sebheitzmann in #43

Full Changelog: v1.6.2...v1.6.3

v1.6.2

What's Changed

• Add buffer configurations to readers and writers by @sebheitzmann in #40

Full Changelog: v1.6.1...v1.6.2

v1.6.1

What's Changed

• RecordIO options for reader by @sebheitzmann in #39

Full Changelog: v1.6.0...v1.6.1

v1.6.0

What's Changed

• adding record nil support by @thomasjungblut in #34
• Compaction without tombstone by @sebheitzmann in #33

New Contributors

• @sebheitzmann made their first contribution in #33

Full Changelog: v1.5.0...v1.6.0

v1.4.0

New Features:

• move to Go 1.19 and generics
• skiplist is now generic
• sstables merging is supporting generics
• the priority queue moved to its own package, supports generics too
• recordio and sstables support DirectIO and aligned block writes
• recordio now support lzw compression
• simpledb is now covered with linearization tests
• simpledb's WAL can make use of DirectIO

Breaking Changes:

• generics in skiplist, priority queue and sstables merging
  ** this comes with changes in iterators and comparators
• simpledb.NotFound is now called simpledb.ErrNotFound

Housekeeping:

• benchmarking with ycsb
• new CI jobs
• update codeql to v2
• badge updates

v1.5.0

What's Changed

• add test for examples by @thomasjungblut in #21
• Bump gopkg.in/yaml.v3 from 3.0.0-20200313102051-9f266ea9e77c to 3.0.0 by @dependabot in #22
• update go to 1.21 by @thomasjungblut in #23
• Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 by @dependabot in #24
• Update to go 1.22 by @thomasjungblut in #25
• defer buffer pool returns by @thomasjungblut in #26
• using errors.is by @thomasjungblut in #27
• ensure close is called by @thomasjungblut in #29
• add crc checksums to sstables by @thomasjungblut in #28

New Contributors

• @dependabot made their first contribution in #22

Full Changelog: v1.4.0...v1.5.0

v1.3.1

minor patch release with bugfix from b83e8ca and without directIO support

v1.3.0

New Features:

• adding simpledb, a fault-tolerant embedded key-value store using all building blocks in this project
• adding simpledb crash tests, a test suite to exercise crash and recovery scenarios which can be reused for any other key/value store
• recordio now features buffer size tuning options and buffer reuse (directIO and io_uring improvements are in the making)
• improved performance in recordio - between 25x for snappy and up to 60x without compression, especially for small files
• new version (v1) for sstables removing protobuf for value serialization (internal change, old tables are still readable and backward compatible)
• improved performance in sstables: up to 10x faster scanning and merging of tables
• sstables is now merging via iterators, several new ways of compacting during merges - e.g. like in map/reduce
• wal features a non-fsync append for better throughput
• wal features manual rotations in addition to the size-based ones
• memstore has support for tombstones and sstable iterators to enable easier merging
• adding SuperSSTableReader: a wrapper for multiple sstables behind a single sstable interface
• adding EmptySSTableReader: for easier testing and nil-safe merging scenarios

Breaking Changes:

• errors are now properly wrapped per package for easier debugging, as per changes from golang 1.13 - some interfaces now require to handle errors via errors.Is compared to direct equality checks
• all New* methods now return the interface, previously it might've been a pointer to the struct

Notable Bug Fixes:

• 5b18807 fix in skiplist iteration and sstable reader - where an iterator would continue to iterate across the upper boundary

Housekeeping:

• updated documentation
• switched dev branch from master to main
• moved examples package to _examples
• adding some new badges
• add go vet in the CI

New protobuf lib

With the new release of the go protobuf library, I cleaned a couple TODOs, added more docs and updated all protos and make their appearance consistent.

That means there are some breaking changes in the API, they are fairly minor compared to the changes in the protobuf upgrade.