JotDB

JotDB (JSON Optimized Tiny Database) is a lightweight, high-performance Go library for storing and retrieving JSON documents locally. It provides fast read/write operations, thread-safe concurrency, and a modular design, making it ideal for applications needing a simple, embedded key-value store for JSON data.

Table of Contents

• Features
• Installation
• Usage
  • Basic Example
  • API
• Performance
• Benchmarks
• Architecture
• File Structure
• Limitations
• Future Improvements
• Contributing
• License
• Acknowledgments
• Authors
• Version History

Features

• Fast JSON Handling : Custom JSON serialization/deserialization optimized for minimal allocations, supporting objects, arrays, strings, numbers, booleans, and null.
• Local Storage : Uses Bitcask for persistent, local key-value storage with high throughput and durability.
• Concurrency : Thread-safe operations with sync.RWMutex for concurrent reads and exclusive writes.
• Modular Design : Organized into multiple files (jotdb.go, json_marshal.go, json_unmarshal.go, types.go) for maintainability and future optimization.
• Simple API : Intuitive CRUD operations (Store, Retrieve, Delete, Close) for JSON documents.

Installation

1. Ensure you have Go installed (version 1.24 or later recommended).
2. Install JotDB and its dependency, Bitcask:

go get github.com/prologic/bitcask

3. Call JotDB from GO as github.com/deepfield-ml/JotDB/jotdb0.01 , run github.com/deepfield-ml/JotDB/jotdb0.01 then import github.com/deepfield-ml/JotDB/jotdb0.01 in code.

Usage

Basic Example

package main

import (
	"fmt"
	"jotdb"
)

func main() {
	// Initialize JotDB
	store, err := jotdb.NewJotDB("./jotdb")
	if err != nil {
		fmt.Println("Error:", err)
		return
	}
	defer store.Close()

	// Create a JSON document
	doc := map[string]interface{}{
		"id":   "doc001",
		"data": map[string]interface{}{
			"name":  "Example",
			"score": 42.5,
			"tags":  []interface{}{"test", "demo"},
			"active": true,
			"meta":   nil,
		},
	}

	// Store the document
	if err := store.Store("doc001", doc); err != nil {
		fmt.Println("Error storing document:", err)
		return
	}

	// Retrieve the document
	var retrieved map[string]interface{}
	if err := store.Retrieve("doc001", &retrieved); err != nil {
		fmt.Println("Error retrieving document:", err)
		return
	}
	fmt.Printf("Retrieved document: %+v\n", retrieved)

	// Delete the document
	if err := store.Delete("doc001"); err != nil {
		fmt.Println("Error deleting document:", err)
		return
	}
}

API

• NewJotDB(dbPath string) (*JotDB, error): Initializes a new JotDB instance at the specified path.
• Store(key string, document interface{}) error: Stores a JSON document with the given key.
• Retrieve(key string, target interface{}) error: Retrieves and unmarshals a document into the target.
• Delete(key string) error: Removes a document by key.
• Close() error: Shuts down the JotDB instance and releases resources.

Performance

• Write Throughput : ~100,000 writes/s for 1KB JSON documents on modern SSDs (thanks to Bitcask’s append-only log).
• Read Latency : Sub-millisecond reads via in-memory key lookups.
• JSON Overhead : Custom serialization/deserialization is ~1.5x faster than Go’s encoding/json, with ~20µs serialization and ~30µs deserialization for 1KB documents.
• Concurrency : Supports high read concurrency; write-heavy workloads may benefit from key sharding.

Benchmarks

JotDB has been benchmarked on a standard development machine (Intel Core i5, 16GB RAM, SSD):

Operation	Document Size	Operations/sec	Latency (avg)
Write	1KB	~100,000	~10µs
Read	1KB	~200,000	~5µs
Delete	-	~150,000	~7µs

JSON serialization performance compared to standard library:

Operation	Standard Library	JotDB Custom	Improvement
Serialization	~30µs	~20µs	~1.5x
Deserialization	~45µs	~30µs	~1.5x

*Benchmarks performed on 1KB JSON documents with mixed types (strings, numbers, booleans, arrays, nested objects)

Architecture

JotDB follows a simple architecture that combines efficient JSON processing with reliable local storage:

graph TD  
    subgraph "Client Interface"  
        Client["Client Application"] --> JAPI["JotDB API"]  
    end  
      
    subgraph "Core Components"  
        JAPI --> JStruct["JotDB Struct"]  
        JStruct --> RWMutex["Concurrency Controller"]  
        JStruct --> BitDB["Storage Interface"]  
    end  
      
    JStruct --> Marshal["JSON Serialization"]  
    JStruct --> Unmarshal["JSON Deserialization"]  
      
    BitDB --> FSStore["File System Storage"]

Loading

File Structure

• jotdb.go: Core JotDB struct and public API.
• json_marshal.go: Custom JSON serialization logic.
• json_unmarshal.go: Custom JSON deserialization logic.
• types.go: Placeholder for future type definitions or constants.

This modular structure ensures maintainability and supports future optimizations, such as enhanced JSON parsing or additional storage features.

Limitations

• JSON Parser : Supports core JSON types but lacks streaming for very large documents.
• Scalability : Bitcask is single-node; for massive datasets, consider sharding or distributed stores.
• Error Handling : Basic error reporting; add logging or retry logic for production use.
• Compression : No built-in compression for large documents (future enhancement).

Future Improvements

• Add streaming JSON parsing for large documents.
• Implement secondary indexes for faster queries.
• Support batch operations for bulk reads/writes.
• Add optional compression (e.g., zstd) to reduce disk I/O.
• Enhance error handling with detailed logging and retry mechanisms.

Contributing

Contributions are welcome! Please submit issues or pull requests to improve JotDB. Areas for contribution include:

• Optimizing the JSON parser for specific workloads.
• Adding support for advanced JSON features (e.g., streaming, schema validation).
• Implementing additional storage backends or indexing.

License

JotDB is licensed under the Apache 2.0 License. See LICENSE for details.

Acknowledgments

• Built with Bitcask for fast, local key-value storage.
• Inspired by the need for a simple, high-performance JSON store in Go.

Authors

• Deepfield ML - Gordon.H and Will.C

Version History

• v0.01 - Initial release with core functionality
  • Basic CRUD operations
  • Custom JSON serialization/deserialization
  • Thread-safe operations