Skip to content

Persist the BlobStore with two-layer modeling #9

@toru

Description

@toru

The original plan for blob storage was to keep things simple, and persist the blobs in a sequence of:

* Blob ID / Value Checksum (32-bytes)
* Reference Count (4-bytes)
* Value Length (8-bytes)
* Value (Length determined by Value Length)
* Metadata Checksum (4-bytes)

While simple, this format does not support lazy-loading efficiently. Instead, we will decouple the blob metadata as a fixed-length record from the unstructured dynamic value data. The new fixed-length metadata format is:

* Blob ID / Value Checksum (32-bytes)
* Reference Count (4-bytes)
* Value Length (8-bytes)
* Value Offset (8-bytes)
* Metadata Checksum (4-bytes)

With this change, value data is stored separately, after the fixed-length regions of the file. By adopting this revised format, we can sort the fixed-length metadata records by their Blob ID, and provide binary search capability for efficient lazy-loading.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions