Skip to content

Add "rolling compression" to file writer and encodings #4371

@westonpace

Description

@westonpace

The compression in 2.1 currently has a problem. We accumulate 8MB of raw data before we run compression. Then we compress the buffer and write a page. However, when the data is highly compressible, this could lead to a very small page being created. Instead of writing a page at that point we should instead wait and add more data.

This will require enhancements to all encoders as it will require them to be able to "save off state" and "resume with more data".

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions