Skip to content

Take advantage of transparent encoding in mini-block #5066

@westonpace

Description

@westonpace

The mini-block encoding scheme does not mandate transparent compression. Generally the cost of decompressing and entire block is cheap compared to the I/O so there is little need for it. However, sometimes users have data in memory, and sometimes the compression chosen is transparent (e.g. for string/binary data in 2.1 we use plain or fsst, both of which are transparent). We should pass in the expected range to the miniblock decompress function. If the encoding can make use of it, then it should. If it can't, then it doesn't need to.

This will have the most benefit on FSST encoding binary data that happens to already be in memory.

This change should be compatible with 2.1 and non-breaking as we are only changing how we read the data, not the format of the data itself.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions