When would Lance V2 Page has more than one buffer?

[mapleFU]

In the lance v2 doc, it says: https://github.com/lancedb/lance/blob/main/protos/file2.proto#L42

// Note: the number of buffers (BN) is independent of the number of columns (CN)
//       and pages.

And when I try to go through the code, I find the:

FieldEncoder::maybe_encode returns EncodeTask
- ArrayEncoder::encode(..., buffer_index: &mut u32)

However, I found most of ArrayEncoder::encode implementations would only add add one buffer_index. So I'm curious that:

1. What's purpose of de-couple page and buffer?
2. Currently, would there any Page would contains more than one buffer?

Thanks!

[westonpace]

1. What's purpose of de-couple page and buffer?

When performing random access we don't read the entire page. This means we need to know where different buffers are located so that we can read into them appropriately. For example, with string data, we first read the offsets (first buffer) and then use those offsets to read into the string data (second buffer). In 2.1 we introduce an "initialize" step where we load (and cache) various small metadata (e.g dictionaries, chunk sizes, etc.) and we need a separate buffer for this metadata.

2. Currently, would there any Page would contains more than one buffer?

I think in 2.0 we use multiple buffers for binary / string data. List data is encoded as two different columns and so we only need a single buffer there.

In 2.1 things are a bit more complex. Small types are encoded using a "mini block" encoding that is more similar to parquet (chunks of data with small read amplification). We always have a tiny "block metadata" buffer that tells us the size of each block (2 bytes per block) and then the data buffer itself (in this case there is only one data buffer, no matter what data type or compression). Large types are encoded using a "full zip" encoding that will have, at a minimum, at least one buffer (e.g. this is how vector embeddings are encoded). There will be two buffers if it is a variable-length type (one buffer for something called the "repetition index").

In both "mini block" and "full zip" we may have additional buffers for metadata like "dictionaries". In the future I think we might combine all these extra metadata buffers into a single metadata buffer (reduce the IOPS on initialization) but this isn't too critical.

[westonpace]

And also, 👋 👋 , good to see a friendly face 😃

[mapleFU]

Thanks! In the document and code, I found that (please point out if I'm wrong) :

1. By default, Page will trying to be 8MB and trying to be less than 32MB
2. Some metadata buffers will be at "Global Buffer" area, and some types, like blob, might have separate buffer or out-of-line file

In the code, does Page buffer_index is just inline buffer? Or it can point to out-side buffer?

Besides, multiple buffer is a great idea when doing memory lookup. When considering io, does "String" type ( with variable length ) do whole-page io once? Or it would separate to a two-phase io scheduling( and would this makes small object-store io)? I found that BinaryPageScheduler try to schedule indices and binary separately. Does this use (possible) higher latency to get higher read bandwidth?

[mapleFU]

There will be two buffers if it is a variable-length type (one buffer for something called the "repetition index").

Oh I got to understand this. BinaryEncoder and other Encoder might have more than one buffer, and PackedStruct PackedStructFixedWidthMiniBlock would also try to uses this if not "transmute" the buffer?

[westonpace]

By default, Page will trying to be 8MB and trying to be less than 32MB

Yes

Some metadata buffers will be at "Global Buffer" area, and some types, like blob, might have separate buffer or out-of-line file

Yes

In the code, does Page buffer_index is just inline buffer? Or it can point to out-side buffer?

Only inline buffers

When considering io, does "String" type ( with variable length ) do whole-page io once? Or it would separate to a two-phase io scheduling( and would this makes small object-store io)? I found that BinaryPageScheduler try to schedule indices and binary separately. Does this use (possible) higher latency to get higher read bandwidth?

If we are reading all rows in the page we should be able to do whole page I/O as a shortcut. That would be better for large scans. However, I don't think I ended up implementing this shortcut (yet) because I haven't had issues with scan performance yet. I may add it back at some point in the future.

When scanning we are usually not IOP bound and instead are BW bound. So introducing a few more IOPS isn't too bad.

Oh I got to understand this. BinaryEncoder and other Encoder might have more than one buffer, and PackedStruct PackedStructFixedWidthMiniBlock would also try to uses this if not "transmute" the buffer?

Yes, I think most of these will use multiple buffers.

[mapleFU]

Thanks! I've no problem now. Excellent design!