LVEE: Lance Video Encoding Extension

[Xuanwo]

I want to introduce Lance Video Encoding Extension (LVEE) as a pluggable module for the Lance columnar format. LVEE delivers two key capabilities:

1. Encode‑on‑write – ingesting raw frames or streams and persist them as compressed containers (e.g. MP4/AV1) atomically alongside other columns.
2. Progressive rewrite – during background compaction LVEE can reorganise or re‑encode data at GOP‑level granularity, gradually reducing random‑access latency and storage cost.

With LVEE, Lance retains its column‑oriented strengths while offering sub‑second, frame‑level random access that modern multi‑modal applications require.

Motivation

Industry patterns

Product / Project	Storage Layout	Random‑Frame Strategy	Query Layer
Google Cloud Video Intelligence	MP4 blobs in GCS	Range requests via offset tables	Full‑text search over AI labels
Hugging Face Datasets Video	MP4 blobs stored as Binary	Lazy seek through torchvision.io.VideoReader	Custom user pipelines
Milvus + Twelve Labs	S3 objects + external offsets	Same as above	ANN search over multi‑modal embeddings
Qdrant × Dailymotion	GOP‑level objects on object storage	Pre‑cut GOPs → 8 ms seek	ANN + metadata filters

All adopt a “encode early → index offsets → external search” model. VEE brings the same model inside Lance’s transactional, columnar ecosystem.

Goals & Non‑Goals

Goals

• Register two encodings – lvee.blob and lvee.gop – and support write, read, and compaction conversion between them.
• Expose a language‑agnostic minimal API, for example:
  • get_frames_in_range(start_ms, end_ms)
  • rewrite_video(codec="h265", gop_size=48)
• Distribute the implementation as plugins:
  • Rust crate / dynamic library lance‑vee
  • Python wheel pylance‑vee as with an optional dependency for pylance

Non‑Goals

• Live RTSP/HLS ingestion (future work)
• GPU‑accelerated transcode (future work)

Alternatives

1. Keep using the generic Lance Blob API

Pros

• No new encoding IDs or plugins to maintain.
• Fastest path for writers: just dump bytes.
• Works out-of-the-box with every existing reader.

Cons

• Lance has no knowledge the bytes are video → cannot expose fps, resolution or duration for push-down filters.
• Random frame access requires decoding the whole file – latency is 100-1000× higher than GOP-aware layouts.
• Compaction cannot target hot segments; you can only rewrite the entire blob.
• Users must hand-roll export / playback scripts – poor developer experience.

2. Ship only lvee.gop and skip lvee.blob

Pros

• Only one encoding to implement and test.
• Every video immediately benefits from optimal random-seek latency.

Cons

• Ingest path must segment into GOPs on the fly – higher CPU usage and slower writes.
• No lossless “container-preserving” mode; the original bit-exact stream is discarded unless re-muxed.
• Sequential playback / archive workloads pay ≈2 % extra storage and manifest bloat they might not need.
• Migrating existing blob data requires a full rewrite with GOP segmentation – no incremental path.

3. Dual-path lvee.blob → lvee.gop (this proposal)

Pros

• Fast ingest through lvee.blob; background compaction upgrades only hot data to lvee.gop.
• Even in blob mode we store video-aware metadata (codec, fps, duration, keyframes).
• Supports lossless round-trip export and progressive optimisation.
• Users opt-in to complexity only when workloads demand it.

Cons

• Two encodings to support and document.
• Slightly more manifest logic to track conversion state.

---

What do you think?

[westonpace]

If the user already has the video stored in a GOP-aware format (e.g. MP4) and they know the keyframe offset they want to retrieve, wouldn't the blob API already allow them to do this using a video reader library?

[Xuanwo]

wouldn't the blob API already allow them to do this using a video reader library?

Yes, I agree with your point: storing MP4 blobs already accomplishes the task (mostly).

However, I think there's a gap we can address:

• Built-in intelligence. Once a column is defined as vee.blob or vee.gop, Lance can automatically expose properties like fps and duration, enabling filters such as WHERE fps > 60 to be pushed down without any additional ETL work.
• Background optimization. The same snapshots used for tracking data can also manage background upgrades from blob to GOP, all without breaking user code.
• A unified user experience.

In short, building an extension on top of our existing encoding framework can transform video blobs from bytes we happen to store into first-class, self-managing assets. This feels like the missing piece that would make Lance a truly complete multimodal solution.

[westonpace]

Built-in intelligence. Once a column is defined as vee.blob or vee.gop, Lance can automatically expose properties like fps and duration, enabling filters such as WHERE fps > 60 to be pushed down without any additional ETL work.

This would be useful. Would you imagine we extract these automatically and store them as separate columns? Or would we have functions to pull it from the metadata (this might be quite expensive since the metadata is not contiguous).

I think both approaches could also be handled at the table level. If we extract as separate columns then the storage layer just sees them as regular columns. If we have function on the metadata then blob API gives us this.

We'd also want to make sure we can index these columns.

Background optimization. The same snapshots used for tracking data can also manage background upgrades from blob to GOP, all without breaking user code.

What is an update from blob to GOP? Is this just the case where the user stored something like a pytorch tensor or some non-GOP format?

In short, building an extension on top of our existing encoding framework can transform video blobs from bytes we happen to store into first-class, self-managing assets. This feels like the missing piece that would make Lance a truly complete multimodal solution.

Agree 100% we want this at the level of "LanceDB". However, I don't know how much needs to be pushed into the table format and how much needs to be pushed into the storage format (and maybe there are other layers?) I don't want to put too much complexity into the file format if we can solve it with the same efficiency at a higher layer.

[jackye1995]

I don't want to put too much complexity into the file format if we can solve it with the same efficiency at a higher layer.

I originally thought having the video encoding was actually quite aligned with the principle of Lance V2 as you wrote in the blog and serves as an example of adding more purpose-built encodings in a flexible way:

Feature 1: Encodings are Extensions
Lance v2 does not have encodings. These are handled entirely by extensions. The file readers and writers are completely unaware of any kind of encoding (even plain encoding). The data producer’s choice of what encoding to use and the details of how that encoding works are determined by plugins. New encodings can be written without any modification to the file format, the file writer, or the file reader. This is done by storing all encoding descriptions as protobuf “any” messages. This is why we refer to Lance v2 as a "columnar container format" (inspired by container formats used for storing video).

Corollary 2: Empowering encoding developers
This makes it extremely easy to add a new encoding to Lance and helps to make the ecosystem fragmentation that Parquet has seen (e.g. mixed support for page lookups, delta encoding, etc.) more manageable. If I develop some new encoding then I just need to publish a .proto file to standardize how the encoding is described and write at least one implementation of the encoder/decoder. If a user tries to read a file written with this encoding, and their reader does not support it, then they will receive a helpful error “this file uses encoding X and the reader has not been configured with a decoder”. They can then figure out how to install the decoder (or implement it if need be). All of this happens without any change to the Lance format itself.

but maybe I misunderstood 😅 sounds like the pluggable encoding in your sense is more "fundamental" (if that is the right word) for features like delta encoding, compared to the video encodings that are proposed here?

[westonpace]

No fair using my own words against me 😆

I can go back and forth on this honestly. I do think a custom video encoding (e.g. some custom GOP encoding) could be an appropriate file format plugin. It could even be at the structural layer (e.g. similar to list) to represent an indexable type and add support for nested access (e.g. if accessing myfile[mycolumn][0] means to grab the first row of mycolumn then maybe myfile[mycolumn][0][0] means to grab the first GOP in the same way that myfile[mylist][0][0] could mean to grab the first item of the first list).

I think my hesitancy here is just general hesitancy faced when reinventing the wheel (e.g. MP4). Maybe instead of inventing a new codec this is about adding more integrated support for an existing one like MP4?

However, if we were going to rewrite an existing format then I think a potential win could be separating the metadata and the content. For example, the video content could be treated like we treat large binary in 2.0. We just dump it as soon as we see it and don't put it in pages. Then the page can contain the video metadata and pointers to the video buffers. This way we can quickly read all video metadata in one contiguous read. E.g. we could quickly answer a query like "give me the position and count of all keyframes in all videos in this column".

Also, what is the format the data is in when the user is going to write the video column? Do we expect it to already be in MP4 or do we need to support conversion from other formats? We probably don't want to rewrite ffmpeg.

Also, with video, we are going to quickly run up against the file size problem. Storing video inline with data leads to files getting too large. We want 1M rows per column generally (for good read patterns of non-video columns) but 1M rows of video is typically impossible.

[westonpace]

However, if we were going to rewrite an existing format then I think a potential win could be separating the metadata and the content.

Also, with video, we are going to quickly run up against the file size problem.

I think this opens the door to an even more interesting question which is "what happens if we write the metadata in the data files and the content in separate files?".

E.g. #3996

[Xuanwo]

I think my hesitancy here is just general hesitancy faced when reinventing the wheel (e.g. MP4). Maybe instead of inventing a new codec this is about adding more integrated support for an existing one like MP4?

I agree. It's definitely not our goal to rewrite ffmpeg.

My current expectation is that users will handle the video's metadata and GOP offsets (such as stts/stsc/stco/stsz in MP4), and LanceDB can offer a way, like a convenient API, for users to write or update them simultaneously.

From this perspective, I think having a separate column just for them could be a great idea.

My concern is that I want those columns to be grouped together, so users must write or update them simultaneously to prevent metadata or GOP from becoming mismatched.

Also, with video, we are going to quickly run up against the file size problem.

I believe it's something maybe we can address in blob v2. I'm start some discussion there.

[jackye1995]

Would you imagine we extract these automatically and store them as separate columns?

This reminds me of the JSON/VARIANT discussion #3841, where we can generate new columns vs implement it as a self-contained variant type. My impression is that, having new columns would always be more clear since user can reference it easily, and things like predicate pushdown could happen naturally without special handling in the reader. Maybe that means we should have dedicated types like MP4 type in Lance table, and that implicitly translates to a few grouped columns and writer will always write to those columns together? (which seems pretty aligned with the column family concept)

[westonpace]

This is an interesting point because I do kind of see value in having JSON / VARIANT as a self-contained variant type in some ways. I think there are two potential advantages:

• The variant type allows for a column to switch type from row to row
• The variant type can lead to fewer overall columns in the schema

Now I think I'm just talking myself in circles 😆

I think in both cases (VARIANT and MP4) we are talking about a column that is more like a struct. With the struct type there is both table-level support (e.g. each field in a struct has a unique field id) and storage-level support (structs fields are independently loadable and are shredded or packed as they are written to the disk)

I think giving field ids to MP4 sub-fields, allowing them to be indexed, etc. is a no-brainer. I think it is a bit trickier with a VARIANT type (do we really want to give field ids to every possible sub-field in the variant?) Maybe the VARIANT starts with no field ids initially but we assign one if the user wants to do something like create an index on the sub-field?

[Xuanwo]

So, in general, it seems we don't need to make changes at the lance file level. Instead, what we might need are a set of new utility APIs for videos.

Do we want to have new logical type just for Video? And is it a good idea for us to parse video files during write? I some how feel it's a good DX, but we may need to play with ffmpeg.

[westonpace]

Maybe a good order is:

1. Arrow extension type for MP4 (storage type is binary or large binary, not sure we need to do much here other than label it)
2. Table support for MP4 (dedicated field IDs for metadata)
3. Storage support for MP4 (pull metadata fields out into special columns, easy to read)

I don't have a strong opinion on who pulls the data into special columns. It can be table format or it can be storage format. I don't know if I know which one is the right one.

[Xuanwo]

Thank you @westonpace and @jackye1995 for joining the discussion. I will take a closer look at the video's use case and put together a new proposal if need.