Amoro integrates lance table format

[majin1102]

Recently, Lance format has sparked phenomenal discussion across technical communities​​. As a next-generation data lake format optimized for AI workloads such as ML pipelines, vector search, and multi-modal processing, Lance offers unique advantages in performance and flexibility. ​​Exploring synergies between Amoro's unified lakehouse management and Lance's AI-centric capabilities could unlock new potentials in scalable, intelligent data architectures.​​

What is Lance

Lance: https://github.com/lancedb/lance
As offical said:

Modern columnar data format for ML. Convert from Parquet in 2-lines of code for 100x faster random access, zero-cost schema evolution, rich secondary indices, versioning, and more.

Lance format could be considered as a new table format like Iceberg. The table format(aka. dataset) api and capabilities are similar except Lance was born in python and rust and more friendly to python and AI users. Under the table format layer, Lance also integrates its file format, which optimized IO and scheduling models for multi-modal data.

The key features of Lance that other table formats like Iceberg don't have:

1. Random access: the basic to support vector and text search. And this is the feature implemented on Lance table format layer
2. Multi-modal data: storing text, image, video, tensor and many other modal data makes Lance different. This is implemented by Lance file format which can be adopted by other table formats.
3. Data evolution: not schema evolution, Lance support appending a column data without coping any existing column data, which is very useful in scenarios like feature engineering.
4. Secondary index: unlike Iceberg, Lance originally supports defining secondary indexes including B+ Tree, inverted, vector index on schema and provides the maintainenance api for them.

[majin1102]

Lance benifits from Amoro

1. Amoro manages the workloads of index building, optimizing and syncronizing in lance datasets in a more automic, cheap and efficiant way
2. Amoro could serve as an out of box namespace(catalog) service for Lance users
3. Easily view and manage metadata of Lance datasets by Amoro

Amoro benifits from Lance

1. Make more general capibilities of data lake management.
2. Lead to broader applications and influence

Roadmap(mindstorm)

1. Integrates Lance namespace to detect new datasets
2. Support viewing lance metadata(table details)
3. Terminal supports SQL on Lance datasets
4. Support compaction, cleanup and index building scheduling and processing

[zhoujinsong]

Thanks for bringing up this discussion.

Lance format has indeed expanded the boundaries of data lake tables. People have often asked me how data lake tables (like Iceberg) can store unstructured data - now it appears that the Lance format provides an excellent answer to this question.

I am very much looking forward to the integration work between Lance and Amoro. From Amoro's integration history with Iceberg, we can see that an excellent management system can greatly simplify the usage burden of data lake tables, accelerate their adoption in enterprises, and further speed up the creation of business value, which is exactly the result my team hopes to achieve.

So big +1, not only as a contributor to Amoro, but also as a member of Tencent Cloud's big data team.

[xxubai]

If Amoro supports Lance, it would be a milestone for the project, as no other lake management platform currently supports this table format. Supporting Lance table management would be a significant step forward for Amoro in the domain of unstructured data. However, it also comes with many challenges. All in all, I believe it's a worthwhile endeavor.