v0.3.0 Rusty Lances and Friendly Neighbors

[changhiskhan]

Sayonara C++, bonjour Rust

What started out as a holiday hack has become a full-blown Rust rewrite.
As we say farewell to our much beloved C++ implementation, we welcome a major new feature to Lance: the vector index.

1. Lance's vector index is fast and has a small memory footprint. From disk, we benchmark average latencies of 1ms on vanilla macbook airs for 1M vectors.
2. Your data, vectors, and index can live in harmony under one roof so you don't need to manage a separate index or service.
3. You can choose to manage and retrieve additional features with the vectors with very little performance impact.

What's Changed

• Only increase cursor if file success to write by @eddyxu in Only increase cursor if file success to write #435
• GHA to add python 3.11 (and upgrade to duckdb 0.6.1) by @changhiskhan in GHA to add python 3.11 (and upgrade to duckdb 0.6.1) #434
• ScannerStream accepts early stop by @eddyxu in ScannerStream accepts early stop #437
• upgrade arrow-rs to 31.0 by @eddyxu in upgrade arrow-rs to 31.0 #438
• L2 distance by @eddyxu in L2 distance #439
• Create DataFragment and DataFile during Dataset write process by @eddyxu in Create DataFragment and DataFile during Dataset write process #440
• Rust Dataset Write API by @eddyxu in Rust Dataset Write API #441
• [Rust] Read Partially from a plain encoded batch by @eddyxu in [Rust] Read Partially from a plain encoded batch #443
• Get range in var-binary encoding by @eddyxu in Get range in var-binary encoding #444
• Productionlize Flat Index by @eddyxu in Productionlize Flat Index #442
• Make Scan an ExecNode by @eddyxu in Make Scan an ExecNode #445
• Take record by Row ID by @eddyxu in Take record by Row ID #446
• Implement Take for dictionary decoder. by @eddyxu in Implement Take for dictionary decoder. #447
• Merge two RecordBatch by @eddyxu in Merge two RecordBatch #449
• Integrate flat index by @eddyxu in Integrate flat index #448
• Support limit offset as ExecNode by @changhiskhan in Support limit offset as ExecNode #450
• Read IVF_PQ index by @eddyxu in Read IVF_PQ index #451
• Cli to operate on dataset indices by @eddyxu in Cli to operate on dataset indices #452
• [RUST] python (re)integration v1 by @changhiskhan in [RUST] python (re)integration v1 #436
• Support writing dictionary values (at the dataset level). by @eddyxu in Support writing dictionary values (at the dataset level). #454
• Replace ObjectReader as a pub trait. by @eddyxu in Replace ObjectReader as a pub trait.  #459
• [Rust] Implement LocalObjectReader that holds an open file to improve performance. by @eddyxu in [Rust] Implement LocalObjectReader that holds an open file to improve performance. #460
• inherit from pyarrow Dataset/Scanner by @changhiskhan in inherit from pyarrow Dataset/Scanner #462
• [RUST] Flat index benchmark by @eddyxu in [RUST] Flat index benchmark #461
• Generate spotify dataset with embeddings. by @eddyxu in Generate spotify dataset with embeddings. #453
• Fix pylance typo and float32 array conversion. by @eddyxu in Fix pylance typo and float32 array conversion. #463
• Write index metadata with a new version by @eddyxu in Write index metadata with a new version  #466
• [rust] fix projection in Dataset:take_rows by @changhiskhan in [rust] fix projection in Dataset:take_rows #464
• blas feature flag by @changhiskhan in blas feature flag #467
• Sift dataset generation by @eddyxu in Sift dataset generation #472
• Improve scan perf by re-enable prefetching in ScanNode by @eddyxu in Improve scan perf by re-enable prefetching in ScanNode #473
• Changhiskhan/new docs by @changhiskhan in Changhiskhan/new docs #474
• Fix AVX and NEON L2 distance computation. by @eddyxu in Fix AVX and NEON L2 distance computation.  #476
• add recall metric computation by @changhiskhan in add recall metric computation #475
• Fix reader assertion on manifest buffer size by @eddyxu in Fix reader assertion on manifest buffer size #478
• [Rust] Minimal dataset append support by @eddyxu in [Rust] Minimal dataset append support #482
• Pass nprobes parameter from python by @changhiskhan in Pass nprobes parameter from python #480
• add a test_dataset function to compute the recall for lance by @changhiskhan in add a test_dataset function to compute the recall for lance #479
• Split sparse index read into chunks based on optimal I/O size for the media by @eddyxu in Split sparse index read into chunks based on optimal I/O size for the media #483
• Fix codespace prebuild by @eddyxu in Fix codespace prebuild #485
• Make ObjectReader prefetch size configurable by @eddyxu in Make ObjectReader prefetch size configurable #486
• Add a refine stage for vector search by @eddyxu in Add a refine stage for vector search #488
• add nprobes as parameter to benchmark by @changhiskhan in add nprobes as parameter to benchmark #484
• refine factor by @changhiskhan in refine factor #489
• Use ordered buffer in plain decoder by @eddyxu in Use ordered buffer in plain decoder #493
• New rust+pyo3 based pylance by @eddyxu in New rust+pyo3 based pylance #494
• Fast count rows by @eddyxu in Fast count rows #490
• Count rows in python dataset, and setup GHA again by @eddyxu in Count rows in python dataset, and setup GHA again #495
• Sayonara C++ by @eddyxu in Sayonara C++ #497
• [Rust] Dataset Overwrite, and Version Checkout by @eddyxu in [Rust] Dataset Overwrite, and Version Checkout #496
• Load S3 credentials using default credentials chain by @eddyxu in Load S3 credentials using default credentials chain #498
• Fix doc build by @eddyxu in Fix doc build #499
• File format spec by @eddyxu in File format spec #500
• Doc build fix by @eddyxu in Doc build fix #501
• Schema evolution document by @eddyxu in Schema evolution document #503
• update the python readme for pypi by @changhiskhan in update the python readme for pypi #504
• Handle null strings for both cases where nullability is set or not. by @eddyxu in Handle null strings for both cases where nullability is set or not. #509
• update main github readme by @changhiskhan in update main github readme #508
• [python] write_dataset returns new dataset by @changhiskhan in [python] write_dataset returns new dataset #517
• Changhiskhan/list versions by @changhiskhan in Changhiskhan/list versions #516
• Refine Factor is None by default by @eddyxu in Refine Factor is None by default  #518

Full Changelog: v0.2.9...v0.3.0

---

This discussion was created from the release v0.3.0 Rusty Lances and Friendly Neighbors.