Is there a huge gap of reading performance between LanceVersion V2_0 and V2_1

[ChongWei905]

I've done some experiments and found that java sdk LanceFileWriter writes files with a bad performance on reading, while python sdk dataset.write_batches did well.
Then I figured out that its about the lance version. I tried to write V2_0 and V2_1 with same data(1,000,000 rows 10 cols all string data), and the reading performance has a huge gap:
V2_0:
random access time: 63.767ms
full scan time: 103.8345ms
V2_1:
random access time: 19.836317208s
full scan time: 2.189263209s

bellow is my read test code:

pub async fn test_full_scan_from_lance_file(
    file_uri_str: &str,
    take_rows_array: Vec<u32>,
) {

    let object_params = ObjectStoreParams {
        storage_options: Some(HashMap::new()),
        ..Default::default()
    };
    let (obj_store, path) = ObjectStore::from_uri_and_params(
        Arc::new(ObjectStoreRegistry::default()),
        &file_uri_str,
        &object_params,
    ).await.unwrap();
    let config = SchedulerConfig::max_bandwidth(&obj_store);
    let scan_scheduler = ScanScheduler::new(obj_store, config);

    let file_scheduler = scan_scheduler
        .open_file(&Path::parse(&path).unwrap(), &CachedFileSize::unknown())
        .await.unwrap();
    let file_reader = FileReader::try_open(
        file_scheduler,
        None,
        Arc::<DecoderPlugins>::default(),
        &LanceCache::no_cache(),
        FileReaderOptions::default(),
    ).await.unwrap();
    let metadata = file_reader.metadata();
    let schema = file_reader.schema();
    let read_batch_params: ReadBatchParams;
    if (take_rows_array.len() > 0) {
        read_batch_params = ReadBatchParams::Indices(UInt32Array::from(take_rows_array));
    } else {
        read_batch_params = ReadBatchParams::RangeFull;
    }
    let mut arrow_stream = file_reader
        .read_stream_projected(
            read_batch_params,
            8192,
            16,
            ReaderProjection::from_whole_schema(file_reader.schema(), file_reader.metadata().version()),
            FilterExpression::no_filter(),
        ).unwrap();

    let mut batches: Vec<RecordBatch> = Vec::new();
    while let batch_result = arrow_stream.next() {
        match batch_result.await { batch => {
            if (batch.is_none()) {
                break;
            }
            let b = batch.unwrap().unwrap();
            println!("batch size: {}", b.num_rows());
        }
        }
    }

And here's how I wrote the files:

    // batches are just recordbatch vector.
    let write_schema = batches[0].schema();
    let batch_refs: Vec<&RecordBatch> = batches.iter().collect();
    let merged_batch = concat_batches(&write_schema, batch_refs).unwrap();
    let storage_options: HashMap<String, String> = HashMap::new();
    let object_params = ObjectStoreParams {
        storage_options: Some(storage_options),
        ..Default::default()
    };
    let write_file_uri_str = "./test_file_lance.lance";
    let (obj_store, path) = ObjectStore::from_uri_and_params(
        Arc::new(ObjectStoreRegistry::default()),
        &write_file_uri_str,
        &object_params,
    ).await.unwrap();
    let obj_store = Arc::new(obj_store);
    let obj_writer = obj_store.create(&path).await.unwrap();
    let mut file_writer = FileWriter::try_new(obj_writer, merged_batch.schema().as_ref().try_into().unwrap(), FileWriterOptions {
        format_version: Some(LanceFileVersion::V2_0),
        ..Default::default()
    }).unwrap();
    file_writer.write_batch(&merged_batch).await.unwrap();
    file_writer.finish().await.unwrap();

[littleDrew]

Hello, could you have a look as for above performance issue. @jackye1995 @majin1102 @westonpace

[jackye1995]

@Xuanwo I think this is related to the perf gap we have found lately, could you confirm it's the same issue?

[Xuanwo]

Could be related, but I don't think 2_1 is that slow, something might be off. @ChongWei905, could you please share the full project so I can take a closer look? Thanks in advance.

[ChongWei905]

yes, the following branch is how i tested the performance. I added a main.rs file in ./rust/lance/src .
https://github.com/ChongWei905/lance/tree/0.32.0-branch-reproduce

[Xuanwo]

Thank you @ChongWei905 for this, I will take a look