Existing Parquet with uint8 vectors to Lance - size grows 10x?

[liquidcarbon]

Hi, I'm trying to convert a large Parquet dataset containing a 1024-dim uint8 vectors to lance.

To create an index, lance wants FixedSizeListArray, so...

pq_in = "s3://mybucket/pq/"
lance_out = "s3://mybucket/lance/"
pfs = list(S3Path.from_uri(pq_in).iterdir())

for i, pf in enumerate(pfs[:9]):
    print(f"{datetime.now():%FT%T} {pf}")  # progress
    table = pq.read_table(pf.as_uri())

    # Replace the vector column with FixedSizeListArray (required by Lance)
    table = table.set_column(
        table.schema.get_field_index(vector_column),
        vector_column,
        pa.FixedSizeListArray.from_arrays(
            pa.array([item for vec in table[vector_column].to_pylist() for item in vec], type=pa.uint8()),
            list_size=1024
        )
)

    lance.write_dataset(table, lance_out, mode="append" if i else "create")

This conversion is 1) very slow 2) the resulting files are almost 10x the size of original ones (~200K row files were ~34MB in snappy Parquet, becoming ~300MB in Lance)

What am I doing wrong?

[wjones127]

This conversion is 1) very slow

I imagine this is where it's particularly slow. You are converting from PyArrow array -> Python list -> PyArrow array.

pa.array([item for vec in table[vector_column].to_pylist() for item in vec], type=pa.uint8()),

I would recommend instead using the pyarow.compute.cast() instead if you want to change the type. https://arrow.apache.org/docs/python/generated/pyarrow.compute.cast.html

2. the resulting files are almost 10x the size of original ones (~200K row files were ~34MB in snappy Parquet, becoming ~300MB in Lance)

This is surprising to us. We haven't tested much with uint8 vectors, so we will look more into this soon when we have some time.

[westonpace]

Ok, this makes sense then. Right now there is no compression of vector embeddings. It hasn't really come up before, since most embeddings are basically lossy compression already.

So these results sound about right.

In fact, the current default storage format (2.0) doesn't have much compression at all. We're slowly adding in more compression of types in 2.1. We could probably trigger compression for these.

Out of curiosity, what do these embeddings represent? Are you able to say much about how are they generated? It would be good to build a test corpus of this kind of data to make sure we handle it well (though I suppose "fixed size sparse vectors" is probably a sufficient description).

[liquidcarbon]

Chemical fingerprints. The common problem in drug discovery is how to find molecules you can physically buy and validate that are similar to those coming from ML / virtual screening.

For example, for a molecule of LSD, just 51 out of 1024 values are not zero:

import numpy as np
from rdkit.Chem import Mol, MolFromSmiles, rdFingerprintGenerator

s = "CCN(CC)C(=O)[C@H]1CN([C@@H]2CC3=CNC4=CC=CC(=C34)C2=C1)C"  # LSD
m = MolFromSmiles(s)
fpgen_morgan = rdFingerprintGenerator.GetMorganGenerator(radius=2, fpSize=1024)
mfpb1 = fpgen_morgan.GetFingerprintAsNumPy(m)  # bits - for Jaccard distance aka Tanimoto similarity
mfpi8 = fpgen_morgan.GetCountFingerprintAsNumPy(m).astype(np.int8)  # counts - for euclidean / cosine
print(mfpi8[np.nonzero(mfpi8)])

# non-zero values: [1 3 1 1 2 1 1 1 1 1 1 1 1 1 2 1 1 1 5 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 5 1 1 1 1 1 2 2 1 1 1 1 1 1 2 2 1]

Tanimoto similarity from Morgan bits appears to be most popular approach, probably for computational speed. Those bits represent non-zero counts of molecular features such as "this atom X bonds away from that atom" -- and the actual counts are available as well. It seems to me you're giving up some information on going from counts to bits, and indeed the search results on count vectors are quite different compared to results on bit vectors (overlap of 50-90% on top 50 hits in my limited study) - but I have no intuition yet for how big a deal it is. What do you think?

What's your take on usearch? Are there any lance vs usearch comparisons? I found usearch quite easy to use in terms of indexing, but the downside is that the keys are strictly only integers and you still have to go through original Parquet files to retrieve the hits.

As far as compression, usearch indices are also bigger than original parquet files: unum-cloud/usearch#597

[liquidcarbon]

test corpus of this kind of data

https://github.com/ashvardanian/usearch-molecules/ - this is an excellent piece of work by @ashvardanian

[westonpace]

What's your take on usearch? Are there any lance vs usearch comparisons? I found usearch quite easy to use in terms of indexing, but the downside is that the keys are strictly only integers and you still have to go through original Parquet files to retrieve the hits.

No comparisons I'm aware of. It looks like usearch is an implementation of HNSW. We have an HNSW index but it isn't often used because it's not as great with scale / cloud storage. Normally we're just using IVF-PQ.

However, we are currently limited to L2 / Cosine / Dot as our distance metrics so adding something like Jaccard or Tanimoto would be cool.

Also, if the embeddings are structured like that we could probably come up with a better way of storing them, both on disk and in memory.

So, I think there is a lot of opportunity here. Without that distance metrics it will be difficult to emulate something like usearch-molecules today.

[liquidcarbon]

Thanks for the insights!
I might take up a side project of building another dataset and perhaps tackle some comparisons.