How is LlamaIndex storing data in Milvus ?

[joybanerjee08]

Hi guys,
I have been tasked with researching viability of using Milvus with LlamaIndex for storing more than a million documents.

I have used a simple code such as this to store a sample of 100 document, and retrieve some data:

vector_store = MilvusVectorStore(
    uri="http://localhost:19530/my_db", collection_name='my_collection', dim=384, overwrite=True
)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)
retriever = index.as_retriever(similarity_top_k=3)
response_chunks = retriever.retrieve("Details of Internal Combustion Engines")
print(response_chunks[0])

I can see the node id, text and similarity scores in the output of the above code.

Next I have used pymilvus to describe the collection, it gave me this output:

{"collection_name": "my_collection", "auto_id": False, "num_shards": 1, "description": "", "fields": [{"field_id": 100, "name": "id", "description": "", "type": <DataType.VARCHAR: 21>, "params": {"max_length": 65535}, "is_primary": True}, {"field_id": 101, "name": "embedding", "description": "", "type": <DataType.FLOAT_VECTOR: 101>, "params": {"dim": 384}}], "aliases": [], "collection_id": 453383211425148447, "consistency_level": 1, "properties": {}, "num_partitions": 1, "enable_dynamic_field": True}

I can see two fields, id and embedding. So where are the texts and metadata ? If they are not stored, then how is LlamaIndex fetching the text ?

[zc277584121]

hello @joybanerjee08 , as you can see, "enable_dynamic_field": True in the description, it means by default , milvus use the dynamic field feature to storage these metadatas and other fields except for some basic fields such as id, vector. Simply speaking, you can regard it as containing fields such as metadata, and it can dynamically add any other fields. You can also dive into our documents to get a better understand. https://milvus.io/docs/enable-dynamic-field.md#Enable-Dynamic-Field

[joybanerjee08]

Hi @zc277584121, I understand that with dynamic fields as true, we can add new fields later on the go. But the new fields like metadata or text are never created by llama Index. After loading all the documents, I'm only seeing two fields: id and embeddings.

[zc277584121]

Because only id and embeddings are standard fields, while other fields like metadata and text are in the dynamic field.
You can try this code to get a random entity in your collection:

print(vector_store.client.search(
    collection_name="my_collection", 
    data=[[0.1] * 384],
    limit=2,
    output_fields=["*"],
))

If everything is right, you will see all the fields.

If you want to get a more directly view of your milvus collections, you can try to use attu, which is a visualization tool for milvus and user friendly to get started.

[Ipomoeabatatas]

Hi... I am asking a question in the opposite way. Given the Milvus datastore, I am able to do a basic query in Llamaindex..

try:
# 1. Connect to Milvus
milvus_store = MilvusVectorStore(
uri=app_config.MILVUS_URL,
dim=app_config.MILVUS_DIM,
overwrite=False,
collection_name="_emails_attachments"
)
print("Connected to Milvus successfully.")
except Exception as e:
print(f"Failed to connect to Milvus: {e}")

index = VectorStoreIndex.from_vector_store(
vector_store=milvus_store
)

response = query_engine.query("Tell me about order #124 ...")

I like to know how can I inert a filter (using the meta data) into the query.

[yhmo]

Hi... I am asking a question in the opposite way. Given the Milvus datastore, I am able to do a basic query in Llamaindex..

try: # 1. Connect to Milvus milvus_store = MilvusVectorStore( uri=app_config.MILVUS_URL, dim=app_config.MILVUS_DIM, overwrite=False, collection_name="_emails_attachments" ) print("Connected to Milvus successfully.") except Exception as e: print(f"Failed to connect to Milvus: {e}")

index = VectorStoreIndex.from_vector_store( vector_store=milvus_store )

response = query_engine.query("Tell me about order #124 ...")

I like to know how can I inert a filter (using the meta data) into the query.

To find the answer, you need to read the open-source code. Different projects make different implementations.
In the query() method, the "string_expr" is generated from _prepare_before_search()
https://github.com/run-llama/llama_index/blob/e5b737b7fff222b1b9010d55a66e002c65268b48/llama-index-integrations/vector_stores/llama-index-vector-stores-milvus/llama_index/vector_stores/milvus/base.py#L734

In the _prepare_before_search(), the "expr" is combined from query.filters, query.doc_ids and query.node_ids
https://github.com/run-llama/llama_index/blob/e5b737b7fff222b1b9010d55a66e002c65268b48/llama-index-integrations/vector_stores/llama-index-vector-stores-milvus/llama_index/vector_stores/milvus/base.py#L797

query.filters, query.doc_ids and query.node_ids are members of VectorStoreQuery class:
https://github.com/run-llama/llama_index/blob/e5b737b7fff222b1b9010d55a66e002c65268b48/llama-index-core/llama_index/core/vector_stores/types.py#L234

They are llama concepts, you can read the llama documents to learn how to set their values.