How to persistent documents while using MilvusDocumentStore?

[HarshTrivedi]

Hi!

Thanks for making this useful library!

How can I persist the documents in MilvusDocumentStore?

I've two scripts: The 1st one populated the MilvusDocumentStore and generates embeddings for it, and the 2nd one reloads the MilvusDocumentStore queries it. I've kept them separate as the 1st one only needs to be run once. But I'm not able to load the documents in the 2nd script.

Since MilvusDocumentStore doesn't have save/load methods, I assume it's supposed to persist automatically (especially after reading this).

# script 1 (populate document_store, skipping the embedding details here)
from haystack.document_stores import MilvusDocumentStore

document_store = MilvusDocumentStore()
document_store.write_documents([{
    "content": "sample text", "meta": {"title": "title", "text": "sample text"}, "id_hash_keys": ["meta"]
}])
print(f"Num documents: {len(document_store.get_all_documents())}")
# Output => Num documents: 1

# script 2 (reload document_store, skipping the querying details here)
from haystack.document_stores import MilvusDocumentStore

document_store = MilvusDocumentStore()
print(f"Num documents: {len(document_store.get_all_documents())}")
# Output => Num documents: 0

Thanks!

[HarshTrivedi]

Can someone help here? (cc @brandenchan @vblagoje @tholor @MichelBartels @lalitpagaria)

[anakin87]

Hello @HarshTrivedi!

Here you find the MilvusDocumentStore docs.

Maybe this old issue can help (even if something has changed): #1600
Did you try to initialize Milvus and specify the sql_url?
Something like document_store = MilvusDocumentStore(sql_url="sqlite:///mydb.db", ...)

[HarshTrivedi]

@anakin87 Thank you for the response! I didn't realize I had to pass sql_url to the initializer to make it persistent. It works now! Perhaps, a warning in this case could be useful, since in pretty much all cases (except maybe debugging), one would want to have it persistent. A default of sql_url="sqlite:///mydb.db", would also perhaps be better.