Export and Import of Document Store

[nicir]

Hi there,

I'm trying to train a retriever for dense passage retrieval. The training works fine. And updating the embeddings of the FAISS document store also works fine.

But I've got a couple of trained dpr models to evaluate, which I have to update individually.

Each time I update a retriever I take the following steps:

• instantiate a FAISSDocumentStore:
  _document_store = FAISSDocumentStore(faiss_index_factory_str="Flat)

• convert files to document store
  docs = convert_files_to_docs(dir_path=doc_store_dir, clean_func=clean_wiki_text, split_paragraphs=True)

• write the documents into the document store
  _document_store.write_documents(docs)

The step "writing" takes very long (about 5 hours), as I have about 4M entities. I would like to export this document store to be able to import it every time I want to update the embeddings with another retriever but I don't see a possibility to export and import the file "faiss_document_store.db".

I appreciate every kind of help!

[nicir]

More detailled:

I dont think of the .faiss-file but the .db-file.

Not the .faiss-file I can import like this new_document_store = FAISSDocumentStore.load("my_faiss_index.faiss")

[vblagoje]

Hey @nicir haven't tried this myself but look at the FAISSDocumentStore. Point the sql_url and faiss_index to the appropriate files and that should be what you need. LMK how it went,

[nicir]

Do I necessarily need a faiss-file to import as well?

I tried to copy the .db-file from another instantiation into my venv, I want to update the embeddings in. So I put the .db.-file (1.7GB) inside the directory my script is in.

Then I tried to do this:
_document_store = FAISSDocumentStore(sql_url="sqlite:///faiss_document_store_BASE.db")

This doesn't work as there is no .faiss-file:
ValueError: The number of documents present in the SQL database (907483) does not match the number of embeddings in FAISS (0). Make sure your FAISS configuration file correctly points to the same database that was used when creating the original index.

After loading the document store I want to update the embeddings. But that is what I want to generate with this script. So loading it before I update the embeddings wouldn't make sence, or am I wrong?

This is what I do after instantiating the document store:
_document_store.update_embeddings(retriever, batch_size=330)
_document_store.save(retriever_dir_root + retriever_name + "/my_faiss_index.faiss")

[vblagoje]

Hey @nicir when you are doing: document_store = FAISSDocumentStore(sql_url="sqlite:///faiss_document_store_BASE.db") why not pass faiss index as well?

[nicir]

Because it's the faiss index I want to train. It's automatically created during updating the embeddings.

I just want to skip the write_documents()-Part because it takes a couple of hours to convert the documents into the .db-file

[anakin87]

As a workaround you can load both db and faiss index, then compute the new embeddings by update_embeddings method.