How to save and load document store with embeddings?

[superchargez]

In tutorial 12 (LFQA) section: Initalize Retriever and Reader/Generator do you always have to update embeddings? This is very time consuming. Is not it possible to retain embeddings or a way to store embeddings to be used next time?

I ran the notebook and then shut it down and immediately ran it again (without running the 'update embedding line') and this does not work.

Kindly guide how this (storing embeddings) can be achieved and recall them next time without invoking the line.

[superchargez]

There is save and load method in document store, kindly add a code below to use them, thank you.

[superchargez]

I think this should/could work only if I know WHEN (or how) to delete previous document store. as when I reach the line where document store is defined

The number of documents present in the SQL database does not match the number of embeddings in FAISS. Make sure your FAISS configuration file correctly points to the same database that was used when creating the original index.

error is generated and I can't delete document store with this command document_store.delete_documents() because this would just say document store is not defined.

Thank you for the effort you put into this.

[superchargez]

Also tried the following and failed (error message in the end)
1> Ran all code (as mentioned in tutorial above, that you wrote, thank you for that)
2> Interrupted Kernel (and even tried after shutting down jupyter and restarting code from begining)
3> Deleted the database file generated (faiss_document_store.db)
4> Ensure that save files exist (my_faiss_index.faiss, my_faiss_index.json)
5> Ran the following code

from haystack.nodes import Seq2SeqGenerator
from haystack.utils import convert_files_to_dicts, fetch_archive_from_http, clean_wiki_text
from haystack.nodes import EmbeddingRetriever, DensePassageRetriever
from haystack.document_stores.faiss import FAISSDocumentStore

document_store = FAISSDocumentStore(faiss_index_factory_str="Flat", vector_dim=128)

retriever = EmbeddingRetriever(document_store=document_store, embedding_model="yjernite/retribert-base-uncased", model_format="retribert")

document_store.delete_documents()
# document_store.save("my_faiss_index.faiss")
new_document_store = FAISSDocumentStore.load("my_faiss_index.faiss")

And this generates following error (as mentioned in commend above)

ValueError: The number of documents present in the SQL database does not match the number of embeddings in FAISS. Make sure your FAISS configuration file correctly points to the same database that was used when creating the original index

Thank you for helping again.

[saugatapaul1010]

+1 Same! Tried this approach. This only works if the existing instance of the .db file is removed before the execution of the above code. The problem is, during deployment, the runtime complexity of the Q/A is very high because everytime you need to create the document store object. Is there any workaround?

[superchargez]

@brandenchan kindly look at the issue again. Thank you

[astha-ts]

I am getting the same error still. Here's my get_document_store function:

def get_document_store(doc_dir, reuse_saved_store=False):
    if os.path.exists("faiss_document_store.db"):
        os.remove("faiss_document_store.db")
        logging.info("removed existing faiss_document_store.db file which doesn't contain embeddings. ")
    doc_store_path = "my_faiss_index.faiss"
    if reuse_saved_store and os.path.exists(doc_store_path):
        document_store = FAISSDocumentStore.load("my_faiss_index.faiss")
        return document_store
    elif reuse_saved_store:
        logging.warning(f"I was asked to reuse saved store but missing fille {doc_store_path}. I will process the docs "
                        f"again. ")
    else:
        pass
    all_docs = convert_files_to_docs(dir_path=doc_dir)
    preprocessor = PreProcessor(
        clean_whitespace=True,
        clean_header_footer=True,
        clean_empty_lines=True,
        split_by="word",
        split_length=100,
        split_overlap=5,
        split_respect_sentence_boundary=True,
    )
    docs = preprocessor.process(all_docs)
    print(f"n_files_input: {len(all_docs)}\nn_docs_output: {len(docs)}")
    document_store = FAISSDocumentStore(faiss_index_factory_str="Flat")
    retriever = EmbeddingRetriever(
        document_store=document_store,
        embedding_model="sentence-transformers/multi-qa-mpnet-base-dot-v1",
        top_k=3
    )
    document_store.write_documents(docs)
    document_store.update_embeddings(retriever)
    if os.path.exists("faiss_document_store.db"):
        os.remove("faiss_document_store.db")
        logging.info("removed existing faiss_document_store.db file which doesn't contain embeddings. ")
    if os.path.exists(doc_store_path):
        os.remove(doc_store_path)
        logging.info(f"Removed existing {doc_store_path}")
    document_store.save(doc_store_path)
    print(f"saved {doc_store_path}")
    document_store = FAISSDocumentStore.load("my_faiss_index.faiss")
    return document_store

I first call it by get_document_store(doc_dir, reuse_saved_store=False)
to generate the embeddings, but I get an error while running this line in the end: document_store = FAISSDocumentStore.load("my_faiss_index.faiss")

The error I get is : ValueError: The number of documents in the SQL database (0) doesn't match the number of embeddings in FAISS (1088). Make sure your FAISS configuration file points to the same database that you used when you saved the original index.

[Unn20]

What about ElasticsearchDocumentStore?

It doesn't have save/load methods, as I see in the API reference.

It would be nice to save calculated embedding for documents.

[ZanSara]

ElasticsearchDocumentStore writes the embedding into permanent storage. Even if you're using an Elasticsearch container, as long as you don't delete the container your embeddings will stay. Therefore, no need for save and load methods like for FAISS, which is stored in-memory and would be lost otherwise.

Hope it helps!

[MariaDavid30]

Hai @ZanSara , I am trying to run the Table QA system locally(in jupyter notebooks). I am running dockerized elastic search container.
"document_store.update_embeddings(retriever=retriever, update_existing_embeddings=False)" took 3-4 hours the first time I ran it. Then I commented the command but got no answers. Should I run the command everytime?

[SnoozingSimian]

There is no way to reliably load and query a Pinecone index as well. This is frustrating.

[ZanSara]

That is strange. Can you share your code?

[SnoozingSimian]

I was unable to recreate the Pinecone issue. Might have been something to do with my initialisation configuration being different to the one originally used to index my documents.