Start/Update elasticsearch instance with 128 embedding dims

[heyt0pe]

Hello, I just got started with Haystack not too long ago and I've been to run a simple question and answer python script and also been able to set up the UI and rest API using docker-compose. For elastic search in docker, I already set the image to "elasticsearch:7.9.1" and not the instance with countries and capitals, so that it will start an empty elasticsearch instance and I can add my documents myself.

But now, whenever I want to set up anything using Elastic search as my document store, I get a series of errors. If I leave the default value for embedding_dims as 768, I get an error saying "Embedding dim. of model (128) doesn't match embedding dim. in DocumentStore (768).Specify the arg embedding_dim when initializing ElasticsearchDocumentStore()" and whenever I specify the embedding_dim to 128, I get another error saying there is a conflict with the existing mapping "'Mapper for [embedding] conflicts with existing mapping: [mapper [embedding] has different dims]'". I want to believe I get this error because the first time I ran Elastic search, it's embedding dims where set to 768 which is also visible when I visit http://localhost:9200/document/_mapping . Now I want to know if there's any way to get around this error, the only way so far is using an entirely different document store like FAISS, but is there a way I can fix this issue while still using Elastic Search. Thanks for your time

[julian-risch]

Hi @heyt0pe great to hear that you started trying out Haystack! The problem that you describe should be easy to fix, let me explain how.
The problem seems to be that your ElasticsearchDocumentStore already contains an index with 768 dimensions for embedding vectors. Now, if you want to store another document in that same DocumentStore but the document has only 128 dimensions in its embedding vector, you cannot store it in the same index. It's a mismatch. The number of dimensions is of the document's embedding vector depends on the model that is chosen to embed the documents. This model is set in the retriever node.
What you need to do whenever you change that model and therefore in some cases implicitly change the number of dimensions of the embeddings is to either delete the existing index before or create a new index with a different name.

When you initialize the document store, you can set recreate_index to True so that an existing index with the same name is overridden. Note that any indexed documents will be lost from that existing index.

import os
from haystack.document_stores import ElasticsearchDocumentStore

# Get the host where Elasticsearch is running, default to localhost
host = os.environ.get("ELASTICSEARCH_HOST", "localhost")
document_store = ElasticsearchDocumentStore(host=host, username="", password="", index="document", recreate_index=True)

As an alternative, when you initialize the document store, you can pick a new name for the index, e.g. document-dim-128

import os
from haystack.document_stores import ElasticsearchDocumentStore

# Get the host where Elasticsearch is running, default to localhost
host = os.environ.get("ELASTICSEARCH_HOST", "localhost")
document_store = ElasticsearchDocumentStore(host=host, username="", password="", index="document-dim-128")

[heyt0pe]

Hey Julian, thanks so much for responding. Would give it a try now

[JamesCHub]

I found you needed to specify both the preferred embedding dim and to recreate_index. I think using a distinct name is a good idea no matter what. So this is what worked for me:

ourDocStore = ElasticsearchDocumentStore(
    host=host,
    username="",
    password="",
    index="dim-384-document",
    recreate_index=True,
    embedding_dim=384
)