How to change the elastic search configuration during runtime ?

[agemagician]

Assume we have a single elastic database engine and must build a single application where each user should ask a question and retrieve an answer. However, each user should have access to a different index "database" on the elastic search.
The current implementation assumes a fixed document store information during application run-time, like :
https://github.com/deepset-ai/haystack-tutorials/blob/main/tutorials/03_Scalable_QA_System.ipynb

The questions are:
How can we change the document store username, password, and index during application runtime?
Otherwise, we must build a duplicate application for each user and each index.

[silvanocerza]

Hey @agemagician what are you trying to achieve? I don't think I'm understanding your use case. 🤔

[agemagician]

The example mentioned above has a fixed username, password, and index for elastic search.

How can we change the username, password, and index during the application's runtime?

Many users can use the same application, and each one could have his one username and password to access the Elastic search database. Furthermore, some users can access specific index on the elastic search.

In all the current examples, we assume the username, password, and index for the elastic search will be permanently fixed during the application runtime.

If we didn't define it early, then we can't define the retriever, and we can't define the rest of the pipeline.

If we defined early, then how we can change it during the pipeline calls ?

[silvanocerza]

You can't change the credentials after the ElasticSearchDocumenStore has been instantiated so you can't do it after it's been added to the Pipeline.

Also you're going to give users direct access to your DB that way, that's really dangerous.
I would suggest you handle authentication and authorization separately, and filter documents in your Pipeline depending on what the user that sent the request has access to.

[agemagician]

Thanks for your answer.

Two follow-up questions:

1. how to apply a filter if the index is fixed? The idea is to have different indexes for different use cases, but the index is permanently fixed in this case. As index in elastic search like database in SQL.
2. I have seen in the "EmbeddingRetriever" class the "document_store" is optional, and the retrieve function has a "document_store" input, does that mean during run-time we can provide a different document_store to the retrieve pipeline?
   For example, my current pipeline call is :

pipe.run(query=question,
                            params={"EmbeddingsRetriever": {"top_k": retriever_top_k},
                                    "Ranker": {"top_k": ranker_top_k},
                                    "prompt_node": {"prompt_template": prompt},
                                    })

Can we change it to ?

pipe.run(query=question,
                            params={"EmbeddingsRetriever": {"top_k": retriever_top_k, "document_store": document_store},
                                    "Ranker": {"top_k": ranker_top_k},
                                    "prompt_node": {"prompt_template": prompt},
                                    })

[ZanSara]

I have seen in the "EmbeddingRetriever" class the "document_store" is optional, and the retrieve function has a "document_store" input, does that mean during run-time we can provide a different document_store to the retrieve pipeline?

Yes, you should be able to do so and that's the approach I recommend if you have different credentials for each user. However it's a fairly rare usecase, so if it doesn't work please let us know which difficulties you face!

[agemagician]

Perfect, thanks a lot :)

I will close it here for now, and I will re-open it in case it doesn't work.