index per request

[naor-pragma]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

My data has a metadata that divides my data, so for each request I can get the data for one of them.
For example, if the value for "x" in my request is "a", I will do the request on data where the "x" in the metadata is "a".
So to make that, I use:
params={"Retriever": {"filters": {"x": request["x"]}}

The issue is that with a big amount of data, I would like that "x" value to be my index to make performance better.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Currently index is a parameter for data store, so if I create a different data store for each request, I'll have to probably create a different retriever which means load the model for each request which won't be ideal.
Is there any solution I miss here, and if not, can this be a feature where I specify the index in the pipe.run function instead?

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

I considered creating a different data store and retriever for each request but I assume it'll be very slow (correct me if I'm wrong).
I also considered to change the data store's index in my predict request with data_store.index = new_index but I'm afraid to do it in a multi-threaded program.

[sjrl]

Hi, @naor-pragma thanks for asking your question. I believe you can achieve what you want without needing to create a new retriever for each document_store.

1. You can add multiple indices to the same document store by specifying the index parameter in the DocumentStore.write_documents() method. The API docs explaining the parameters can be found here: https://haystack.deepset.ai/reference/document-store
2. It's possible to specify the index parameter to the retrieve() method of the retriever. Here are the API docs on all parameters for the Retriever.retrieve() method https://haystack.deepset.ai/reference/retriever.

So I believe you can pass this on by specifying

params = {"Retriever": {"index": YOUR_INDEX}}

With these two steps you should only need one document store to contain all of your different types of data and then only need one retriever which can search through your different indices by passing the index parameter.

Let me know if this helps!

[sjrl]

@naor-pragma I've also converted this into a discussion since this is the space we like to collect questions like this.

[naor-pragma]

@sjrl Thank you! That worked!