Creating different instances of document store and keeping track of the data files uploaded in each of them

[Harshal1810]

Hi all..I am new to Haystack.. I am using elasticsearch document store to create QA pipeline and question generation pipeline.. I am using FastAPI to create api-enpoints. I want to keep track of all the documents once we store them in elasticsearch document stores. I want to create different elasticsearch instances for different datasets for my project. Is there any way to do this? Has anyone did any project of this?

[masci]

Hi @Harshal1810 and thanks for using Haystack!

Haystack abstracts document databases with components called Document Stores. There is one that's specific for Elasticsearch called ElasticsearchDocumentStore that you can use to connect to one index from a certain cluster.

Whether you have different indexes in the same cluster or different clusters, the idea is that you create an instance of ElasticsearchDocumentStore for each one of them, something like:

ds_one = ElasticsearchDocumentStore(
    host=example.com,
    index="documents"
)

# same cluster, different index
ds_two = ElasticsearchDocumentStore(
    host=example.com,
    index="reviews"
)

# different cluster
ds_three = ElasticsearchDocumentStore(
    host=example.ai,
    index="documents"
)

Hope this helps!

[Harshal1810]

I am having this architecture in mind. Need to think how to design a classifier

[Harshal1810]

Any suggestion on how this classifier can be designed?

[masci]

I think that really depends on the logic you use for the partitioning - is there any way I can deduce whether the answer to a certain question lives in DS-1 rather than DS-3?

[Harshal1810]

yesterday you were talkin about parallelly calling all the indices. But how it can be done. Python script will exhibit lines of code one after the other na?

[Harshal1810]

Hi @masci Can we host Haystack Rest API on AWS S3 bucket to make use of GPU?