How to reduce response time gap?

[dkbs12]

As a result of modularization using the notebook introduced in the tutorial, a response time gap occurred between the existing notebook codes and the modularized notebook codes, so I would like to inquire about this.
I made an Extractive QA model using BM25 Retriever and Embedding Retriever by referring to the Haystack Tutorial.
When I use existing notebook coding, it takes less than 1 second to answer in the question & answer step
However, if it is modularized using functions, it takes more than 10 seconds on average to qnswer the question.

In my opinion, when processing the question & answer, only the Inferencing Samples process is performed in existing(original) notebook codings, but the additional Updating Embeddings process is performed in modularized codings, so I guess it takes more response time.
Even in a modularized case, is there any way to chieve the same response time of less than 1 second as with existing notebook coding?
Please note that one original notebook coding and two modularized codings have been uploaded to github as shown below.

• extractiveQA_original_notebook.ipynb
• extractiveQA_modulization_01.ipynb
• extractiveQA_modulization_02.ipynb

Any advice or tips would be of great help to me.
thank you.

[vblagoje]

Hi @dkbs12 I looked at the notebooks - why don't you simply use the original notebook? I don't understand what's the problem. Of course, in the second notebook, every time you ask a question you run everything from scratch, including downloading data, inserting data in db, pipeline creation etc etc.

[dkbs12]

Hi @vladblagoje
Thank you for your quick response.
Actually, I'm planning to deploy those notebook codings using streamlit on colab.
As you know, when using streamlit, all the codings to deploy should be included in one app.py file.
So method of interactive interpreter such as original_notebook is not applicable. (every step in the notebook process in series without break in one app.py file.)
That's why I asked you how to reduce the time gab between two types of methods.

[vblagoje]

If I'm not mistaken, streamlit has a single thread of execution mechanism, so watch out for that. Please read a bit about how to do one-time initialization in streamlit and put those indexing and pipeline preparation function invocations there and you should be fine.