Data Pipeline recommendation

[Novumfinance]

Hi guys,

I would like some tips/recommendation for my use-case.

I would like to build a knowledge database (eg. with data coming from google drive (pdf's, excel, word..) and other data sources like SAP for instance, ingested via airbyte connectors. Then, build a user interfance (chatbot) that addresses questions and taps into the ingested knowledge base to provide answers to the user (RAG).

Could you please recommend an architecture/pipeline that would be most efficient in terms of performance and also cost-wise?

Example:

1. Ingest PDF, word etc. via connectors
2. Store data into a database like PostgreSQL?
3. Vectorize the documents? -> Pinecone? Does it make sense to vectorize all the documents? Imagine if the knowledge base as thousands of documents, what is the best and most efficient way?
4. Run LLM on top of Pinecone + user interface

It would be great if someone could help me sort this out and share their experience or any ressources.

Thank you