blog/rag-pipeline-tutorial

[giscus[bot]]

blog/rag-pipeline-tutorial

Step by step tutorial: how to build a production-ready RAG pipeline.

https://zackproser.com/blog/rag-pipeline-tutorial

[scott91e1]

Thanks for publishing the tutorial, very helpful. I have a question about the project I am working on. Claude suggested that for JSON RAG that this would be an acceptable format:

{
  "table_metadata": {
    "title": "Product Inventory",
    "last_updated": "2025-01-03"
  },
  "columns": ["product_id", "name", "quantity", "price"],
  "data": [
    {"product_id": "P001", "name": "Widget A", "quantity": 100, "price": 19.99},
    {"product_id": "P002", "name": "Widget B", "quantity": 150, "price": 24.99}
  ]
}

Would it be possible to ingest and query these types of documents into your solution?

[zackproser]

@scott91e1

Thank you so much for the feedback. I'm really glad to hear the tutorial was helpful!

Would you mind telling me how you found it?

In terms of your project goals - yes, this general RAG pipeline could be adopted to handle many different types of documents - and depending upon your requirements that may potentially involve:

• Some tweaks to the data processing Jupyter Notebook to ingest / chunk / upsert documents of that format
• Modifications to the prompt you send to the LLM along with the retrieved context / related items - mostly to specify if you have a specific output format you're looking for - there's also now zod and Structured Outputs from OpenAI which would likely be relevant as well.
• Modifications to the UI code to display your related products, if they have a significantly different format than the related posts example objects in the companion site for this tutorial

Feel free to send me more info about what you're trying to accomplish end to end and I'll see if there's any other helpful examples or info I can share.

[scoulter05]

Hey I'm going through the tutorial and I'm running into a runtime error

---> 11 vectorstore = PineconeVectorStore.from_documents(split_docs, embeddings, index_name=index_name)

RateLimitError: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}

I've checked my OpenAI and Pinecone accounts and I don't have any charges

I've tried importing time per AI advice but I haven't had any progress. If you could give me some tips or know how to fix this I'd greatly appreciate it!

[scoulter05]

I've resolved the issue, it was so silly. I needed money on my Open AI account.

[zackproser]

Nice, glad you got it sorted and thanks for confirming.