Seeking Advice: Building a Cost-Efficient Discord Bot with Answer Certainty Evaluation

[holabuenastardes6854]

Hi everyone! I'm building a Discord bot using LightRAG that aims to act like a natural server member - automatically identifying when users ask questions and responding based on the knowledge base.

Current Progress

I've implemented Phase 1: A local HuggingFace question detector that filters out ~70% of non-questions at zero cost.

Now I'm working on Phase 2: an Answer Certainty Evaluator that assesses whether the available knowledge can satisfactorily answer a detected question BEFORE processing it with LightRAG. The goal is to only respond when the bot can provide genuinely helpful answers.

My Approach

My current approach involves evaluating:

• Entity coverage: Relevant entities available for the question's keywords
• Relation coverage: Connections between entities that form answer pathways
• Text coverage: Relevant text chunks that directly address the question
• Context coherence: Whether all knowledge pieces form a complete, useful context

I'm thinking of reusing LightRAG's existing keyword extraction logic and querying entities_vdb, relationships_vdb, and chunks_vdb to calculate a combined certainty score.

The Real Goal

The whole point of this detector system is making the bot cost efficient - I don't want to send every single user's message to OpenAI's API when the knowledge base can't provide a satisfactory answer anyway.

My Question

Is this the right architectural approach, or would you recommend a different strategy?

I'm particularly interested in:

• If there are more efficient ways to assess knowledge base "completeness" for a given question
• Performance considerations when querying multiple vector stores
• Any alternative approaches the community has tried?

Technical Details

I've attached my complete planning document below that goes into much more depth about the implementation approach, code examples, and the three-phase roadmap I'm following.

Would really appreciate your thoughts on the best way to implement this efficiently within LightRAG!
plan.md