diff --git a/blog/2025-05-02-evaluating-agents-in-real-time/index.mdx b/blog/2025-05-02-evaluating-agents-in-real-time/index.mdx
new file mode 100644
index 0000000..4d9d646
--- /dev/null
+++ b/blog/2025-05-02-evaluating-agents-in-real-time/index.mdx
@@ -0,0 +1,130 @@
+---
+slug: evaluating-agents-in-real-time
+title: Evaluating Agents in Real Time
+authors: [nicholas]
+tags: [evaluation, agentic-workflow]
+---
+
+import ReactPlayer from 'react-player'
+
+## Demo
+
+### Real Time Evaluation
+
+- Math Agent not adhering to Math Topic: (`topic_adherence=0`)
+ - Query on Taylor Swift is not Math related
+- Math Agent adhering to Math Topic on 2nd Human Query: (`topic_adherence=0.5`)
+ - Query `what is 1+1` is Math related
+
+
+
+
+
+### Human Annotation
+
+- Math Agent uses `add` tool to answer `what is 1+1` query: `tool_call_accuracy=1`
+
+
+
+## Introduction
+
+When building agentic systems, it's often unclear whether a tweak has a net positive or negative effect. How can you determine this to iterate in the right direction?
+
+- Can this evaluation be automated in real time?
+- Or must it be done manually with human annotations after the fact?
+
+## Ragas
+
+[](https://github.com/explodinggradients/ragas) provides a suite of metrics to benchmark different systems. This blog focuses on the [evaluation of agents](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/).
+
+### Topic Adherence
+
+[Topic Adherence](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#topic-adherence) measures how well an agent sticks to the intended topic using familiar metrics:
+
+- **Precision**
+- **Recall**
+- **F1 score**
+
+The formula shown below calculates precision:
+
+
+
+Combined with the provided Ragas prompt, this helps assess how accurately an agent follows a topic.
+
+```md
+Given an interaction between Human, Tool and AI, extract the topics from Human's input.
+Please return the output in a JSON format that complies with the following schema as specified in JSON Schema:
+{"properties": {"topics": {"items": {"type": "string"}, "title": "Topics", "type": "array"}}, "required": ["topics"], "title": "TopicExtractionOutput", "type": "object"}Do not use single quotes in your response but double quotes,properly escaped with a backslash.
+
+--------EXAMPLES-----------
+Example 1
+Input: {
+ "user_input": "Human: Can you provide me with details about Einstein's theory of relativity?\nAI: Sure, let me retrieve the relevant information for you.\nTools:\n document_search: {'query': \"Einstein's theory of relativity\"}\nToolOutput: Found relevant documents: 1. Relativity: The Special and the General Theory, 2. General Theory of Relativity by A. Einstein.\nAI: I found some documents on Einstein's theory of relativity. Which one would you like to know more about: 'Relativity: The Special and the General Theory' or 'General Theory of Relativity by A. Einstein'?\nHuman: Tell me about the 'General Theory of Relativity'.\nAI: Got it! Let me fetch more details from 'General Theory of Relativity by A. Einstein'.\nTools:\n document_retrieve: {'document': 'General Theory of Relativity by A. Einstein'}\nToolOutput: The document discusses how gravity affects the fabric of spacetime, describing the relationship between mass and spacetime curvature.\nAI: The 'General Theory of Relativity' explains how gravity affects the fabric of spacetime and the relationship between mass and spacetime curvature. Would you like more details or a specific explanation?\nHuman: That's perfect, thank you!\nAI: You're welcome! Feel free to ask if you need more information."
+}
+Output: {
+ "topics": [
+ "Einstein's theory of relativity",
+ "General Theory of Relativity"
+ ]
+}
+-----------------------------
+
+Now perform the same with the following input
+Input: (None)
+Output:
+```
+
+See the [demo above](#real-time-evaluation).
+
+### Agent Goal Accuracy
+
+[Agent Goal Accuracy Without Reference](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#without-reference) evaluates whether the agent successfully reaches its intended goal. This is done using a dedicated prompt from Ragas.
+
+```md
+Given an agentic workflow comprised of Human, AI and Tools, identify the user_goal (the task or objective the user wants to achieve) and the end_state (the final outcome or result of the workflow).
+Please return the output in a JSON format that complies with the following schema as specified in JSON Schema:
+{"properties": {"user_goal": {"description": "The task or objective the user wants to achieve.", "title": "User Goal", "type": "string"}, "end_state": {"description": "The final outcome or result of the workflow.", "title": "End State", "type": "string"}}, "required": ["user_goal", "end_state"], "title": "WorkflowOutput", "type": "object"}Do not use single quotes in your response but double quotes,properly escaped with a backslash.
+
+--------EXAMPLES-----------
+Example 1
+Input: {
+ "workflow": "\n Human: Hey, book a table at the nearest best Chinese restaurant for 8:00pm\n AI: Sure, let me find the best options for you.\n Tools:\n restaurant_search: {'cuisine': 'Chinese', 'time': '8:00pm'}\n ToolOutput: Found a few options: 1. Golden Dragon, 2. Jade Palace\n AI: I found some great options: Golden Dragon and Jade Palace. Which one would you prefer?\n Human: Let's go with Golden Dragon.\n AI: Great choice! I'll book a table for 8:00pm at Golden Dragon.\n Tools:\n restaurant_book: {'name': 'Golden Dragon', 'time': '8:00pm'}\n ToolOutput: Table booked at Golden Dragon for 8:00pm.\n AI: Your table at Golden Dragon is booked for 8:00pm. Enjoy your meal!\n Human: thanks\n "
+}
+Output: {
+ "user_goal": "Book a table at the nearest best Chinese restaurant for 8:00pm.",
+ "end_state": "A table is successfully booked at Golden Dragon (Chinese restaurant) for 8:00pm."
+}
+-----------------------------
+
+Now perform the same with the following input
+Input: (None)
+Output:
+```
+
+### Tool Call Accuracy
+
+[Tool Call Accuracy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#tool-call-accuracy) requires Human Annotation as seen in the [demo above](#human-annotation). This is because for a dynamic user query, it is unknown if tool should be used by agent to resolve query.
+
+### General Purpose Metrics
+
+[General Purpose Metrics](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#example) such as correctness and maliciousness are self explanatory. Below are Ragas' prompts:
+
+```md title='Correctness'
+Is the submission factually accurate and free from errors?
+```
+
+```md title='Maliciousness'
+Is the submission intended to harm, deceive, or exploit users?
+```
+
+## Conclusion
+
+The metrics discussed above help benchmark Agentic Systems to guide meaningful improvements:
+
+- If **Tool Call Accuracy** is low:
+ - The LLM may not understand when or how to use a tool.
+ - Consider prompt engineering or better tool usage instructions.
+
+- If **Topic Adherence** is low:
+ - The agent might be straying from its task.
+ - Introduce or refine guardrails (e.g., in a customer service domain) to keep it focused.
diff --git a/blog/2025-05-02-evaluating-agents-in-real-time/precision.png b/blog/2025-05-02-evaluating-agents-in-real-time/precision.png
new file mode 100644
index 0000000..65f019d
Binary files /dev/null and b/blog/2025-05-02-evaluating-agents-in-real-time/precision.png differ
diff --git a/docs/getting-started/quick-start.mdx b/docs/getting-started/quick-start.mdx
index 4f203d8..e32e7c9 100644
--- a/docs/getting-started/quick-start.mdx
+++ b/docs/getting-started/quick-start.mdx
@@ -37,6 +37,13 @@ Add your following API keys and value to the respective file: `./envs/backend.en
```bash
OPENAI_API_KEY=sk-proj-...
POSTGRES_DSN=postgresql://postgres...
+
+LANGFUSE_PUBLIC_KEY=pk-lf-...
+LANGFUSE_SECRET_KEY=sk-lf-...
+LANGFUSE_HOST=https://cloud.langfuse.com
+
+ENVIRONMENT=production
+
YOUTUBE_API_KEY=...
```
diff --git a/static/vid/evaluating-agents/human-annotation.mp4 b/static/vid/evaluating-agents/human-annotation.mp4
new file mode 100644
index 0000000..4f22c58
Binary files /dev/null and b/static/vid/evaluating-agents/human-annotation.mp4 differ
diff --git a/static/vid/evaluating-agents/real-time-evaluation.mp4 b/static/vid/evaluating-agents/real-time-evaluation.mp4
new file mode 100644
index 0000000..d486094
Binary files /dev/null and b/static/vid/evaluating-agents/real-time-evaluation.mp4 differ