LLM and Agent Evaluations

[madhurprash]

Foundation models exhibit inherent stochasticity, meaning their outputs contain fundamental randomness even when provided with identical inputs and inference parameters across multiple inference runs. This non-deterministic behavior creates significant challenges when developing sophisticated generative AI systems and AI agents that rely on these models as their core reasoning engines.

The challenge becomes particularly acute when building AI agents that need to execute multi-step workflows, make consistent decisions, or maintain coherent behavior across extended interactions. Unlike traditional software systems that produce predictable outputs, AI agents powered by foundation models may exhibit variable responses to the same scenarios, potentially leading to inconsistent task execution, unpredictable decision-making patterns, and difficulty in debugging or validating agent behavior.

This stochastic nature conflicts with the expectations of reliability and reproducibility that are essential for deploying AI agents in production environments, where consistent performance and predictable outcomes are critical for user trust and system dependability. As we build increasingly complex agentic systems that chain multiple model calls, perform reasoning over time, or interact with external tools and APIs, managing and mitigating this inherent randomness becomes a fundamental engineering challenge.

This is a discussion pertained to a few components for evaluation of models and agents and a few frameworks that can help with the same. These are just a few examples as a starting point for evaluating various models and agents in the multi-agent architecture.

Model evaluations

1. Foundation model benchmarking tool (FMBench): https://github.com/aws-samples/foundation-model-benchmarking-tool

2. FMEval: https://github.com/aws/fmeval

3. OpenAI Evals: https://github.com/openai/evals

4. LangFuse for LLM evals: https://langfuse.com/docs/scores/overview

5. Arize AI evaluations: https://arize.com/docs/ax/concepts/how-does-evaluation-work

Agent evaluations

1. HAL - Holistic Agent Leaderboard: https://hal.cs.princeton.edu/

2. Agent evals: https://github.com/awslabs/agent-evaluation

3. Galileo: https://docs.galileo.ai/galileo/gen-ai-studio-products/galileo-evaluate/how-to/evaluate-and-optimize-agents--chains-or-multi-step-workflows

4. RAGAS: https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/agents/