This talk presents a robust framework for evaluating the performance of Large Language Model (LLM)-powered conversational agents by analyzing and replaying historical interaction data. The methodology involves capturing real-world user-agent conversation traces, transforming them into a standardized format, and systematically re-executing key LLM-driven decision steps—specifically topic selection and tool invocation—with different models. Performance is assessed through a suite of granular metrics, including Topic Agreement with the original interaction, Function Name Matching, and Function Schema Conformance. To evaluate the semantic quality of generated outputs, the framework employs an LLM-as-a-judge to score the contextual accuracy of function arguments and the relevance of free-text responses against the original outcomes. We provide a comprehensive comparative analysis of multiple leading LLMs (including models from OpenAI, Anthropic, and Google) across several specialized agent roles. The results demonstrate the framework’s effectiveness in providing quantitative benchmarks for regression testing, characterizing model-specific behaviors, and enabling data-driven decision-making for agent development and deployment.