Beyond Accuracy: Evaluating Agentic AI Systems in the Real World
As organizations increasingly deploy agentic AI systems capable of reasoning, planning, and taking actions, evaluating success becomes significantly more complex. Traditional AI metrics such as accuracy, precision, and recall are often insufficient for determining whether an agent is effectively helping users accomplish their goals.
This session explores practical approaches for evaluating agentic AI systems beyond model performance. We will discuss how organizations can measure task completion, decision quality, conversational effectiveness, reliability, adaptability, and user satisfaction. The talk will also examine common evaluation challenges, including multi-step workflows, tool usage, autonomous decision making, and complex customer interactions.
Attendees will learn a practical framework for designing evaluation strategies that align technical performance with real-world outcomes. The session will provide examples of how evaluation approaches must evolve as organizations move from predictive AI models to agentic systems that actively participate in customer journeys and business processes.
About the speaker
Snehal Bhole
Sr Data Science Manager – AI, Recommendation & Personalization at Expedia
Snehal Bhole is a Senior Data Science Manager and analytics leader with over a decade of experience applying artificial intelligence, machine learning, and data-driven decision making across digital products and customer-facing platforms. Her work focuses on helping organizations build intelligent systems that improve customer experiences and create long-term value. Snehal’s current areas of interest include agentic AI, conversational experiences, personalization, behavioral intelligence, and long-term value optimization. She is particularly interested in how AI agents and conversational interfaces can help users navigate complex decisions, discover relevant information, and achieve their goals more effectively. Her work explores the intersection of agentic systems, customer behavior, and intelligent decision frameworks. In addition to her professional work, Snehal actively mentors aspiring data scientists and analytics professionals and enjoys sharing insights on emerging trends in AI, analytics, and intelligent customer experiences. She holds a Master of Science in Business Analytics from the University of Washington and an MBA from Jamnalal Bajaj Institute of Management Studies (JBIMS), India.