AI Model Evaluation

De-risk AI models, validate real-world performance, and align output with product goals. Before you trust critical business systems to an AI model, you need to answer a few questions. Will it be fast enough? Will the system satisfy user expectations? Is it safe? Can you trust the output? This book will help you answer these questions and more before you roll out an AI system—and make sure it runs smoothly after you deploy. In AI Model Evaluation you’ll learn how to: • Build diagnostic offline evaluations that uncover model behavior • Use shadow traffic to simulate production conditions • Design A/B tests that validate model impact on key product metrics • Spot nuanced failures with human-in-the-loop feedback • Use LLMs as automated judges to scale your evaluation pipeline In AI Model Evaluation author Leemay Nassery shares her hard-won experiences specializing in experimentation and personalization across companies such as Spotify, Comcast, Dropbox, and Etsy. The book is packed with insights on what it really takes to get a model ready for production. You’ll go beyond basic performance evaluations to discover how you can measure model effectiveness on the product, spot latency issues as you introduce the model in your end-to-end architecture, and understand the model’s real-world impact. About the book AI Model Evaluation teaches you how to effectively evaluate and assess machine learning models for better scaling and integration into production systems. Each chapter tackles a different evaluation method. You'll start with offline evaluations, then move into live A/B tests, shadow traffic deployments, qualitative evaluations, and LLM-based feedback loops. You’ll learn how to evaluate both model behavior and engineering system performance, with a hands-on example grounded in a movie recommendation engine. About the reader For practitioners with experience in machine learning, data science, or software engineering. Familiarity with Python is recommended. About the author Leemay Nassery is an engineering leader specializing in experimentation and personalization. With a notable track record that includes evolving Spotify's A/B testing strategy for the Homepage, launching Comcast's For You page, and establishing data warehousing teams at Etsy, she firmly believes that the key to innovation at any company is the ability to experiment effectively.