{"product_id":"9798868827204","title":"Engineering Online Experimentation and ML Evaluations Architecture, Statistics and Machine Learning for Production-Scale Systems","description":"\u003ch1\u003eEngineering Online Experimentation and ML Evaluations\u003c\/h1\u003e\u003ch2\u003eArchitecture, Statistics and Machine Learning for Production-Scale Systems\u003c\/h2\u003e\u003ch3\u003eMing Lei\u003c\/h3\u003e\u003cdiv\u003e\u003cb\u003eComputers \/ Artificial Intelligence \/ General\u003c\/b\u003e\u003c\/div\u003e\u003cbr\u003e\u003cdiv\u003e\n\u003cp class=\"MsoNormal\"\u003e\u003cspan style=\"font-family: 'Times New Roman',serif;\"\u003eOnline experimentation is now essential for modern software and machine learning teams. This book provides an engineer-first, end-to-end guide to building and operating production-ready experimentation platforms.\u003c\/span\u003e\u003c\/p\u003e\r\n\u003cp class=\"MsoNormal\"\u003e\u003cspan style=\"font-family: 'Times New Roman',serif;\"\u003eThe book begins with Part I establishing the core foundations of credible experimentation, including hypothesis testing, power analysis, sample sizing, metric design, and common pitfalls such as peeking, multiple testing, and novelty or learning effects. Part II focuses on platform engineering—traffic and identity management, mutual exclusion, event and logging design, ETL\/ELT pipelines, building a stats engine with SciPy and statsmodels, SRM detection, integrating deployments with feature flags and canaries, and setting up guardrail and health monitoring. Part III presents advanced designs that improve speed and sensitivity: sequential testing with alpha spending, bootstrap intervals for ratios and quantiles, A\/B\/n testing with ANOVA, interleaving for ranking systems, switchback and geo experiments, and multi-armed bandits. Part IV connects experimentation to ML workflows, covering offline, shadow, canary, and A\/B evaluation pipelines; Bayesian optimization for adaptive experimentation; counterfactual and IPS methods for learning from logs; and safe retraining supported by strong governance.\u003c\/span\u003e\u003c\/p\u003e\r\n\u003cp class=\"MsoNormal\"\u003e\u003cspan style=\"font-family: 'Times New Roman',serif;\"\u003eWhat you will learn:\u003c\/span\u003e\u003c\/p\u003e\r\n\u003cul style=\"margin-top: 0cm;\" type=\"disc\"\u003e\r\n\u003cli class=\"MsoNormal\" style=\"mso-list: l0 level1 lfo1; tab-stops: list 36.0pt;\"\u003e\u003cspan style=\"font-family: 'Times New Roman',serif;\"\u003eDesign trustworthy experiments with proper metrics, guardrails, α\/power\/MDE settings, and safeguards against peeking and multiple-testing errors\u003c\/span\u003e\u003c\/li\u003e\r\n\u003cli class=\"MsoNormal\" style=\"mso-list: l0 level1 lfo1; tab-stops: list 36.0pt;\"\u003e\u003cspan style=\"font-family: 'Times New Roman',serif;\"\u003eBuild a production-ready experimentation stack with assignment, identity\/diversion, logging, ETL\/ELT, a stats engine, and SRM checks\u003c\/span\u003e\u003c\/li\u003e\r\n\u003cli class=\"MsoNormal\" style=\"mso-list: l0 level1 lfo1; tab-stops: list 36.0pt;\"\u003e\u003cspan style=\"font-family: 'Times New Roman',serif;\"\u003eRun advanced designs at scale, including sequential tests, bootstrap CIs, interleaving, switchback\/geo experiments, and multi-armed bandits\u003c\/span\u003e\u003c\/li\u003e\r\n\u003cli class=\"MsoNormal\" style=\"mso-list: l0 level1 lfo1; tab-stops: list 36.0pt;\"\u003e\u003cspan style=\"font-family: 'Times New Roman',serif;\"\u003eEvaluate ML systems from offline to online, leverage experiment logs for learning, and enable safe retraining with governance\u003c\/span\u003e\u003c\/li\u003e\r\n\u003c\/ul\u003e\r\n\u003cp\u003e\u003cspan style=\"font-family: 'Times New Roman',serif;\"\u003eWho this book is for:\u003c\/span\u003e\u003c\/p\u003e\r\n\u003cp\u003e\u003cspan style=\"font-family: 'Times New Roman',serif;\"\u003eThe primary audience for this book includes Data Engineers, ML Engineers, and Platform or Software Architects. It is also well suited for Product and Data Scientists who want a deeper understanding of experimentation systems and the engineering principles behind them.\u003c\/span\u003e\u003c\/p\u003e\n\u003c\/div\u003e\u003cdiv\u003e\u003cp\u003eMing Lei is a data and ML engineering leader with 20 years of experience building end-to-end ML systems for Internet Ads and Search, and experimentation platforms for E-commerce. He has designed large-scale systems that operationalize rigorous statistical methods — such as sequential testing, bootstrapping, multi-armed bandits, and Bayesian optimization — and support ML evaluation from offline analysis to online deployment. His leadership spans roles at eBay, Meta (Facebook), Google, and Appen. He holds multiple US patents and advanced degrees in computer science (UC Riverside) and economics (Clark University), along with a B.S. in physics (Wuhan University). He is based in the Northwest of US.\u003c\/p\u003e\u003c\/div\u003e\u003cbr\u003e\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd\u003ePublication Date: \u003c\/td\u003e\n\u003ctd\u003e20 July 2026\u003c\/td\u003e\n\u003c\/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003ePublisher: \u003c\/td\u003e\n\u003ctd\u003eApress\u003c\/td\u003e\n\u003c\/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eImprint: \u003c\/td\u003e\n\u003ctd\u003eApress\u003c\/td\u003e\n\u003c\/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eISBN-13: \u003c\/td\u003e\n\u003ctd\u003e9798868827204\u003c\/td\u003e\n\u003c\/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eFormat: \u003c\/td\u003e\n\u003ctd\u003ePaperback \/ softback\u003c\/td\u003e\n\u003c\/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003ePage Count: \u003c\/td\u003e\n\u003ctd\u003e540\u003c\/td\u003e\n\u003c\/tr\u003e\n\u003c\/table\u003e","brand":"Apress","offers":[{"title":"Default Title","offer_id":46630689308812,"sku":"9798868827204","price":53.99,"currency_code":"USD","in_stock":true}],"thumbnail_url":"\/\/cdn.shopify.com\/s\/files\/1\/0710\/9545\/1788\/files\/9798868827204.jpg?v=1780604845","url":"https:\/\/fh90cf-fv.myshopify.com\/products\/9798868827204","provider":"Late Knight Books and Services, LLC","version":"1.0","type":"link"}