Data Science Ninja – Course 2: Experimentation (AB Test)

Data Science Ninja – Course 2: Experimentation (AB-Test)

Difficulty
Intermediate to advanced

About the course
We start from wider range of causal inference and narrow down to designed experiments and the details of running AB test in product development cycle. We cover topics in classic AB test such as doing the right math and telling the right story. We then solve into common mistakes and solutions related to multiple testing, tiered metrics, assumption checking, violations and remedies. In advanced topics, we cover sequential test, MAB, Network and multi-sided marketplace and how to speed up experimentation. In addition, we share knowledge, solution and experience in industry scale experimentation platform from several leading tech companies.

By working through examples with increasing levels of complexity, students will walk away with practical knowledge on how to design, setup, measure an AB test to evaluate product changes, and recommend actions based findings.

Syllabus

Week 1: Overview, AB Testing Setup, Randomization
Week 2: Common Pitfalls and Making Reliable Comparisons
Week 3: Quality Control & Advanced Solutions
Week 4: The Cutting Edge of Experimentation

Prerequisites
Graduate level statistics and probability, statistical inference