A/B Test Designer
The analytical round in product, APM, BizOps and data interviews nearly always lands here. The failure mode is rarely arithmetic: most candidates can compute a p-value and cannot say what would change their mind. Size a test, watch the sample explode as the effect you are chasing shrinks, then read a result and say honestly what it entitles you to claim.
Includes the question everyone gets asked — a 2% lift at p = 0.06, what do you do.
Design the test
Why the sample explodes
Sample size scales with the inverse square of the effect. Halve the lift you want to detect and you need roughly four times the users. This is the single most useful thing to know in the room, because it turns “let’s just test it” into a costed decision.
| Detect | Per arm | Days | vs current |
|---|---|---|---|
| +2% | 950,884 | 159 | 24.1× |
| +3% | 424,617 | 71 | 10.8× |
| +5% | 154,302 | 26 | 3.9× |
| +7.5% | 69,377 | 12 | 1.8× |
| +10% | 39,473 | 7 | 1.0× |
| +15% | 17,940 | 3 | 0.5× |
| +20% | 10,314 | 2 | 0.3× |
| +30% | 4,780 | 1 | 0.1× |
Read a result
Four ways a test lies to you
Peeking
Checking daily and stopping when it crosses 5% turns a 5% false-positive rate into something closer to 30%. Fix the sample size before you start, or use a sequential test designed for it.
Multiple comparisons
Twenty metrics at 5% means one lights up by chance. Name your primary metric before the test runs; everything else is a guardrail or a hypothesis for next time.
Novelty and primacy
Users react to change itself. Early lifts fade; early drops recover. A week is usually the minimum, and a returning-user cut tells you which one you are seeing.
Sample ratio mismatch
If the split is meant to be 50/50 and it is 51/49 at scale, something is broken in the assignment and the result cannot be trusted. Check it first, every time.
Share your result 🧪
Instagram and TikTok have no desktop share link, so copy the caption and paste it into the app.