// The Blog

Ecommerce A/B testing, explained honestly.

Most testing advice is written by people selling you a testing tool. This isn't. Practical, no-spin guides to running experiments that hold up: how much traffic you actually need, why winners shrink after launch, and how to tell a real result from a lucky one. Every claim backed by the math it comes from.

Sample size Winner's curse Platform trust Program measurement
Latest
New Chart of how many weeks an A/B test takes to reach significance at 10,000 visitors a month
Jul 5, 2026Sample size7 min read

How much traffic do you need to A/B test an online store?

"My store does 10,000 visitors a month and my tests never reach significance. Am I doing something wrong?" No. At that traffic, detecting a 10% lift takes 56 weeks. Here's the arithmetic to run before any test, and what actually works at small-store traffic.

Read the guide →
What this blog covers
Sample size & power
Will this test ever conclude?
How to size a test before you run it, why underpowered tests waste months, and the traffic math nobody prints on the box.
→ Try Lockbox
Winner's curse
Why winners shrink after launch
Significant tests systematically overstate their effect. How to deflate a result to its honest size before you announce it.
→ Try Reality Check
Platform trust & SRM
Is your platform telling the truth?
Sample ratio mismatch, tracking loss, and the A/A test that catches a broken tool before it biases every result you read.
→ Try the Platform Validator
Program measurement
Did the wins reach revenue?
Why announced lift and realized lift diverge, and how to reconcile a year of "winning" tests against your actual numbers.
→ Try the Program Ledger