Sample Ratio Mismatch (SRM) Calculator
Calculator Configuration
Expected split: 50% / 50%
Results
With 1 degree of freedom: p = 0.00058
⚠ Sample ratio mismatch detected
Your split is broken. Something in assignment, logging, or filtering is treating the two arms differently. Don’t trust the experiment results until you find and fix the cause.
A gap this size happens by chance about 1 in 1,721 experiments.
Why severity doesn't matter: the p-value proves the break is real, but the size of the bias depends on the cause. So no confirmed SRM is small enough to ignore.
Confirmed a mismatch? Sample Ratio Mismatch (SRM): what it means and how to debug it
How it works
What is this?
A sample ratio mismatch (SRM) is when the number of users actually observed in each arm of your experiment doesn't match the split you configured. If you asked for 50/50 and got 5,000 vs 5,350, that gap is either random noise or a broken assignment/logging pipeline. This calculator tells you which.
Why it matters
An SRM means users were added to (or dropped from) one arm in a way that isn't random. Maybe a redirect loses slow connections, a bot filter fires unevenly, or a logging bug drops events. Whatever the cause, it almost never affects both arms equally, so every metric you compute downstream is biased. The experiment results can't be trusted until the cause is found and fixed.
For the full story on what causes SRMs, real examples, and a step-by-step debugging checklist, read our guide: Sample Ratio Mismatch (SRM): what it means and how to debug it.
The test
We run a chi-squared goodness-of-fit test comparing your observed counts against the counts your configured split would produce:
Where:
- and are the observed user counts in each arm
- and are the expected counts under your configured split
With two arms there is 1 degree of freedom, and the p-value is the probability of seeing a gap at least this large when the split is actually healthy. Following common SRM practice, we flag p < 0.001 as a confirmed mismatch and p < 0.01 as suspicious.
Calculation with Your Values
Given:
- Expected split: 50% / 50%
- Observed: 5,000 users in A, 5,350 users in B (10,350 total)
Step 1: Expected Counts
E_A = 10,350 × 0.5000 = 5175
E_B = 10,350 × 0.5000 = 5175
Step 2: Chi-Squared Statistic
χ² = (5000 − 5175)² / 5175 + (5350 − 5175)² / 5175
χ² = 11.8357
Step 3: P-Value
p = P(χ²₁ ≥ 11.8357) = 0.00058
Step 4: Compare to the SRM Thresholds
The thresholds we use:
- p < 0.001: confirmed SRM
- p < 0.01: suspicious
- p ≥ 0.01: healthy split
p = 0.00058, which is < 0.001, so this is a confirmed sample ratio mismatch
Step 5: The “1 in X Experiments” Number
The p-value is the chance a healthy split shows a gap this big, so on average it takes 1 / p healthy experiments to see one:
X = 1 / p = 1 / 0.00058 ≈ 1,721
A gap this size happens by chance about 1 in 1,721 experiments.
Conclusion:
Sample ratio mismatch detected. A gap this size happens by chance about 1 in 1,721 experiments.
Work with us
Found an SRM you can't explain?
Book a free 15-minute call. Bring a test you're running or planning. We'll look at the design, the instrumentation, or the results together.
Book a 15-minute call