View source for Sample ratio mismatch

{{Short description|Statistical error}}

{{Orphan|date=May 2023}}
In the [[design of experiments]], a '''sample ratio mismatch''' ('''SRM''') is a [[statistically significant]] difference between the expected and actual ratios of the sizes of [[treatment and control groups]] in an experiment. Sample ratio mismatches also known as unbalanced sampling<ref>{{Cite book |last1=Esteller-Cucala |first1=Maria |last2=Fernandez |first2=Vicenc |last3=Villuendas |first3=Diego |title=Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization |chapter=Experimentation Pitfalls to Avoid in A/B Testing for Online Personalization |date=2019-06-06 |chapter-url=https://dl.acm.org/doi/10.1145/3314183.3323853 |language=en |publisher=ACM |pages=153–159 |doi=10.1145/3314183.3323853 |isbn=978-1-4503-6711-0|s2cid=190007129 }}</ref> often occur in [[Web-based experiments|online]] [[controlled experiment]]s due to failures in [[randomization]] and [[telemetry|instrumentation]].<ref>{{Cite book |last1=Fabijan |first1=Aleksander |last2=Gupchup |first2=Jayant |last3=Gupta |first3=Somit |last4=Omhover |first4=Jeff |last5=Qin |first5=Wen |last6=Vermeer |first6=Lukas |last7=Dmitriev |first7=Pavel |title=Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining |chapter=Diagnosing Sample Ratio Mismatch in Online Controlled Experiments |date=2019-07-25 |chapter-url=https://dl.acm.org/doi/10.1145/3292500.3330722 |language=en |publisher=ACM |pages=2156–2164 |doi=10.1145/3292500.3330722 |isbn=978-1-4503-6201-6|s2cid=196199621 }}</ref>

Sample ratio mismatches can be detected using a [[chi-squared test]].<ref>{{Cite book |last1=Nie |first1=Keyu |last2=Zhang |first2=Zezhong |last3=Xu |first3=Bingquan |last4=Yuan |first4=Tao |title=Proceedings of the 31st ACM International Conference on Information & Knowledge Management |chapter=Ensure A/B Test Quality at Scale with Automated Randomization Validation and Sample Ratio Mismatch Detection |date=2022-10-17 |chapter-url=https://dl.acm.org/doi/10.1145/3511808.3557087 |language=en |publisher=ACM |pages=3391–3399 |doi=10.1145/3511808.3557087 |arxiv=2208.07766 |isbn=978-1-4503-9236-5|s2cid=251594683 }}</ref> Using methods to detect SRM can help non-experts avoid making discussions using biased data.<ref>{{Cite book |last1=Vermeer |first1=Lukas |last2=Anderson |first2=Kevin |last3=Acebal |first3=Mauricio |title=The International Conference on Evaluation and Assessment in Software Engineering 2022 |chapter=Automated Sample Ratio Mismatch (SRM) detection and analysis |date=2022-06-13 |chapter-url=https://dl.acm.org/doi/10.1145/3530019.3534982 |language=en |publisher=ACM |pages=268–269 |doi=10.1145/3530019.3534982 |isbn=978-1-4503-9613-4|s2cid=249579055 |url=http://resolver.tudelft.nl/uuid:31d2806f-2bf3-43c3-9e71-3985fd7368a6 }}</ref> If the sample size is large enough, even a small discrepancy between the observed and expected group sizes can invalidate the results of an experiment.<ref name="KDD19">{{cite book|chapter-url=https://exp-platform.com/Documents/2019_KDDFabijanGupchupFuptaOmhoverVermeerDmitriev.pdf|last1=Fabijan|first1=Aleksander|last2=Gupchup|first2=Jayant|last3=Gupta|first3=Somit|last4=Omhover|first4=Jeff|last5=Qin|first5=Wen|last6=Vermeer|first6=Lukas|last7=Dmitriev|first7=Pavel|title=Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining |chapter=Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners |date=2019 |pages=2156–2164 |doi=10.1145/3292500.3330722|isbn=9781450362016 |s2cid=196199621 }}</ref><ref>{{Cite news |last1=Kohavi |first1=Ron |last2=Thomke |first2=Stefan |date=2017-09-01 |title=The Surprising Power of Online Experiments |work=Harvard Business Review |url=https://hbr.org/2017/09/the-surprising-power-of-online-experiments |access-date=2023-05-19 |issn=0017-8012}}</ref>

== Example ==

Suppose we run an [[A/B test]] in which we randomly assign 1000 users to equally sized treatment and control groups (a 50–50 split). The expected size of each group is 500. However, the actual sizes of the treatment and control groups are 600 and 400.

Using [[Pearson's chi-squared test|Pearson's chi-squared]] [[goodness of fit]] test, we find a sample ratio mismatch with a ''p''-value of 2.54{{e-sp|-10}}. In other words, if the assignment of users were truly random, the probability that these treatment and control group sizes would occur by chance is 2.54{{e-sp|-10}}.<ref name="srm-checker">{{cite web|url=https://www.lukasvermeer.nl/srm/docs/faq/|title=Frequently Asked Questions|website=SRM Checker|last=Vermeer|first=Lukas|access-date=2022-09-15}}</ref>

== References ==

{{reflist}}

[[Category:Design of experiments]]
[[Category:Research]]


{{Statistics-stub}}