改进A/B测试统计效率，提升小样本与非正态分布下的实验效果
Beyond Basic A/B testing: Improving Statistical Efficiency for Business Growth
- 采用估计方程与U统计量提升小样本和非正态数据下的检验能力
- 提出双重稳健广义U统计量，统一处理样本小、分布偏、回报率等复杂问题
- 在领英真实实验中验证有效，适合数据质量差或资源受限的业务场景
主流工业界A/B测试多基于t检验，但在实际业务中常因样本量小、分布非正态或投资回报率（ROI）考量导致统计功效低下。本文（i）证明了估计方程与U统计量在分别应对上述问题上的统计效率优势；（ii）提出一种新型双重稳健广义U统计量，可在单一框架内灵活定义处理效应，同时处理小样本、分布鲁棒性、ROI及混杂因素。我们给出了渐近理论与效率界限结果，并通过理论分析揭示效率增益来源。进一步开展全面模拟研究，并在领英多个真实A/B测试中应用，分享具有广泛价值的结果与经验。
原文摘要 · Abstract (English)
The standard A/B testing approaches are mostly based on t-test in large scale industry applications. These standard approaches however suffers from low statistical power in business settings, due to nature of small sample-size or non-Gaussian distribution or return-on-investment (ROI) consideration. In this paper, we (i) show the statistical efficiency of using estimating equation and U statistics, which can address these issues separately; and (ii) propose a novel doubly robust generalized U that allows flexible definition of treatment effect, and can handles small samples, distribution robustness, ROI and confounding consideration in one framework. We provide theoretical results on asymptotics and efficiency bounds, together with insights on the efficiency gain from theoretical analysis. We further conduct comprehensive simulation studies, apply the methods to multiple real A/B tests at LinkedIn, and share results and learnings that are broadly useful.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。