arXiv:2606.18750stat.APcs.LG2026-06

解决CUPED在在线实验中的五个关键问题,提升结果可信度。

Ensuring Trustworthy Online A/B Testing: Addressing Five Key Questions on CUPED

论文配图:Ensuring Trustworthy Online A/B Testing: Addressing Five Key Questions on CUPED
图 1 · 摘自论文原文
  • 对比多种后处理估计器,找出最优调整方式。
  • 发现标准方差估计在复杂场景下会误导结论。
  • 方法已落地字节跳动平台,适合实验设计者参考。

A/B测试已成为大型在线实验中数据驱动决策的黄金标准,为功能发布、定价优化和用户体验改进提供关键指导。为最大化统计敏感性,众多科技公司普遍采用利用预实验数据的控制实验(CUPED),该方法可显著降低方差且保持平均处理效应估计的无偏性。尽管广泛应用,CUPED的若干方法论与实际应用细节仍被忽视。本文系统回答了五个常见但被忽略的问题:首先,比较多种后CUPED估计器,识别最优调整设定;其次,评估基于回归的调整有效性,并提出适用于此类框架的稳健方差估计方法;最后,将研究扩展至多臂实验和两阶段抽样等常见复杂场景。研究发现,在这些情况下,依赖标准方差估计可能导致严重误判。通过严格的理论分析与广泛的实验验证,本工作深化了对CUPED的理解。所推荐的方法已成功部署并集成至字节跳动的实验平台。

原文摘要 · Abstract (English)

A/B testing has become the gold standard for data-driven decision-making in large-scale online experimentation, providing critical guidance for feature launch, pricing optimization, and user experience enhancement. To maximize statistical sensitivity, many technology companies routinely employ Controlled-experiment Using Pre-Experiment Data (CUPED), a technique that achieves substantial variance reduction while preserving the unbiasedness of estimating the average treatment effect. Despite its widespread adoption, several critical methodological and practical nuances of CUPED remain underexplored. This paper systematically addresses five frequently encountered yet overlooked questions regarding the application of CUPED. First, we provide a comparative analysis of various post-CUPED estimators to identify the optimal adjustment specification. Second, we evaluate the validity of regression-based adjustments and delineate robust variance estimation methods tailored for such frameworks. Finally, we extend our investigation to complex but common scenarios, including multi-arm experiments and two-stage sampling designs. Our findings reveal that in these settings, naive reliance on standard variance estimators can lead to severely misleading inferences. By offering rigorous theoretical insights and extensive experimental validation, this work deepens the conceptual understanding of CUPED. Notably, the recommended methodologies have been successfully deployed and integrated into ByteDance's experimentation platform.

A/B测试CUPED实验设计方差降低

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。