融合实验前后数据，显著降低A/B测试方差，提升决策效率。
Variance reduction combining pre-experiment and in-experiment data
- 结合实验前与实验中数据，构建更稳健的方差缩减框架。
- 在Etsy实测中，仅用少量实验后变量即实现显著方差下降。
- 方法简单高效，适合大规模线上实验部署，尤其适合追求高敏感度的团队。
在线受控实验（A/B测试）是众多企业数据驱动决策的基础。在固定样本量下提升实验灵敏度，关键在于降低平均处理效应（ATE）估计器的方差。现有方差缩减技术如CUPED和CUPAC依赖实验前数据，其效果取决于这些数据对实验结果的预测能力。而实验中数据通常与结果相关性更强，但随意使用治疗后变量可能引入偏差。本文提出一种通用、鲁棒且可扩展的框架，结合预实验与实验中数据以实现方差缩减。该框架简洁、可解释且计算高效，适合实际部署。我们建立了所提估计器的渐近理论，并提供了一致的方差估计。Etsy的多个线上实验结果显示，即使仅引入少数治疗后协变量，该方法相比当前流程仍能实现显著额外的方差减少。这证明了本框架在提升实验灵敏度、加速数据驱动决策方面的有效性。
原文摘要 · Abstract (English)
Online controlled experiments (A/B testing) are fundamental to data-driven decision-making in many companies. Improving the sensitivity of these experiments under fixed sample size constraints requires reducing the variance of the average treatment effect (ATE) estimator. Existing variance reduction techniques such as CUPED and CUPAC use pre-experiment data, but their effectiveness depends on how predictive those data are for outcomes measured during the experiment. In-experiment data are often more strongly correlated with the outcome, but using arbitrary post-treatment variables can introduce bias. In this paper, we propose a general, robust, and scalable framework that combines both pre-experiment and in-experiment data to achieve variance reduction. Our framework is simple, interpretable, and computationally efficient, making it practical for real-world deployment. We develop the asymptotic theory of the proposed estimator and provide consistent variance estimators. Empirical results from multiple online experiments conducted at Etsy demonstrate substantial additional variance reduction over current pipeline, even when incorporating only a few post-treatment covariates. These findings underscore the effectiveness of our framework in improving experimental sensitivity and accelerating data-driven decision-making.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。