arXiv:2607.01958cs.LG2026-07KDD被引 1

用共享数据构造虚拟中间算法,让A/B测试更准。

A More Accurate Algorithm Comparison through A/B Testing using Offline Evaluation Methods

论文配图:A More Accurate Algorithm Comparison through A/B Testing using Offline Evaluation Methods
图 1 · 摘自论文原文
  • 引入虚拟中间算法,通过分步共享数据提升评估相关性
  • 实测仅需一半数据量即达现有方法误差水平
  • 适合在线服务中需高精度算法选型的场景

A/B测试是线上服务选择最优算法的金标准。尽管离线评估因实验成本高和潜在用户体验与收入风险而被视作更安全的替代方案,但普遍认为其估计精度远低于A/B测试。与此常规认知相反,我们揭示了一个反直觉现象:A/B测试的算法选择错误率可能高于离线评估。这是因为A/B测试使用的样本均值估计器不产生正相关性,而正相关性对减少关键选择错误(低估真正优算法、高估真正劣算法)至关重要。相反,离线评估因依赖共享离线数据,在比较多个算法性能时无意中生成了有益的相关性。基于此洞察,我们提出一种有意诱导正相关性的估计器:引入一个假设的中间算法,分步利用共享数据依次估计算法A、M、B之间的性能差异。该方法使每一步均可应用离线评估技术,从而实现正相关性并降低关键选择错误。此外,我们推导出最小方差对应的最优中间算法,并通过偏差-方差分析证明其优于现有方法。真实数据实验表明,本方法在仅使用一半传统A/B测试数据的情况下,即可达到相同的选择错误率。

原文摘要 · Abstract (English)

A/B testing is the gold standard for selecting the better algorithm in online services. While offline evaluation has attracted attention as a safer alternative due to the high experimental costs and the potential risk of degrading user experience and revenue in A/B testing, it is widely recognized that the estimation accuracy of offline evaluation is substantially lower. As a result, final selection decisions are typically made through A/B testing. Contrary to this conventional view, we reveal a counterintuitive phenomenon in which A/B testing can produce a higher algorithm selection error rate than offline evaluation. This occurs because the sample mean estimator used in A/B testing does not induce positive correlation, which is crucial for reducing critical selection errors, namely underestimating the truly superior algorithm and overestimating the truly inferior one. In contrast, offline evaluation methods unintentionally generate this beneficial correlation by relying on shared offline data when estimating and comparing the performance of multiple algorithms. Building on this insight, we propose an estimator that intentionally induces positive correlation to improve algorithm selection in A/B testing. The key idea is to introduce a hypothetical middle algorithm and to estimate the performance difference between algorithms A, M, and B in a stepwise manner using shared data at each step. This approach enables the application of offline evaluation techniques in each step, thereby inducing positive correlation and reducing critical selection errors. Furthermore, we derive the optimal middle algorithm regarding the resulting variance and analyze its advantages over existing methods through bias-variance analysis. Experiments on real-world data demonstrate that our estimator achieves the same selection error rate as existing approaches while using only one half of the A/B testing data.

算法评估A/B测试离线评估统计优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。