在固定预算下优化多指标实验设计，提升高效筛选最优方案的能力。
Multi-Metric Adaptive Experimental Design Under a Fixed Budget with Validation
- 分两阶段：先自适应探索选最优处理，再用A/B测试验证并推断效果。
- 提出SHRVar算法，误差概率指数下降，适用于多种指标和方差差异场景。
- 适合在线实验中需同时评估多个关键指标的研究者或工程师使用。
在线实验中的A/B测试在同时评估多个候选方案时面临统计功效不足的问题，而单纯的自适应实验设计（AED）难以准确推断诸如平均处理效应等实验统计量，尤其当涉及多个指标（如收入、安全性）且方差异质时。本文提出一种固定预算下的多指标自适应实验设计框架，采用两阶段结构：第一阶段为自适应探索，用于识别最优处理；第二阶段为验证阶段，通过A/B测试确认处理质量并推断统计结果。提出SHRVar算法，该算法在顺序减半（SH）基础上引入基于相对方差的采样策略及基于奖励z值的剔除机制。其误差概率可证明呈指数下降，指数H3分别推广了传统SH与SHVar在同质与异质方差下的复杂度度量。数值实验验证了其性能与鲁棒性。
原文摘要 · Abstract (English)
A/B tests in online experiments face statistical power challenges when testing multiple candidates simultaneously, while adaptive experimental designs (AED) alone fall short in inferring experiment statistics such as the average treatment effect, especially with many metrics (e.g., revenue, safety) and heterogeneous variances. This paper proposes a fixed-budget multi-metric AED framework with a two-phase structure: an adaptive exploration phase to identify the best treatment, and a validation phase with an A/B test to verify the treatment's quality and infer statistics. We propose SHRVar, which generalizes sequential halving (SH) with a novel relative-variance-based sampling and an elimination strategy built on reward z values. It achieves a provable error probability that decreases exponentially, where the exponent H3 generalizes the complexity measure for SH and SHVar with homogeneous and heterogeneous variances, respectively. Numerical experiments demonstrate its performance and robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。