arXiv:2509.22677stat.APcs.LG2025-09

用贝叶斯框架优化多版本实验,直接提升利润而非只看转化率。

Profit over Proxies: A Scalable Bayesian Decision Framework for Optimizing Multi-Variant Online Experiments

  • 构建分层贝叶斯模型,同时估算转化概率与单次交易价值。
  • 基于期望损失设计停止规则,避免高转化低利润的陷阱。
  • 适合追求真实业务收益、重视实验严谨性的产品与数据团队。

在线受控实验(如A/B测试)是数字经济中数据驱动决策的核心。然而,其实际应用常受两大缺陷制约:一是使用统计有偏的启发式方法(如“p值窥探”),导致假阳性率升高;二是过度依赖转化率等代理指标,可能做出损害核心盈利能力的决策。本文提出一个全面且可扩展的贝叶斯决策框架,用于多版本(A/B/n)实验中的利润优化。该框架采用分层贝叶斯模型,同时估计转化概率(使用Beta-Bernoulli模型)和单次交易金额的均值(使用稳健的贝叶斯模型)。在此基础上,基于期望损失设计决策论停止规则,使实验可在识别出更优变体时终止,也可在确认所有变体均无实际收益提升时提前停止(停止于无效性)。该框架有效规避了“收入陷阱”,即转化率高但净收益为负的情况,能早期终止无效实验以节约资源,并在整个监控过程中保持严格的统计完整性。本研究为组织从简单A/B测试迈向成熟的利润导向实验文化提供了实用而原则性的方法,确保统计结论直接转化为战略商业价值。

原文摘要 · Abstract (English)

Online controlled experiments (A/B tests) are fundamental to data-driven decision-making in the digital economy. However, their real-world application is frequently compromised by two critical shortcomings: the use of statistically flawed heuristics like "p-value peeking", which inflates false positive rates, and an over-reliance on proxy metrics like conversion rates, which can lead to decisions that inadvertently harm core business profitability. This paper addresses these challenges by introducing a comprehensive and scalable Bayesian decision framework designed for profit optimization in multi-variant (A/B/n) experiments. We propose a hierarchical Bayesian model that simultaneously estimates the probability of conversion (using a Beta-Bernoulli model) and the monetary value of that conversion (using a robust Bayesian model for the mean transaction value). Building on this, we employ a decision-theoretic stopping rule based on Expected Loss, enabling experiments to be concluded not only when a superior variant is identified but also when it becomes clear that no variant offers a practically significant improvement (stopping for futility). The framework successfully navigates "revenue traps" where a variant with a higher conversion rate would have resulted in a net financial loss, correctly terminates futile experiments early to conserve resources, and maintains strict statistical integrity throughout the monitoring process. Ultimately, this work provides a practical and principled methodology for organizations to move beyond simple A/B testing towards a mature, profit-driven experimentation culture, ensuring that statistical conclusions translate directly to strategic business value.

贝叶斯推理A/B测试利润优化实验设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。