提出高效多批次方法,精准评估用户学习下的长期影响与终身价值变化
Efficient Multi-Cohort Inference for Long-Term Effects and Lifetime Value in A/B Testing with User Learning

- 采用逆方差加权融合多批次估计,降低长期效应估算方差
- 通过参数衰减模型还原渐进效应与累计价值,提升精度
- 适合关注长期产品决策的平台方,避免短视误判
流媒体平台用户流失成本极高,但传统A/B测试通常只在有限实验周期内评估结果。即使同时考虑短期与预测长期参与度,仍可能无法捕捉处理对用户留存的真实影响。因此,一项干预措施可能在短期内表现良好、长期中性,却因用户流失导致总价值低于对照组。为此,本文提出一种在用户学习背景下,基于短时多批次A/B测试高效估计长期处理效应(LTE)和残余终身价值变化(ΔERLV)的方法。为高效估计随时间变化的处理效应,引入逆方差加权估计器,结合多个批次的估计结果,相比文献中标准方法显著降低方差。随后将估计的处理轨迹建模为参数化衰减函数,以恢复渐近处理效应及随时间累积的价值。该框架可在单次实验中同步评估稳态影响与剩余用户价值。实证结果显示,对LTE和ΔERLV的估计精度明显提升,并识别出仅依赖短期或长期指标会导致错误产品决策的情形。
原文摘要 · Abstract (English)
In streaming platforms churn is extremely costly, yet A/B tests are typically evaluated using outcomes observed within a limited experimental horizon. Even when both short- and predicted long-term engagement metrics are considered, they may fail to capture how a treatment affects users' retention. Consequently, an intervention may appear beneficial in the short term and neutral in the long term while still generating lower total value than the control due to users churn. To address this limitation, we introduce a method that estimates long-term treatment effects (LTE) and residual lifetime value change ($ΔERLV$) in short multi-cohort A/B tests under user learning. To estimate time-varying treatment effects efficiently, we introduce an inverse-variance weighted estimator that combines multiple cohorts estimates, reducing variance relative to standard approaches in the literature. The estimated treatment trajectory is then modeled as a parametric decay to recover both the asymptotic treatment effect and the cumulative value generated over time. Our framework enables simultaneous evaluation of steady-state impact and residual user value within a single experiment. Empirical results show improved precision in estimating LTE and $ΔERLV$ and identify scenarios in which relying on either short-term or long-term metrics alone would lead to incorrect product decisions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。