揭示内容平台隔离实验的隐性成本,指出扩容无法消除其影响。
The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing
- 用极值理论分析匹配质量尾部,判断隔离导致的损耗是否随规模减小
- 实验证明即使在百万创作者平台,隔离仍造成可测量的用户参与度损失
- 提供预估工具帮助实验者提前评估并选择替代方案
在双端内容平台中,对创作者和观众同时进行对称隔离的A/B测试广泛用于消除跨组市场干扰。然而,隔离会缩小每位用户的候选内容池,直观上认为随着平台增长,这种损耗应趋于消失:小比例的大池子仍是大池子。本文基于顺序统计模型发现,这一直觉是否成立取决于匹配质量的上尾分布特征。极值理论表明,轻尾或有界尾部下损耗随候选池扩大而消失,但在重尾情况下,损耗收敛为与规模无关的常数,即无论池子多大,损耗均无法消除。两个真实生产实验(覆盖数百万活跃创作者)验证了该结论:纯A/A流量扫描显示存在可测量的深度相关参与度损失;单边内容池缩减实验独立证实用户池变薄是损失来源;从小探索池校准的尾指数可准确预测全量池中的实际损失。因此,隔离存在不可忽视的成本,实验者需像预算其他成本一样预先考虑。本文为实践者提供前置评估流程,可预估成本、调整流量,并在成本超限时推荐备用设计。
原文摘要 · Abstract (English)
On two-sided content platforms, symmetric two-sided isolation (assigning matched fractions of creators and viewers to isolated treatment and control submarkets) is widely used for creator-side and cold-start experiments because it removes cross-arm marketplace interference. Isolation, however, thins each viewer's candidate catalog, and intuition suggests the resulting engagement cost should fade as the platform grows: a small fraction of a vast catalog is still vast. We show that, in an order-statistics model of engagement, whether this intuition holds depends on the upper tail of match quality. Extreme-value theory yields tail-class loss laws with a sharp dichotomy: for light or bounded tails the loss vanishes as the candidate pool grows, whereas under heavy tails it converges to a size-independent constant, so expanding the candidate pool, even by orders of magnitude, does not asymptotically eliminate the cost. Evidence from two production experiments on a platform with millions of active creators is consistent with this picture: a pure A/A traffic sweep reveals a measurable, depth-graded engagement cost; a one-sided catalog ablation independently shows that per-viewer thinning contributes to the loss; and a tail index calibrated on the small exploration pool predicts an effect consistent with the one observed in the far larger full-catalog ablation. Isolation thus carries a price that experimenters should budget for, like any other cost. We give practitioners a preflight procedure that estimates it before launch, sizes traffic accordingly, and recommends a fallback design when the predicted cost exceeds a chosen tolerance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。