预测排序模型的点击与视频观看等目标的分发比例,提前发现线上效果可能下降的风险。
Impression Share Prediction: An Offline Evaluation Task for Ranking Systems

- 构建因果模型,从历史数据推断候选模型的各目标分发分布。
- 随机森林在训练数据上将误差降低49%,但新模型首小时表现差于基线。
- 引入模拟滚动架构,在新模型评估中提升22%精度,适合早期在线验证前使用。
离线评估是排序模型在线A/B测试前的关键环节。现有指标仅衡量预测准确性,却无法反映下游效用:模型可能提升指标,却导致点击、视频观看等目标的曝光分配失衡。目前尚无离线方法能提前揭示此类曝光分布变化。本文提出「印象分发预测」作为新离线评估任务:给定候选排序模型,预测其在不同优化目标(如点击、视频观看)间的曝光分布。该任务为反事实问题,因候选模型未实际服务流量。我们建立结构化因果模型,揭示模型预测与系统承载能力共同决定曝光分配,并证明反事实效应可从观测数据中识别。基于此,我们设计统计学习框架,利用早期交互置信度信号与当前系统状态,预测曝光分布,基于历史数据训练。在多个排序模型族数据上,随机森林在训练见过的模型上使L1误差比恒定基线降低49%;对于未见模型,按首次上线时间评估,首小时最接近真实在线表现且最难预测,此时随机森林低于基线,因容量状态仍反映旧模型。采用编码器条件化的架构,模拟最近拍卖动态下的两小时渐进式部署,使该场景下L1误差提升22%。
原文摘要 · Abstract (English)
Offline evaluation is a major gateway before online evaluation of ranking models in A/B testing. Standard offline metrics measure predictive accuracy, but are only a surrogate for downstream utility: a model can improve them while redistributing impressions across objective buckets in ways that degrade downstream utility. No offline method surfaces these impression share shifts before online evaluation. We propose \emph{impression share prediction} as an offline evaluation task: given a candidate ranking model, predict the distribution of impressions it would produce across objective buckets - impressions grouped by optimization goal (e.g., click, video view). The task is inherently counterfactual, since the candidate has never served live traffic. We propose a structural causal model of how model predictions and delivery capacity jointly determine impression allocation, and show the counterfactual effect is identified from observational data. Building on this, we develop a statistical learning framework that predicts impression shares from a candidate's early-interaction confidence signals and current system state, trained on historical data. On data from multiple ranking model families, a Random Forest reduces L1 error by 49\% over a constant baseline for models seen during training. For held-out models, evaluated by time since first appearance, the first hour is the closest analog to true online evaluation and the hardest: the Random Forest falls below the baseline because the capacity state still reflects the prior model. An encoder-conditioned architecture that simulates a 2-hour rollout over recent auction dynamics recovers $+$22\% L1 in this regime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。