提出一种可预测线上效果的离线评估方法，提升推荐系统迭代效率。
Identifying Offline Metrics that Predict Online Impact: A Pragmatic Strategy for Real-World Recommender Systems
- 基于帕累托前沿逼近，设计多组测试并行的离线评估策略
- 在OTTO平台验证，离线指标与点击率、转化率等线上指标显著相关
- 适用于多种神经网络架构，适合工业界快速决策
推荐系统面临的核心挑战是如何建立可靠的离线指标与线上表现之间的关联。受帕累托前沿近似进展的启发，本文提出一种务实策略，用于识别与线上影响对齐的离线指标。该方法关键优势在于，能通过单一模型同时服务于多个具有不同离线性能指标的测试组，在线上实验中实现高效对比。该方法对采用神经网络骨干的系统具有模型无关性，适用范围广。我们在OTTO电商平台上大规模验证了该策略，结果显示离线指标与真实世界的点击率、后点击转化率及销量之间存在显著关联。该方法为产业界提供了理解离线-线上指标关系的有力工具，支持数据驱动的决策制定。
原文摘要 · Abstract (English)
A critical challenge in recommender systems is to establish reliable relationships between offline and online metrics that predict real-world performance. Motivated by recent advances in Pareto front approximation, we introduce a pragmatic strategy for identifying offline metrics that align with online impact. A key advantage of this approach is its ability to simultaneously serve multiple test groups, each with distinct offline performance metrics, in an online experiment controlled by a single model. The method is model-agnostic for systems with a neural network backbone, enabling broad applicability across architectures and domains. We validate the strategy through a large-scale online experiment in the field of session-based recommender systems on the OTTO e-commerce platform. The online experiment identifies significant alignments between offline metrics and real-word click-through rate, post-click conversion rate and units sold. Our strategy provides industry practitioners with a valuable tool for understanding offline-to-online metric relationships and making informed, data-driven decisions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。