用模型自有的代码树做离线评估,能提升推荐系统效果预测精度。
Off-Policy Evaluation for Semantic ID Recommenders: Does the Model's Own Code Hierarchy Help?
- 利用代码前缀聚类替代单个物品,解决离线评估中样本过少问题
- 粗粒度聚类可显著降低评估误差,关键在于解码器能高效计算聚类质量
- 代码层级深度是核心调节参数,适用于资源有限的生成式推荐场景
生成式推荐系统越来越多地使用语义标识符(SIDs):每个物品由残差量化器生成的一段分层离散代码序列表示,并通过自回归方式解码。在投入昂贵的A/B测试前,团队可借助离线策略评估(OPE)判断不同解码器或重排序方案是否值得测试。本文提出一个核心问题:模型自身的SID树能否作为OPE的动作抽象?答案有三部分:(i) 在近最大值采样策略下,基于单个物品的OPE几乎无效——因生产日志中有效样本量通常太小;但将物品聚合到代码前缀簇后,可恢复可估计的支持集并降低误差。(ii) 这一收益源于粗粒度化,而非层级结构本身;但正是SID树使得粗粒度化在生成系统中可行——每个簇的质量可通过解码器精确且低成本计算,而平坦聚类需枚举所有物品/叶节点质量,这在仅含代码的解码器中无法直接获取。(iii) 分辨率深度是关键调控参数——支持稀疏时应更粗粒度;条件偏差界表明,粗粒度化偏差与量化器最坏重构残差及目标-日志分布差异相关。
原文摘要 · Abstract (English)
Generative recommenders increasingly emit semantic IDs (SIDs): each item is a short sequence of hierarchical discrete codes from a residual quantizer, decoded autoregressively. Before spending scarce A/B-test, a team may decide offline which decoder or reranking variants are worth testing - a job for off-policy evaluation (OPE). We ask a simple question: can the model's own SID tree serve as the action abstraction for that OPE? Our answer has three parts. (i) Under the near-argmax logging real recommenders use, per-item OPE is hopeless - as item-level effective sample size is usually small on production logs - but marginalizing items to code-prefix clusters restores estimable support and cuts error. (ii) This gain is thanks to coarsening, not to the hierarchy specifically; but the SID tree is what makes coarsening feasible in a generative system - each cluster's mass is exactly and cheaply returned by the decoder, whereas flat clustering requires enumerating item/leaf masses that a code-only decoder does not directly expose. (iii) Resolution depth is the operative knob - coarser under scarce support - and a conditional bias bound links the coarsening bias to the quantizer's worst-case reconstruction residual and the target-logging divergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。