解决大模型推荐系统更新中的曝光偏差问题,提升推荐准确性。
Don't Let Bandit Feedback Pull Continual LLM-Recommender Updates Off Target
- 引入锚点机制,用实际曝光项校准策略优化的相对比较基准。
- 对正反馈和无响应采用不同处理方式,避免模糊反馈导致误更新。
- 在亚马逊和电影数据集上显著减少偏差,提升推荐效果,适合持续学习场景。
基于生成式大模型的推荐系统(LLM-Rec)需持续部署后更新,但部署日志仅提供政策相关的上下文带宽反馈:仅能观察到先前服务策略暴露过的项目结果,导致曝光偏差,产生部分且不对称的信号——正反馈较可靠,无响应则模糊。本文提出锚定带宽策略优化(ABPO)框架,结合组内相对策略优化(GRPO)与显式暴露偏差和反馈模糊性处理。具体地,在每组回溯中插入实际曝光的推荐作为固定锚点,使组内归一化以先前策略暴露行为为基准,而非仅依赖新采样回溯。由于正反馈与无响应均仅通过先前策略暴露被观测,我们对固定锚点应用自归一化逆倾向评分来纠正策略不匹配。同时,对两类反馈采取不对称可靠性处理:正反馈为直接认可信号,无响应则因可能反映真实不感兴趣或外部未观测因素而保持模糊。为防止无响应引发过度激进更新,使用模型输出词元置信度作为验证器免费的可靠性信号,调节其惩罚强度。在亚马逊评论与MovieLens的五个领域中,本方法在更新后一致提升推荐准确率,且比现有基线更有效缓解由先前策略引起的曝光偏差。
原文摘要 · Abstract (English)
Generative LLM-based recommenders (LLM-Rec) require continual post-deployment updates, yet deployment logs provide only policy-shaped contextual bandit feedback: outcomes are observed solely for items exposed by a prior serving policy, inducing exposure bias and yielding partial, asymmetric signals consisting of relatively reliable positive responses and ambiguous no-responses. We propose an Anchored Bandit Policy Optimization (ABPO) framework for continual LLM-Rec updates that combines group-relative policy optimization (GRPO) with explicit treatment of exposure bias and feedback ambiguity. Specifically, we insert the exposed recommendation as a logged anchor into each GRPO rollout group, so that group-relative normalization is calibrated against the action actually exposed by the prior policy rather than against newly sampled rollouts alone. Because both positive- and no-responses are observed only through prior-policy exposure, we apply self-normalized inverse propensity scoring to the fixed anchor for both feedback types to correct for policy mismatch. At the same time, we treat the two feedback types asymmetrically in reliability: positive responses provide relatively direct endorsement signals, whereas no-responses remain ambiguous because they may reflect either true disinterest or unobserved external factors. To avoid overly aggressive updates from ambiguous no-responses, we temper their penalties with self-certainty, using the model's output-token confidence as a verifier-free reliability signal. Across five domains from Amazon Reviews and MovieLens, our method yields consistent post-update gains in recommendation accuracy while mitigating prior-policy-induced exposure bias more effectively than prior baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。