arXiv:2601.08403cs.AI2026-01被引 1

用博弈论方法精准分配生成文本的贡献度,提升大模型推荐效果

Owen-Shapley Policy Optimization: A Principled RL Algorithm for Generative Search LLMs

  • 基于谢帕利-欧文值重构奖励,按语义单元评估每个词的贡献
  • 在亚马逊和H&M数据集上优于基线,对未知检索器仍保持稳定
  • 适合需要理解用户隐含意图的个性化生成场景

大语言模型越来越多地通过强化学习进行个性化推荐训练,但标准方法如GRPO依赖稀疏的序列级奖励,难以定位哪些词真正提升了输出质量,造成信用分配鸿沟。这一问题在缺乏明确标签、需从模糊语言推断用户潜在意图时尤为严重,这类推理模式在预训练中罕见,却常见于实际部署。本文提出奥文-谢帕利策略优化(OSPO),通过谢帕利-欧文归因将序列级优势重新分配至各词元的边际贡献,将任务反馈转化为基于潜力的奖励塑造,在不使用参数化价值函数的前提下实现段落级信用分配并保持最优策略。通过构建语义连贯单元(如描述产品属性的短语或表达偏好的句子)形成联盟,识别影响性能的关键响应部分。在Amazon ESCI和H&M Fashion数据集上的受控生成任务实验显示,相比基线有持续提升,并对训练中未见的分布外检索器表现出显著测试时鲁棒性。

原文摘要 · Abstract (English)

Large language models are increasingly trained via reinforcement learning for personalized recommendation tasks, but standard methods like GRPO rely on sparse, sequence-level rewards. These obscure which tokens actually contribute to high-quality outputs, creating a credit assignment gap. This gap is especially problematic when models must infer latent user intent from under-specified language without ground truth labels, which is a reasoning pattern rarely seen during pretraining but commonly required in deployment. We introduce Owen-Shapley Policy Optimization (OSPO), a framework that redistributes sequence-level advantages based on tokens' marginal contributions to outcomes. OSPO transforms task feedback into potential-based reward shaping via Shapley-Owen attributions to assign segment-level credit while preserving the optimal policy, all without parametric value models. By forming coalitions of semantically coherent units (e.g., phrases describing product attributes or sentences capturing preferences), OSPO identifies which response parts drive performance. Experiments on Amazon ESCI and H&M Fashion datasets including controlled generation tasks show consistent gains over baselines and notable test-time robustness to out-of-distribution retrievers unseen during training.

强化学习生成式搜索信用分配大模型推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。