arXiv:2608.11980cs.IRcs.AI2026-08中稿 · CIKM 2026

让生成式推荐在难以达成目标时仍能学习,通过提示引导突破优化瓶颈。

Learning from Unreachable Rewards: Hint-Conditioned Reinforcement Learning for Generative Recommendation

论文配图:Learning from Unreachable Rewards: Hint-Conditioned Reinforcement Learning for Generative Recommendation
图 1 · 摘自论文原文
  • 用检查点回溯诊断困难样本,仅对无法达标者提供最小提示前缀。
  • 将零奖励组转化为有效对比,使模型在提示分支中生成后缀并提升性能。
  • 适合做生成式推荐的开发者,尤其关注奖励信号稀疏场景。

语义ID生成式推荐将每个物品表示为离散语义标记序列,并通过自回归生成预测下一个物品。该范式统一了物品ID、历史记录和文本的生成接口,但在基于奖励的微调阶段产生结构化优化瓶颈:当早期语义标记进入错误分支时,有限滚动群体极少到达真实物品,导致组内相对优化获得相同零奖励,无法产生有效改进。我们提出提示条件生成推荐(HCGRec),一种可恢复困难训练实例学习信号的框架。HCGRec通过检查点回溯诊断每例,仅在当前生成器无法到达正确物品时提供最小目标前缀提示。模型随后在提示语义分支下生成未提示后缀,将零奖励组转为关于物品标记完成的有效比较。提示还改变了标记身份:提示前缀为真值提供的物品上下文,未提示后缀为采样生成动作。为此引入提示感知信用分解,使用监督学习保持提示标记的物品语义与前缀结构对齐,同时使用GRPO优化采样后缀。在序列推荐基准上的实验表明,HCGRec显著优于监督微调和原始奖励后训练,同时将零优势训练样本从超过70%降至20%以下。代码已开源:https://github.com/WncFht/GRec。

原文摘要 · Abstract (English)

Semantic-ID generative recommenders represent each item as a short sequence of discrete semantic tokens and predict the next item by autoregressively generating this token sequence. This paradigm enables a unified generation interface for item IDs, histories, and item text, but it also creates a structured optimization bottleneck during reward-based post-training: when an early semantic token enters the wrong branch of the item-token space, finite rollout groups rarely reach the ground-truth item, so group-relative optimization receives identical zero rewards and produces no useful advantage. We propose Hint-Conditioned Generative Recommendation (HCGRec), a semantic-ID generative recommendation framework that recovers learning signal for such hard training instances. HCGRec diagnoses each instance with checkpoint rollouts and supplies a minimal target-prefix hint only when the current generator cannot reach the correct item. The model then generates the unhinted suffix under the hinted semantic branch, turning zero-reward groups into informative comparisons over item-token completions. Hinting also changes token identity: hinted prefix tokens are oracle-provided item context, while unhinted suffix tokens are sampled generation actions. We therefore introduce hint-aware credit decomposition, using supervised learning to preserve item-semantic and prefix-structure alignment for hinted tokens and GRPO to optimize the sampled suffix. Experiments on sequential recommendation benchmarks show that HCGRec substantially improves over supervised fine-tuning and vanilla reward-based post-training, while reducing zero-advantage training samples from over 70% to below 20%. The code is accessible at https://github.com/WncFht/GRec.

生成推荐强化学习提示机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。