arXiv:2605.17648cs.AI2026-05

针对生成推荐中推理步骤的信用分配难题,提出分步对齐优化方法。

SAPO: Step-Aligned Policy Optimization for Reasoning-Based Generative Recommendation

论文配图:SAPO: Step-Aligned Policy Optimization for Reasoning-Based Generative Recommendation
图 1 · 摘自论文原文
  • 按推理步骤分别计算优势值,精准分配奖励给对应思维块和标识符词元。
  • 在三个真实数据集上稳定训练并显著提升推荐效果,尤其在稀疏反馈场景下优势明显。
  • 适合研究生成式推荐、强化学习与结构化生成的学者或工程师。

生成式推荐将下一步物品预测视为自回归物品标识符生成。具体而言,物品被编码为语义标识符(SIDs),即短的粗粒度到细粒度的词元序列,早期词元捕捉整体语义,后期词元逐步细化。近期工作通过引入推理轨迹,并使用可验证奖励进行强化学习优化,通常采用基于精确匹配反馈的结果奖励算法。然而,在大规模商品库推荐中,对生成的SID仅提供最终物品是否正确的精确匹配反馈;当生成的SID不匹配时,结果奖励无法识别是哪个词元预测导致错误,可能同时惩罚正确与错误的位置。我们发现此场景下的自然信用分配单位是单个推理步骤(一个思维块配一个SID词元)。为此,我们提出SAPO(Step-Aligned Policy Optimization):不将单一优势广播至整个输出,而是为每个推理步骤计算独立的组相对优势,并仅应用于对应的思维块与SID词元。在三个真实世界推荐数据集上,SAPO稳定了强化学习训练,并持续优于现有生成式推荐基线,尤其在稀疏精确匹配反馈使推理步骤信用分配至关重要的情况下提升最大。结果表明,结构化生成的强化学习目标应反映解码器自身对输出的分解方式。

原文摘要 · Abstract (English)

Generative recommendation treats next-item prediction as autoregressive item-identifier generation. Specifically, items are encoded as semantic identifiers (SIDs), which are short coarse-to-fine token sequences whose early tokens capture broad semantics and later tokens refine them. Recent work augments this paradigm with reasoning traces and optimizes them via reinforcement learning with verifiable rewards, typically outcome-reward algorithm with exact-match feedback on the generated SID. However, in large-catalog recommendation, exact-match feedback on the generated SID only reports whether the final item is correct; when a generated SID mismatches, outcome-reward cannot identify which SID-token prediction caused the mismatch and may penalize matched SID-token positions together with the mismatched position. We identify that the natural unit of credit assignment in this setting is a single reasoning step (one thinking block paired with one SID token). We instantiate this idea in SAPO (Step-Aligned Policy Optimization): rather than broadcasting one advantage to the whole response, SAPO computes a separate group-relative advantage for each reasoning step and applies it only to the corresponding thinking block and SID token. Across three real-world recommendation datasets, SAPO stabilizes reinforcement-learning training and consistently improves over existing generative recommendation baselines, with the largest gains where sparse exact-match feedback makes reasoning-step credit assignment important. Our results suggest that reinforcement-learning objectives for structured generation should mirror the decoder's own decomposition of the output.

生成推荐强化学习结构化生成信用分配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。