arXiv:2604.22169cs.LGcs.AI2026-04

针对生成式推荐中稀疏反馈导致学习信号失效的问题,提出修复与对比的信号重构方法。

ReCast: Recasting Learning Signals for Reinforcement Learning in Generative Recommendation

  • 先修复全零组的可学习性,再用强正负样本对比替代整体归一化
  • 在Pass@1上相对基线提升最高达36.6%,预算仅需4.1%仍达目标性能
  • 适合大规模生成式推荐系统,显著降低内存和更新时间开销

通用分组强化学习假设采样组可直接作为学习信号,但在稀疏命中场景下,许多采样组根本无法构成有效学习信号。我们提出ReCast,一种先修复后对比的学习信号框架:首先恢复全零组的最小可学习性,然后将全组奖励归一化替换为聚焦边界的一对强正/难负样本的对比更新。ReCast不改变外层强化学习框架,仅调整组内信号构建方式,并部分解耦回放搜索宽度与执行端更新宽度。在多个生成式推荐任务中,ReCast持续优于OpenOneRec-RL,Pass@1最高实现36.6%相对提升。其等预算优势更显著:仅需4.1%的回放预算即可达到基线目标性能,且随模型规模扩大优势进一步增强。相同设计带来系统级收益:执行端更新时间减少16.60倍,峰值内存降低16.5%,执行利用率(MFU)提升14.2%。机制分析表明,ReCast缓解了长期存在的全零/单命中状态,当自然正样本稀缺时恢复可学习性,并将原本浪费的回放预算转化为更稳定的策略更新。这些结果表明,对于生成式推荐,强化学习的关键不仅是如何分配奖励,更是如何从稀疏、结构化监督中构建可优化事件。

原文摘要 · Abstract (English)

Generic group-based RL assumes that sampled rollout groups are already usable learning signals. We show that this assumption breaks down in sparse-hit generative recommendation, where many sampled groups never become learnable at all. We propose ReCast, a repair-then-contrast learning-signal framework that first restores minimal learnability for all-zero groups and then replaces full-group reward normalization with a boundary-focused contrastive update on the strongest positive and the hardest negative. ReCast leaves the outer RL framework unchanged, modifies only within-group signal construction, and partially decouples rollout search width from actor-side update width. Across multiple generative recommendation tasks, ReCast consistently outperforms OpenOneRec-RL, achieving up to 36.6% relative improvement in Pass@1. Its matched-budget advantage is substantially larger: ReCast reaches the baseline's target performance with only 4.1% of the rollout budget, and this advantage widens with model scale. The same design also yields direct system-level gains, reducing actor-side update time by 16.60x, lowering peak allocated memory by 16.5%, and improving actor MFU by 14.2%. Mechanism analysis shows that ReCast mitigates the persistent all-zero / single-hit regime, restores learnability when natural positives are scarce, and converts otherwise wasted rollout budget into more stable policy updates. These results suggest that, for generative recommendation, the decisive RL problem is not only how to assign rewards, but how to construct learnable optimization events from sparse, structured supervision.

强化学习推荐系统生成式模型信号重构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。