用强化学习训练模型在隐空间推理,提升推荐系统效率与效果。
Reinforced Latent Reasoning for LLM-based Recommendation
- 不依赖思维链数据,通过隐式表示实现紧凑推理
- 在多个推荐任务上超越现有方法,性能显著提升
- 适合追求高效推理的工业级推荐系统应用
大语言模型在复杂问题求解中展现出强大推理能力,推动其在推荐系统中的偏好推理应用。现有方法多依赖高质量思维链(CoT)数据进行微调,但面临数据难获取和推理延迟高的问题。本文提出一种新思路:从显式思维链转向紧凑、信息密集的隐式推理,避免生成显式推理过程,仅用少量隐向量即可捕捉完整推理流程。为此,我们提出端到端的强化学习框架 LatentR$^3$,无需任何CoT数据即可优化隐式推理。该框架采用两阶段训练:先监督微调初始化隐式推理模块,再通过基于规则的奖励设计进行纯强化学习训练。采用改进的GRPO算法,降低训练开销并引入连续奖励信号以提升学习效率。大量实验表明,LatentR$^3$ 在不依赖直接推理监督的情况下实现了有效推理,集成至不同基于LLM的推荐方法后性能显著提升。代码已开源于 https://github.com/xuwenxinedu/R3。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated impressive reasoning capabilities in complex problem-solving tasks, sparking growing interest in their application to preference reasoning in recommendation systems. Existing methods typically rely on fine-tuning with explicit chain-of-thought (CoT) data. However, these methods face significant practical limitations due to (1) the difficulty of obtaining high-quality CoT data in recommendation and (2) the high inference latency caused by generating CoT reasoning. In this work, we explore an alternative approach that shifts from explicit CoT reasoning to compact, information-dense latent reasoning. This approach eliminates the need for explicit CoT generation and improves inference efficiency, as few latent tokens can effectively capture the entire reasoning process. Building on this idea, we propose \textit{\underline{R}einforced \underline{Latent} \underline{R}easoning for \underline{R}ecommendation} (LatentR$^3$), a novel end-to-end training framework that leverages reinforcement learning (RL) to optimize latent reasoning without relying on any CoT data. LatentR$^3$ adopts a two-stage training strategy: first, supervised fine-tuning to initialize the latent reasoning module, followed by pure RL training to encourage exploration through a rule-based reward design. Our RL implementation is based on a modified GRPO algorithm, which reduces computational overhead during training and introduces continuous reward signals for more efficient learning. Extensive experiments demonstrate that LatentR$^3$ enables effective latent reasoning without any direct supervision of the reasoning process, significantly improving performance when integrated with different LLM-based recommendation methods. Our codes are available at https://github.com/xuwenxinedu/R3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。