让大模型在生成和推理间动态切换，提升思维质量。
TARPO: Token-Wise Latent-Explicit Reasoning via Action-Routing Policy Optimization
- 每步自主决定用词元生成还是连续推理，灵活适应任务
- 在多个大模型上表现优于现有方法，7B模型提升显著
- 适合需要高质量逻辑推理的应用场景
隐式推理作为大语言模型中离散思维链（CoT）的替代方案，通过连续表示实现更丰富的推理能力。然而，连续表示的固有确定性限制了强化学习中的策略探索。为此，我们提出TARPO（基于动作路由策略优化的词元级隐式推理），一个纯强化学习框架，可在每一步自适应地在离散词元生成与连续隐式推理之间切换。TARPO引入轻量级动作头路由机制，根据当前隐藏状态从二元模式选择空间采样路由决策，保留了词汇表采样中的随机性。将语言模型主干与路由模块联合端到端优化，共享组相对优势信号。在Qwen2.5（1.5B至7B）和Llama-3.1-8B等多模型上的实验表明，TARPO在多个基准测试中持续优于现有显式与隐式推理强化学习基线。进一步分析显示，TARPO能学习到自适应的词元级切换行为，同时保持稳定的训练动态。代码已开源：https://github.com/NKU-LITI/TARPO-master。
原文摘要 · Abstract (English)
Latent reasoning has emerged as a promising alternative to discrete Chain-of-Thought (CoT) in large language models (LLMs), enabling more expressive reasoning by operating over continuous representations. However, the inherently deterministic nature of continuous representations limits policy exploration in reinforcement learning (RL). To address this, we propose TARPO (Token-Wise Latent-Explicit Reasoning via Action-Routing Policy Optimization), a pure RL framework that adaptively switches between discrete token generation and continuous latent reasoning at each step. TARPO introduces a lightweight action head router that observes the current hidden state and samples a routing decision from a binary mode-selection space, preserving the stochasticity of discrete token sampling from the vocabulary. The LLM backbone and router are jointly optimized end-to-end with a shared group-relative advantage signal. Extensive experiments across Qwen2.5 (from 1.5B to 7B) and Llama-3.1-8B backbones demonstrate that TARPO consistently outperforms existing explicit and latent reasoning RL baselines across diverse benchmarks. Further analysis shows that TARPO learns adaptive token-wise switching behaviors while maintaining stable training dynamics. Our code is available at https://github.com/NKU-LITI/TARPO-master.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。