arXiv:2605.31228cs.LGcs.AI2026-05被引 7

通过回声机制挖掘失效推理路径中的学习信号,提升大模型强化训练效果。

EchoRL: Reinforcement Learning via Rollout Echoing

论文配图:EchoRL: Reinforcement Learning via Rollout Echoing
图 1 · 摘自论文原文
  • 基于熵模式识别优质推理片段,作为辅助监督信号
  • 在10个基准上平均提升12.3%性能,且开销极小
  • 适合希望提升推理能力的LLM后训练场景

基于可验证奖励的强化学习是增强大语言模型推理能力的有效方法。然而随着训练进行,学习信号会逐渐衰减:越来越多的提示生成的推理路径出现优势退化——所有自生成路径均显示验证成功,导致其奖励标准差为零,相应的优势值也变为零。这使得策略梯度最终消失,限制了训练性能。我们发现,尽管现有方法忽略这些路径,其中仍蕴含有价值的学习信号。受外部专家模型生成的黄金轨迹熵模式启发,本文提出EchoRL,一种轻量级模块:首先从验证成功的推理路径中根据步级熵值识别出一个回声片段(EchoClip),再将其作为辅助监督信号注入强化学习目标。在10个基准、5种主流大模型和4种常用RLVR方法上的实验表明,EchoRL能持续提升后训练性能,且计算开销极低。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards is an effective route for post-training to strengthen the reasoning capability of large language models. However, as training proceeds, the learning signal can collapse thus makes the training gain become marginal and ineffective. Specifically, a growing fraction of prompts' rollouts become advantage-degenerated: all the self-generated rollouts show verified-success, making the standard deviation over their rewards be zero; accordingly each rollout's advantage becomes degenerated (zero) as well. Given such rollouts' advantages, the policy-gradient for model optimization eventually vanishes, capping the training performance. We argue that some of these rollouts still contain valuable learning signals but unfortunately omitted with the existing RLVR methods. In this paper, inspired through analyzing the entropy pattern behind golden trajectories produced by external expert models, we propose EchoRL for better exploiting the advantage-degenerated rollouts to further improve the training performance. EchoRL is a lightweight module that first identifies an EchoClip from verified-success rollouts based on their step-level entropy values, and then feeds this clip back as an auxiliary supervision signal in the RL objective. Extensive experiments across 10 benchmarks, 5 LLM backbones, and 4 popular RLVR post-training methods demonstrate that EchoRL consistently improves RLVR post-training with minimal overhead.

强化学习大模型推理后训练熵分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。