arXiv:2604.06268cs.LG2026-04被引 11

发现强化学习中推理模板坍塌问题,提出用互信息诊断并改进模型输入响应能力。

RAGEN-2: Reasoning Collapse in Agentic RL

  • 分解推理质量为熵与互信息,用互信息替代熵诊断稳定性
  • 互信息比熵更紧密关联最终性能,揭示隐藏的模板坍塌现象
  • 基于信噪比设计轻量筛选机制,提升多任务推理表现

多轮大模型智能体的强化学习训练本质上不稳定,推理质量直接决定任务表现。现有方法普遍使用熵来追踪推理稳定性,但熵仅衡量同一输入内的多样性,无法判断推理是否对不同输入作出响应。在RAGEN-2中,我们发现即使熵稳定,模型仍可能依赖固定模板,看似多样实则与输入无关,称为模板坍塌——一种熵和现有指标均无法检测的失效模式。为此,我们将推理质量分解为输入内多样性(熵)和跨输入可区分性(互信息,MI),引入一组互信息代理用于在线诊断。在多种任务中,互信息与最终性能相关性远高于熵,是更可靠的推理质量代理。我们进一步通过信噪比(SNR)机制解释模板坍塌:低奖励方差削弱任务梯度,使正则项主导,抹除跨输入差异。为此,我们提出SNR感知过滤,在每轮迭代中利用奖励方差作为轻量代理筛选高信号提示。在规划、数学推理、网页导航和代码执行任务中,该方法一致提升了输入依赖性和任务性能。

原文摘要 · Abstract (English)

RL training of multi-turn LLM agents is inherently unstable, and reasoning quality directly determines task performance. Entropy is widely used to track reasoning stability. However, entropy only measures diversity within the same input, and cannot tell whether reasoning actually responds to different inputs. In RAGEN-2, we find that even with stable entropy, models can rely on fixed templates that look diverse but are input-agnostic. We call this template collapse, a failure mode invisible to entropy and all existing metrics. To diagnose this failure, we decompose reasoning quality into within-input diversity (Entropy) and cross-input distinguishability (Mutual Information, MI), and introduce a family of mutual information proxies for online diagnosis. Across diverse tasks, mutual information correlates with final performance much more strongly than entropy, making it a more reliable proxy for reasoning quality. We further explain template collapse with a signal-to-noise ratio (SNR) mechanism. Low reward variance weakens task gradients, letting regularization terms dominate and erase cross-input reasoning differences. To address this, we propose SNR-Aware Filtering to select high-signal prompts per iteration using reward variance as a lightweight proxy. Across planning, math reasoning, web navigation, and code execution, the method consistently improves both input dependence and task performance.

强化学习推理质量模板坍塌互信息

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。