用多智能体强化学习压缩思维链,既缩短输出又提升准确率。
Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learning
- 分两步识别冗余:拆解推理过程并评分重要性。
- 压缩后长度减少11.1%至39.0%,准确率提升4.33%至10.02%。
- 适合追求高效推理的模型部署场景,尤其大模型应用。
大型推理模型因冗余推理带来的推理开销,严重影响交互体验并阻碍部署。现有基于强化学习(RL)的方法通过引入长度惩罚与结果奖励的简单加权来解决此问题,但难以兼顾简洁性与准确性,过度强调简洁可能损害关键推理逻辑。本文提出多智能体强化学习框架SCMA,通过两个专用智能体实现冗余检测与评估:分割智能体将推理过程分解为逻辑块,评分智能体量化每块的重要性。二者协同构建重要性加权的长度惩罚,在训练中激励推理智能体优先保留核心逻辑,且部署时无额外开销。跨模型规模的实证评估表明,SCMA使响应长度减少11.1%至39.0%,准确率提升4.33%至10.02%。消融实验与定性分析验证,多智能体框架中的协同优化催生了涌现行为,显著优于传统RL范式。
原文摘要 · Abstract (English)
The inference overhead induced by redundant reasoning undermines the interactive experience and severely bottlenecks the deployment of Large Reasoning Models. Existing reinforcement learning (RL)-based solutions tackle this problem by coupling a length penalty with outcome-based rewards. This simplistic reward weighting struggles to reconcile brevity with accuracy, as enforcing brevity may compromise critical reasoning logic. In this work, we address this limitation by proposing a multi-agent RL framework that selectively penalizes redundant chunks, while preserving essential reasoning logic. Our framework, Self-Compression via MARL (SCMA), instantiates redundancy detection and evaluation through two specialized agents: \textbf{a Segmentation Agent} for decomposing the reasoning process into logical chunks, and \textbf{a Scoring Agent} for quantifying the significance of each chunk. The Segmentation and Scoring agents collaboratively define an importance-weighted length penalty during training, incentivizing \textbf{a Reasoning Agent} to prioritize essential logic without introducing inference overhead during deployment. Empirical evaluations across model scales demonstrate that SCMA reduces response length by 11.1\% to 39.0\% while boosting accuracy by 4.33\% to 10.02\%. Furthermore, ablation studies and qualitative analysis validate that the synergistic optimization within the MARL framework fosters emergent behaviors, yielding more powerful LRMs compared to vanilla RL paradigms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。