用强化学习让大模型自动压缩思维链，既省时间又不丢准
Reinforcement Learning for Chain of Thought Compression with One-Domain-to-All Generalization
- 基于样本级软强化学习，只在已解题时压缩冗长推理
- 跨领域压缩效果显著：数学训练可缩短代码/问答等任务40%输出
- 适用于所有需要思考的场景，尤其适合追求效率的部署应用
大型语言模型的思维链推理易陷入‘过度思考’陷阱：更长的推理路径虽增加计算成本与延迟，但准确率提升有限。现有方法采用全局静态控制，可能抑制必要推理。本文提出一种基于精通门控的、样本级别的软强化学习压缩机制，仅在模型已解决任务且生成了更短推理路径时，对长推理进行惩罚。在多个基准测试中，该方法将响应长度减少20%-40%，同时保持或提升准确率，并实现从单一领域到全领域的泛化：仅在数学任务上训练的模型，可自发压缩未见过的任务（如代码生成、指令遵循、通用知识问答），且不影响准确性。进一步发现，非代理型思维链与工具使用代理间存在双向迁移：非代理训练使SWE-Bench Verified回合减少13%；压缩思维代理可减少SWE轨迹67%的词数和52%的回合数，同时使非代理输出缩短达44%。压缩并非表面简洁，而是本质的计算策略——决定保留什么，舍弃什么。
原文摘要 · Abstract (English)
Chain-of-thought reasoning in large language models can trigger an "overthinking trap": longer rollouts raise cost and latency yet often yield unreliable accuracy gains. Existing methods use global, static controls that may suppress needed reasoning. We propose mastery-gated, sample-level, soft reinforcement learning compression that penalizes long rollouts only when the model already solves the problem and has produced a shorter rollout. Across benchmarks, it cuts response length by 20-40% with comparable or higher accuracy and generalizes across domains: a model trained on math spontaneously shortens unseen tasks (code, instruction following, general-knowledge QA) without hurting accuracy. We further show two-way transfer between non-agent CoT and tool-use agents: non-agent training reduces SWE-Bench Verified rounds by 13%, while compressing a thinking agent cuts SWE trajectories by 67% tokens and 52% rounds and shortens non-agent outputs by up to 44%. Compression is thus not cosmetic brevity, but an inherent computation policy -- what to keep, and what to forget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。