arXiv:2601.04973cs.AIcs.CL2026-01被引 2

用强化学习压缩推理过程,减少冗余但保持准确率。

ConMax: Confidence-Maximizing Compression for Efficient Chain-of-Thought Reasoning

  • 通过奖励驱动优化,自动修剪推理路径中的冗余内容。
  • 推理长度减少43%,准确率仅下降0.7%。
  • 适合需要高效训练数据的大型推理模型研究者。

近期大型推理模型(LRMs)的突破表明,生成详尽的思维链(CoT)对实现自验证、回溯等复杂认知行为至关重要。然而,这种能力常导致“过度思考”,即生成冗余推理路径,增加计算开销却未提升准确率。尽管监督微调(SFT)是推理轨迹训练的标准方法,但现有压缩技术往往损害逻辑连贯性或带来高昂采样成本。本文提出ConMax(置信度最大化压缩),一种新型强化学习框架,可自动压缩推理轨迹并保留关键推理模式。ConMax将压缩建模为奖励驱动优化问题,通过冻结的辅助大模型训练策略,以最大化答案置信度(预测保真度)与思考置信度(推理有效性)的加权组合。在五个推理数据集上的大量实验表明,ConMax实现了更优的效率-性能权衡:相比强基线,推理长度减少43%,准确率仅下降0.7%,证明其在生成高质量、高效训练数据方面的有效性。

原文摘要 · Abstract (English)

Recent breakthroughs in Large Reasoning Models (LRMs) have demonstrated that extensive Chain-of-Thought (CoT) generation is critical for enabling intricate cognitive behaviors, such as self-verification and backtracking, to solve complex tasks. However, this capability often leads to ``overthinking'', where models generate redundant reasoning paths that inflate computational costs without improving accuracy. While Supervised Fine-Tuning (SFT) on reasoning traces is a standard paradigm for the 'cold start' phase, applying existing compression techniques to these traces often compromises logical coherence or incurs prohibitive sampling costs. In this paper, we introduce ConMax (Confidence-Maximizing Compression), a novel reinforcement learning framework designed to automatically compress reasoning traces while preserving essential reasoning patterns. ConMax formulates compression as a reward-driven optimization problem, training a policy to prune redundancy by maximizing a weighted combination of answer confidence for predictive fidelity and thinking confidence for reasoning validity through a frozen auxiliary LRM. Extensive experiments across five reasoning datasets demonstrate that ConMax achieves a superior efficiency-performance trade-off. Specifically, it reduces inference length by 43% over strong baselines at the cost of a mere 0.7% dip in accuracy, proving its effectiveness in generating high-quality, efficient training data for LRMs.

推理压缩强化学习大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。