通过测试时外部干预,让大模型推理更高效不绕弯。
Beyond Model Scaling: Test-Time Intervention for Efficient Deep Reasoning
- 在推理中途的关键连接词处暂停,接收外部反馈
- 准确率提升7.19%,推理长度减少81%(AIME24)
- 适合需要安全、创意或长文本推理的场景
大型推理模型在多步推理中表现优异,但常因过度思考或偏离方向导致计算开销大、性能下降。现有方法为闭环运行,缺乏外部干预机制。本文提出Think-with-Me,一种测试时交互式推理范式,将外部反馈引入推理过程。核心洞察是:过渡连接词是自然的干预点,恰当使用可延长推理以提升性能,过度则适得其反。基于此,Think-with-Me在这些节点暂停推理,接收多标准评估(合理性与完整性)生成的反馈,由人类或大模型代理提供,并通过组相对策略优化(GRPO)训练目标模型以适应该模式。实验表明,在有限上下文窗口下,Think-with-Me实现了准确率与推理长度的更优平衡:在AIME24上,相比QwQ-32B,准确率提升7.19%,平均推理长度减少81%。该范式还适用于安全与创造性任务。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) excel at multi-step reasoning but often suffer from inefficient reasoning processes like overthinking and overshoot, where excessive or misdirected reasoning increases computational cost and degrades performance. Existing efficient reasoning methods operate in a closed-loop manner, lacking mechanisms for external intervention to guide the reasoning process. To address this, we propose Think-with-Me, a novel test-time interactive reasoning paradigm that introduces external feedback intervention into the reasoning process. Our key insights are that transitional conjunctions serve as natural points for intervention, signaling phases of self-validation or exploration and using transitional words appropriately to prolong the reasoning enhances performance, while excessive use affects performance. Building on these insights, Think-with-Me pauses reasoning at these points for external feedback, adaptively extending or terminating reasoning to reduce redundancy while preserving accuracy. The feedback is generated via a multi-criteria evaluation (rationality and completeness) and comes from either human or LLM proxies. We train the target model using Group Relative Policy Optimization (GRPO) to adapt to this interactive mode. Experiments show that Think-with-Me achieves a superior balance between accuracy and reasoning length under limited context windows. On AIME24, Think-with-Me outperforms QwQ-32B by 7.19% in accuracy while reducing average reasoning length by 81% under an 8K window. The paradigm also benefits security and creative tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。