让大模型在不训练的情况下,自我反思并自动提前停止,节省计算资源。
Training-Free Inference-Time Self-Reflection and Cost-Bounded Early Stopping for Large Language Models

- 推理时通过生成-自评-修改循环实现自我反思,无需梯度更新。
- 在100个Big-Bench Hard问题上,82%-88%的题目在约2.1次生成内提前终止,准确率不变。
- 适合追求高效推理、控制计算成本的研究者和应用开发者。
现有基于强化学习训练推理型大模型(如GRPO)的方法成本高昂,且依赖可控环境,每项改进都需完整训练流程。本文提出EvoResearcher,一种无需训练、在推理阶段执行的协议,为单一冻结的LLM主干添加成本受限的自我反思能力。该协议持续进行生成→自评→修正,直至达到最大深度D或自评返回CONFIRMED哨兵信号,实现隐式提前终止,在严格计算预算下让主干自我验证答案。四个自反思元奖励组件(正确性、效率、反思深度、工具调用多样性)以提示级机制实现,其优势无需梯度更新即可获得。在Big-Bench Hard(100题)、GSM8K(500题)和MATH(500题)上验证了跨领域行为,且在Qwen2.5-72B模型上复现成功。所有实验均使用纯推理基准;工具调用多样性组件以提示级形式验证,环境级与多智能体扩展留待未来工作。在纯净的BBH数据集上,协议未使准确率超出95%威尔逊区间;其核心价值在于成本受限的自我验证,即在保持准确率的同时,82%-88%的样本可在约2.1次生成内终止。
原文摘要 · Abstract (English)
Reinforcement-learning training of reasoning LLMs (e.g., GRPO) is expensive and requires a controllable environment, committing every contribution to a full training pipeline. We present EvoResearcher, a training-free, inference-time protocol that adds cost-bounded self-reflection to a single frozen LLM backbone. The protocol iterates generate -> self-critique -> revise until a maximum depth D is reached or the critique returns the CONFIRMED sentinel, an implicit early stop that lets the backbone self-verify its answer under a strict compute budget. Four self-reflective meta-reward components (correctness, efficiency, reflection depth, tool-call diversity) act as design principles instantiated as prompt-level mechanisms, so their benefits accrue with zero gradient updates. We validate the protocol on Big-Bench Hard (100 questions) and establish cross-domain behavior on GSM8K (500) and MATH (500) on the same frozen backbone, with cross-model replication on Qwen2.5-72B. All experiments use pure-reasoning benchmarks; the tool-call diversity component is validated in prompt-level form, and the environment-level and multi-agent extensions are design blueprints left to future work. On clean BBH the protocol does not raise accuracy beyond the 95% Wilson interval; its value is cost-bounded self-verification, with the CONFIRMED early stop terminating 82-88% of items at equal accuracy (about 2.1 generations per question).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。