让大模型学会何时该思考，省下20%以上推理成本。
Rational Metareasoning for Large Language Models
- 用认知科学中的元推理机制，训练模型判断是否需要中间思考步骤。
- 在多个数据集上保持性能的同时，减少20%-37%的生成token数。
- 适合关注推理效率、降低算力开销的研究者与工程师。
提示大语言模型进行推理已成为提升任务表现的核心方法，但随着模型规模和应用的扩大，推理开销日益严重。如何优化推理的成本-性能平衡？本文提出一种基于认知科学中元推理计算模型的新方法，训练模型仅在必要时才使用中间推理步骤。我们设计了一种包含计算价值的奖励函数，通过惩罚不必要的推理来引导模型学习，并结合专家迭代（Expert Iteration）进行训练。相较于少样本思维链提示和STaR方法，本方法在三个模型上显著降低了推理成本（生成的token数减少20%-37%），同时在多个数据集上保持了相当的任务性能。
原文摘要 · Abstract (English)
Being prompted to engage in reasoning has emerged as a core technique for using large language models (LLMs), deploying additional inference-time compute to improve task performance. However, as LLMs increase in both size and adoption, inference costs are correspondingly becoming increasingly burdensome. How, then, might we optimize reasoning's cost-performance tradeoff? This work introduces a novel approach based on computational models of metareasoning used in cognitive science, training LLMs to selectively use intermediate reasoning steps only when necessary. We first develop a reward function that incorporates the Value of Computation by penalizing unnecessary reasoning, then use this reward function with Expert Iteration to train the LLM. Compared to few-shot chain-of-thought prompting and STaR, our method significantly reduces inference costs (20-37\% fewer tokens generated across three models) while maintaining task performance across diverse datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。