让大模型用更少的词做出更好推理,提升效率与效果。
Learning to Think: Information-Theoretic Reinforcement Fine-Tuning for LLMs
- 基于信息论设计奖励机制,衡量每步推理的信息增益。
- 通过快速估算减少计算开销,实现高效强化学习微调。
- 适合追求推理效率和资源优化的研究者与开发者。
大语言模型(LLMs)在复杂任务中表现出色,得益于其推理能力的进步。然而,现有方法忽视了推理有效性与效率之间的权衡,常导致不必要的长推理链和令牌浪费。为此,我们提出「学会思考」(L2T),一种基于信息论的强化微调框架,使模型在更少令牌下实现最优推理。具体而言,L2T将每个查询-响应交互视为多阶段的层级会话,并提出一种通用的密集过程奖励,即量化参数层面的每阶段信息增益,无需额外标注或特定任务评估器。我们提出基于PAC-Bayes界和Fisher信息矩阵的快速估计方法,理论分析表明其显著降低计算复杂度且估计精度高。通过即时奖励每阶段贡献并惩罚过度更新,L2T利用强化学习优化模型,最大化每阶段作用并实现有效更新。在多个推理基准和基础模型上的实证结果表明,L2T在不同任务中均表现优越,同时提升了推理的有效性与效率。
原文摘要 · Abstract (English)
Large language models (LLMs) excel at complex tasks thanks to advances in their reasoning abilities. However, existing methods overlook the trade-off between reasoning effectiveness and efficiency, often encouraging unnecessarily long reasoning chains and wasting tokens. To address this, we propose Learning to Think (L2T), an information-theoretic reinforcement fine-tuning framework for LLMs to make the models achieve optimal reasoning with fewer tokens. Specifically, L2T treats each query-response interaction as a hierarchical session of multiple episodes and proposes a universal dense process reward, i.e., quantifies the episode-wise information gain in parameters, requiring no extra annotations or task-specific evaluators. We propose a method to quickly estimate this reward based on PAC-Bayes bounds and the Fisher information matrix. Theoretical analyses show that it significantly reduces computational complexity with high estimation accuracy. By immediately rewarding each episode's contribution and penalizing excessive updates, L2T optimizes the model via reinforcement learning to maximize the use of each episode and achieve effective updates. Empirical results on various reasoning benchmarks and base models demonstrate the advantage of L2T across different tasks, boosting both reasoning effectiveness and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。