用信息瓶颈理论优化大模型推理,提升准确率且无需额外计算
Revisiting LLM Reasoning via Information Bottleneck
- 基于信息瓶颈原理设计推理优化框架,让思考过程既关键又通用
- 在多个数学推理数据集上,性能普遍提升,最高增益达12.7%
- 可无缝接入现有强化训练流程,仅需修改一行代码
大语言模型通过基于可验证奖励的强化学习(RLVR)在推理能力上取得显著进展。利用简单的规则奖励,强化学习能有效激励模型生成更长的思维链(CoT),逐步引导至正确答案。然而,现有方法仍多依赖直觉和经验,缺乏系统性理论指导。本文基于信息瓶颈(IB)原理,提出一种新的推理优化框架IBRO,使推理路径在保留与最终答案相关的关键信息的同时具备跨提示的泛化能力。我们推导出适用于词元级别的近似目标函数,并提出轻量级正则化方法。该方法可无额外开销地集成到现有基于强化学习的后训练框架中,仅需一行代码修改。实验表明,在多个数学推理基准测试及不同强化学习算法下,该正则化方法均带来稳定性能提升。
原文摘要 · Abstract (English)
Large language models (LLMs) have recently demonstrated remarkable progress in reasoning capabilities through reinforcement learning with verifiable rewards (RLVR). By leveraging simple rule-based rewards, RL effectively incentivizes LLMs to produce extended chain-of-thought (CoT) reasoning trajectories, progressively guiding them toward correct answers. However, existing approaches remain largely heuristic and intuition-driven, limiting the development of principled methodologies. In this paper, we present a theoretical characterization of LLM reasoning grounded in information bottleneck (IB) principle, introducing IB-aware reasoning optimization (IBRO), a framework that encourages reasoning trajectories to be both informative about the final correct answer and generalizable across diverse prompts. We derive a practical token-level surrogate objective and propose an efficient approximation, resulting in the lightweight IB regularization method. This technique integrates seamlessly into existing RL-based post-training frameworks without additional computational overhead, requiring only a one-line code modification. Empirically, we validate IB regularization across multiple mathematical reasoning benchmarks and RL algorithms, demonstrating consistent improvements in LLM reasoning performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。