纠正大模型推理时的过度乐观，提升生成准确性
Supervised Optimism Correction: Be Confident When LLMs Are Sure
- 在监督微调中引入辅助损失，校正令牌级价值估计
- 在GSM8K等数学推理数据集上显著降低错误率
- 适合追求高精度推理的开发者和研究者使用
本文建立监督微调与离线强化学习在令牌级马尔可夫决策过程下的新理论联系，揭示大语言模型在推理时实际上学习了隐式的$Q$-函数。通过这一理论视角，我们证明广泛使用的束搜索方法存在不可接受的过度乐观问题，次优步骤的价值估计被放大，导致推理误差累积。为此，我们提出监督乐观性校正（SOC），在监督微调阶段引入简单有效的辅助损失，对令牌级$Q$-值估计进行隐式价值正则化。该方法增强模型对专家示范响应的置信度，抑制对未充分监督响应的过度乐观。在GSM8K、MATH和GAOKAO等多个数学推理基准上的大量实验表明，结合束搜索时，所提出的SOC在多个开源模型上均表现出显著优势。
原文摘要 · Abstract (English)
In this work, we establish a novel theoretical connection between supervised fine-tuning and offline reinforcement learning under the token-level Markov decision process, revealing that large language models indeed learn an implicit $Q$-function for inference. Through this theoretical lens, we demonstrate that the widely used beam search method suffers from unacceptable over-optimism, where inference errors are inevitably amplified due to inflated $Q$-value estimations of suboptimal steps. To address this limitation, we propose Supervised Optimism Correction(SOC), which introduces a simple yet effective auxiliary loss for token-level $Q$-value estimations during supervised fine-tuning. Specifically, the auxiliary loss employs implicit value regularization to boost model confidence in expert-demonstrated responses, thereby suppressing over-optimism toward insufficiently supervised responses. Extensive experiments on mathematical reasoning benchmarks, including GSM8K, MATH, and GAOKAO, showcase the superiority of the proposed SOC with beam search across a series of open-source models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。