让大模型根据问题不确定性动态调整学习强度,提升数学推理准确率。
SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization
- 用语义熵衡量答案多样性,反映模型对问题的不确定程度。
- 在高不确定问题上减小更新幅度,低不确定问题保持原学习信号。
- 在5个数学基准上达新最好成绩,适合追求鲁棒性推理的场景。
大型语言模型(LLMs)在不同输入提示下表现出不同的置信度:某些问题产生一致、语义相近的回答,而另一些则导致多样甚至矛盾的输出。这种差异反映了模型对输入的不确定性,是其理解能力边界的重要信号。然而,传统的组相对策略优化(GRPO)在策略更新时对所有提示同等对待,忽略了这一关键信息。为此,我们提出SEED-GRPO(语义熵增强的GRPO),通过显式测量提示的语义熵来捕捉模型的不确定性。语义熵基于给定提示生成的多个回答之间的语义多样性,用于调节策略更新的幅度。该不确定性感知训练机制使策略更新能够根据问题的不确定性动态调整:在高不确定性问题上采取更保守的更新,而在高信心问题上保留原有学习信号。在五个数学推理基准上的实验结果表明,SEED-GRPO在平均准确率上达到新最优(AIME24 56.7,AMC 68.7,MATH 83.4,Minerva 34.2,OlympiadBench 48.0),验证了不确定性感知策略优化的有效性。
原文摘要 · Abstract (English)
Large language models (LLMs) exhibit varying levels of confidence across input prompts (questions): some lead to consistent, semantically similar answers, while others yield diverse or contradictory outputs. This variation reflects LLM's uncertainty about the input prompt, a signal of how confidently the model understands a given problem. However, vanilla Group Relative Policy Optimization (GRPO) treats all prompts equally during policy updates, ignoring this important information about the model's knowledge boundaries. To address this limitation, we propose SEED-GRPO (Semantic Entropy EnhanceD GRPO), which explicitly measures LLMs' uncertainty of the input prompts semantic entropy. Semantic entropy measures the diversity of meaning in multiple generated answers given a prompt and uses this to modulate the magnitude of policy updates. This uncertainty-aware training mechanism enables dynamic adjustment of policy update magnitudes based on question uncertainty. It allows more conservative updates on high-uncertainty questions while maintaining the original learning signal on confident ones. Experimental results on five mathematical reasoning benchmarks (AIME24 56.7, AMC 68.7, MATH 83.4, Minerva 34.2, and OlympiadBench 48.0) demonstrate that SEED-GRPO achieves new state-of-the-art performance in average accuracy, validating the effectiveness of uncertainty-aware policy optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。