通过互信息学习提升大模型多尝试下的回答多样性与正确率。
UpSkill: Mutual Information Skill Learning for Structured Response Diversity in LLMs
- 用互信息奖励机制鼓励模型在多次尝试中生成更具体的解题路径。
- 在GSM8K数据集上，强基座模型的pass@k平均提升约3%，pass@1不降。
- 适合关注模型探索能力与多样化推理策略的研究者。
强化学习结合可验证奖励（RLVR）提升了大语言模型在数学与编程任务上的推理能力，但传统单次尝试准确率优化方法会抑制重复尝试中的回答多样性，限制探索并忽略未充分覆盖的解题策略。本文提出UpSkill，一种基于互信息技能学习（MISL）的训练阶段方法，用于优化pass@k准确率。我们设计了一种新的奖励机制，并集成到组相对策略优化（GRPO）中：在词元级别引入互信息（MI）奖励，以增强轨迹对隐变量z的特异性。在GSM8K数据集上，使用Llama 3.1-8B、Qwen 2.5-7B和R1-Distilled-Qwen2.5-Math-1.5B三个开源模型进行实验，结果表明，UpSkill在更强基座模型上有效提升了多尝试指标，使Qwen和Llama的pass@k平均提升约3%，且不损害pass@1性能。同时，我们提供了实证与理论证据，证明pass@k的提升与互信息目标密切相关。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning abilities of large language models (LLMs) on mathematics and programming tasks, but standard approaches that optimize single-attempt accuracy can inadvertently suppress response diversity across repeated attempts, narrowing exploration and overlooking underrepresented strategies. We introduce UpSkill, a training time method that adapts Mutual Information Skill Learning (MISL) to LLMs for optimizing pass@k correctness. We propose a novel reward that we implement within Group Relative Policy Optimization (GRPO): a token-level mutual information (MI) reward that encourages trajectory specificity to z. Experiments on GSM8K with three open-weight models, Llama 3.1-8B, Qwen 2.5-7B, and R1-Distilled-Qwen2.5-Math-1.5B, show that UpSkill improves multi-attempt metrics on the stronger base models, yielding mean gains of ~3% in pass@k for both Qwen and Llama without degrading pass@1. Additionally, we find both empirical and theoretical evidence that improvements in pass@k are closely tied to the mutual information objective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。