arXiv:2604.14646cs.AI2026-04ACL被引 1

提出统一熵控框架,解决强化学习中探索不足与训练不稳问题。

Targeted Exploration via Unified Entropy Control for Reinforcement Learning

论文配图:Targeted Exploration via Unified Entropy Control for Reinforcement Learning
图 1 · 摘自论文原文
  • 通过动态调节熵,针对性增强难样本的探索能力。
  • 在Geometry3K上相比GRPO提升37.9%相对性能。
  • 适合需要稳定训练的大模型推理任务研究者。

近期强化学习进展提升了大语言模型(LLM)和视觉-语言模型(VLM)的推理能力。然而,广泛使用的组相对策略优化(GRPO)常出现熵坍塌,导致策略过早收敛、多样性丢失。现有探索方法引入额外偏差或方差,难以维持优化稳定性。本文提出统一熵控强化学习框架(UEC-RL),提供针对性探索与稳定机制。该框架在困难提示上激活更多探索,以寻找潜在且有价值的推理路径;同时,稳定器防止熵无序增长,保障模型在固化可靠行为过程中训练稳定。两者协同,在需要时扩展搜索空间,同时保持全程优化鲁棒性。在LLM与VLM推理任务上的实验表明,UEC-RL在Pass@1和Pass@$k$上均优于主流强化学习基线。在Geometry3K数据集上,相较GRPO实现37.9%的相对性能提升,证明其能在不牺牲收敛性的前提下持续有效探索,是推动大模型强化学习推理的关键技术。代码已开源。

原文摘要 · Abstract (English)

Recent advances in reinforcement learning (RL) have improved the reasoning capabilities of large language models (LLMs) and vision-language models (VLMs). However, the widely used Group Relative Policy Optimization (GRPO) consistently suffers from entropy collapse, causing the policy to converge prematurely and lose diversity. Existing exploration methods introduce additional bias or variance during exploration, making it difficult to maintain optimization stability. We propose Unified Entropy Control for Reinforcement Learning (UEC-RL), a framework that provides targeted mechanisms for exploration and stabilization. UEC-RL activates more exploration on difficult prompts to search for potential and valuable reasoning trajectories. In parallel, a stabilizer prevents entropy from growing uncontrollably, thereby keeping training stable as the model consolidates reliable behaviors. Together, these components expand the search space when needed while maintaining robust optimization throughout training. Experiments on both LLM and VLM reasoning tasks show consistent gains over RL baselines on both Pass@1 and Pass@$k$. On Geometry3K, UEC-RL achieves a 37.9\% relative improvement over GRPO, indicating that it sustains effective exploration without compromising convergence and underscoring UEC-RL as a key for scaling RL-based reasoning in large models. Our code is available at https://github.com/597358816/UEC-RL.

强化学习大模型推理探索机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。