动态分配采样预算，让大模型在强化学习中更高效探索
Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs
- 按题目难易动态分配采样次数，避免简单题浪费资源
- 自适应调整温度保持熵稳定，防止模型过早收敛
- 既提升回答准确率，又保留发现新路径的探索能力
推理型大语言模型（LLMs）在复杂任务中表现优异，促使研究者关注其强化学习（RL）训练。然而，现有方法对所有问题均分配相同数量的采样轨迹，效率低下：简单问题训练收益小，而难题需更多采样才能获得正确答案。此外，尽管强化学习提升了响应精度，却削弱了模型的探索能力，可能导致性能低于预强化学习的基线模型。为此，本文提出基于问题难度动态分配采样预算的机制，实现更高效的强化学习训练。同时引入自适应动态温度调整策略，维持熵水平稳定，以保障充分探索。该方法使大模型在提升回答准确性的同时，仍能有效探索潜在正确路径。代码与数据已公开于：https://github.com/LiaoMengqi/E3-RL4LLMs
原文摘要 · Abstract (English)
Reasoning large language models (LLMs) excel in complex tasks, which has drawn significant attention to reinforcement learning (RL) for LLMs. However, existing approaches allocate an equal number of rollouts to all questions during the RL process, which is inefficient. This inefficiency stems from the fact that training on simple questions yields limited gains, whereas more rollouts are needed for challenging questions to sample correct answers. Furthermore, while RL improves response precision, it limits the model's exploration ability, potentially resulting in a performance cap below that of the base model prior to RL. To address these issues, we propose a mechanism for dynamically allocating rollout budgets based on the difficulty of the problems, enabling more efficient RL training. Additionally, we introduce an adaptive dynamic temperature adjustment strategy to maintain the entropy at a stable level, thereby encouraging sufficient exploration. This enables LLMs to improve response precision while preserving their exploratory ability to uncover potential correct pathways. The code and data is available on: https://github.com/LiaoMengqi/E3-RL4LLMs
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。