针对大模型强化学习中任务熵差异问题,提出分组熵控策略优化方法。
Group Entropy-Controlled Policy Optimization

- 按任务分组动态调节优势函数,实现差异化探索控制
- 在13个基准上优于GRPO和最新熵控方法,跨任务性能均衡提升
- 适合需要多任务平衡探索与利用的复杂指令对齐场景
熵控制已成为大语言模型强化学习中平衡探索与利用的有效工具。然而,当前方法常在异构任务混合环境下运行,同一策略下不同任务组的熵水平差异显著,导致全局或标记级熵调控无法满足各任务的探索需求。这种异质性进一步使GRPO风格的归一化优势产生依赖熵的偏差,导致不同提示组之间的优势信号在统计上不可比较。为此,本文提出分组熵控策略优化(GEPO),作为GRPO的轻量级扩展,利用现有分组样本估计的组熵,实现熵条件下的非对称优势调整。GEPO通过自适应阈值,在低熵组减弱正优势以减少过开发,在高熵组增强负优势以保留探索能力。在两个基础模型上,覆盖数学、物理、科学、代码生成和指令遵循等领域的十三个基准上进行的大量实验表明,GEPO持续优于GRPO及近期熵控方法,在训练全程保持任务特定的探索水平的同时实现跨任务性能的均衡提升。
原文摘要 · Abstract (English)
Entropy control has become an effective tool in reinforcement learning (RL) of large language models (LLMs), helping balance exploration-exploitation trade-off during alignment process. Such RL paradigm is often conducted on mixtures of heterogeneous tasks, which induce distinct entropy regimes under the same policy, making global or token-level entropy regulation insufficient to corresponding heterogeneous needs of exploration. This heterogeneity further makes GRPO-style normalized advantages induce an entropy-dependent bias, making advantage signals across prompt groups statistically non-comparable. To address this issue, we propose Group Entropy-Controlled Policy Optimization (GEPO), a lightweight extension to GRPO that uses group entropy, estimated from existing grouped samples to perform entropy-conditioned asymmetric advantage shaping. GEPO attenuates positive advantages in low-entropy groups to reduce over-exploitation, and negative advantages in high-entropy groups to preserve exploration, with adaptive thresholds derived from historical entropy statistics. Extensive experiments on two base models across thirteen benchmarks spanning mathematics, physics, science, code generation, and instruction following show that GEPO consistently outperforms GRPO and recent entropy-controlled methods, delivering balanced cross-task improvements while preserving task-specific exploration levels throughout training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。