用比例积分控制稳定大模型训练中的熵，防止探索失效。
EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control
- 通过动态调节正负样本损失系数来稳定熵
- 实验显示可维持目标熵水平，避免训练崩溃
- 适合需要长期稳定训练的大语言模型研究者
大语言模型的长期训练需要保持稳定的探索能力，以防止模型陷入次优行为。熵在此过程中至关重要，它控制探索并避免过早收敛到次优解。然而，现有强化学习方法难以维持适当的熵水平，因为训练过程同时包含正负样本，它们对熵的影响在不同步骤中各异。为此，我们提出基于比例-积分控制的熵稳定方法（EntroPIC），通过自适应调整正负样本的影响，动态调节其损失系数，从而在训练全程稳定熵，确保高效探索与持续进展。我们提供了针对在线与离线学习设置的完整理论分析，证明该方法在大规模大模型训练中有效控制熵。实验结果表明，该方法能成功维持期望的熵水平，实现大语言模型的稳定且最优的强化学习训练。
原文摘要 · Abstract (English)
Long-term training of large language models (LLMs) requires maintaining stable exploration to prevent the model from collapsing into sub-optimal behaviors. Entropy is crucial in this context, as it controls exploration and helps avoid premature convergence to sub-optimal solutions. However, existing reinforcement learning methods struggle to maintain an appropriate level of entropy, as the training process involves a mix of positive and negative samples, each affecting entropy in different ways across steps. To address this, we propose Entropy stabilization via Proportional-Integral Control (EntroPIC), a novel method that adaptively adjusts the influence of positive and negative samples by dynamically tuning their loss coefficients. This approach stabilizes entropy throughout training, ensuring efficient exploration and steady progress. We provide a comprehensive theoretical analysis for both on-policy and off-policy learning settings, demonstrating that EntroPIC is effective at controlling entropy in large-scale LLM training. Experimental results show that our method successfully maintains desired entropy levels, enabling stable and optimal RL training for LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。