用熵引导权重提升RL微调大模型的探索效率
Entropy-guided sequence weighting for efficient exploration in RL-based LLM fine-tuning
- 根据奖励和不确定性动态加权生成序列
- 提升样本效率，增强推理能力
- 适合强化学习微调大模型的研究者
我们提出熵引导序列加权（EGSW），通过结合优势值与熵对生成输出进行动态加权，以改进基于强化学习的大语言模型微调中的探索-利用平衡。EGSW将熵正则化与基于优势的加权融合，实现策略更新的平衡，在高维状态空间中实现高效探索。采用温度缩放的softmax加权机制，优先选择高奖励、高不确定性的步骤，同时保持训练稳定性。尽管最初为提升组相对策略优化（GRPO）而设计，EGSW可泛化至其他强化学习算法，并适用于逐步或轨迹级设置。实验表明，EGSW显著提升GRPO的推理能力，改善样本效率。未来工作将探索其在更先进强化学习方法中的应用。
原文摘要 · Abstract (English)
We introduce Entropy-Guided Sequence Weighting (EGSW), a novel approach that enhances the exploration-exploitation tradeoff by dynamically assigning weights to generated outputs based on their advantage and entropy for Reinforcement Learning-based Large Language Model fine-tuning. EGSW integrates entropy regularization with advantage-based weighting to balance policy updates, enabling efficient exploration in high-dimensional state spaces. By employing temperature-scaled softmax weighting over sequences, EGSW prioritizing high-reward, high-uncertainty steps while maintaining training stability. Although originally developed to improve Group Relative Policy Optimization (GRPO) during large language model (LLM) fine-tuning, EGSW is generalizable to other reinforcement learning (RL) algorithms and can be implemented in both step-wise and trajectory-wise settings. Empirical evaluations demonstrate that EGSW enhances GRPO reasoning ability, yielding improvements in sample efficiency. Future work will explore the application of EGSW to advanced RL methodologies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。