根据词元熵动态调整优化策略,提升大模型强化学习的精细控制能力。
Heterogeneous Adaptive Policy Optimization: Tailoring Optimization to Every Token's Nature
- 基于词元级别熵实时调节采样温度与梯度剪裁,实现细粒度优化
- 在多任务测试中,数学推理、代码生成等任务上优于现有方法
- 适合需要精细化强化学习优化的LLM应用开发者
利用熵作为异质性度量来指导大语言模型的强化学习优化已成为关键研究方向。然而,现有方法通常将熵视为离散过滤器或事后调节器,而非核心优化驱动因素。为充分挖掘熵的潜力并实现细粒度调控,我们提出异质自适应策略优化(HAPO),一种基于词元感知的算法,可在整个训练过程中持续依据词元级熵动态调整优化行为。该算法包含四个关键组件:(1) 自适应温度采样,实时调整采样温度,在高熵词元处促进探索;(2) 词元级组平均优势估计,考虑序列长度影响的同时保持无偏处理;(3) 差异化优势重分配,利用熵与重要性比率调整信号明确词元的优势值;(4) 非对称自适应裁剪,根据词元级熵动态调整裁剪边界。通过系统性研究熵的作用,我们将词元级处理嵌入每个训练阶段。在多个模型上针对数学推理、代码生成和逻辑任务的大量实验表明,HAPO 持续优于 DAPO。代码见 https://github.com/starriver030515/HAPO。
原文摘要 · Abstract (English)
Using entropy as a measure of heterogeneity to guide optimization has emerged as a crucial research direction in Reinforcement Learning for LLMs. However, existing methods typically treat it as a discrete filter or post-hoc regulator rather than a core optimization driver. To fully leverage the potential of entropy and achieve fine-grained regulation, we introduce Heterogeneous Adaptive Policy Optimization (HAPO), a token-aware algorithm that continuously adapts optimization dynamics based on token-level entropy throughout the entire training process. Our algorithm includes four key components: (1) Adaptive Temperature Sampling that adjusts sampling temperature in real time, promoting exploration at high-entropy tokens. (2) Token-Level Group Average Advantage Estimation that estimates advantages at token level, accounting for sequence-length effects while preserving non-biased treatment.(3) Differential Advantage Redistribution that leverages entropy and importance ratios to adjust advantages for tokens with clear signals. (4) Asymmetric Adaptive Clipping that adynamically adjusts clipping boundaries based on token-level entropy. Through systematic investigation of entropy, we embed token-level treatment into every stage. Extensive experiments on mathematical reasoning, code, and logic tasks across multiple models demonstrate HAPO's consistent superiority over DAPO. Our code can be found in https://github.com/starriver030515/HAPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。