新算法让强化学习更新更准，避免罕见错误动作干扰。
Delightful Policy Gradient
- 用优势与意外度乘积加门控，调节每条梯度贡献。
- 单场景下提升方向准确性，多场景下逼近最优监督梯度。
- 适合难任务、对噪声敏感的强化学习场景使用。
标准策略梯度仅用优势值加权采样动作，忽略动作在当前策略下的概率。这导致两个问题：单个决策场景中（如一张图像或提示），低概率但负优势的动作会过度扭曲更新方向；批量处理多个场景时，期望梯度会过度分配资源给策略已擅长的场景。本文提出「愉悦策略梯度」（Delightful Policy Gradient, DG），将每个梯度项通过优势与动作意外度（负对数概率）乘积的sigmoid函数进行门控。对于K臂老虎机问题，DG在单个场景中可证明提升方向精度，在多个场景下使期望梯度严格趋近于监督交叉熵最优解。该改进非单纯方差降低，即使样本无限也持续有效。实验表明，DG在MNIST、Transformer序列建模和连续控制任务中均优于REINFORCE、PPO及基线方法，尤其在困难任务上提升更显著。
原文摘要 · Abstract (English)
Standard policy gradients weight each sampled action by advantage alone, regardless of how likely that action was under the current policy. This creates two pathologies: within a single decision context (e.g. one image or prompt), a rare negative-advantage action can disproportionately distort the update direction; across many such contexts in a batch, the expected gradient over-allocates budget to contexts the policy already handles well. We introduce the \textit{Delightful Policy Gradient} (DG), which gates each term with a sigmoid of \emph{delight}, the product of advantage and action surprisal (negative log-probability). For $K$-armed bandits, DG provably improves directional accuracy in a single context and, across multiple contexts, shifts the expected gradient strictly closer to the supervised cross-entropy oracle. This second effect is not variance reduction: it persists even with infinite samples. Empirically, DG outperforms REINFORCE, PPO, and advantage-weighted baselines across MNIST, transformer sequence modeling, and continuous control, with larger gains on harder tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。