将扩散模型的引导机制引入强化学习，实现无需重训练的测试时行为控制。
Policy Gradient Guidance Enables Test Time Control
- 在策略梯度中加入无条件分支，通过插值实现测试时控制。
- 在离散任务和小样本场景下提升样本效率与稳定性。
- 首次将引导机制扩展至经典策略梯度方法，适用于在线可控强化学习。
我们提出策略梯度引导（PGG），将扩散模型中的无分类器引导思想拓展至经典策略梯度方法。PGG在策略梯度中引入无条件分支，并对条件与无条件分支进行插值，形成无需重训练即可调节行为的测试时控制机制。理论推导表明，在优势估计下额外的归一化项会消失，从而获得干净的引导策略更新。实验在离散与连续控制基准上验证了PGG的有效性：在简单离散任务和低样本场景下，条件丢弃带来性能提升；但在连续控制中丢弃会引发不稳定性。适度增大引导强度（γ>1）可一致提升稳定性、样本效率与可控性。结果表明，引导机制可从扩散策略扩展至标准在线策略方法，为可控在线强化学习开辟新方向。
原文摘要 · Abstract (English)
We introduce Policy Gradient Guidance (PGG), a simple extension of classifier-free guidance from diffusion models to classical policy gradient methods. PGG augments the policy gradient with an unconditional branch and interpolates conditional and unconditional branches, yielding a test-time control knob that modulates behavior without retraining. We provide a theoretical derivation showing that the additional normalization term vanishes under advantage estimation, leading to a clean guided policy gradient update. Empirically, we evaluate PGG on discrete and continuous control benchmarks. We find that conditioning dropout-central to diffusion guidance-offers gains in simple discrete tasks and low sample regimes, but dropout destabilizes continuous control. Training with modestly larger guidance ($γ>1$) consistently improves stability, sample efficiency, and controllability. Our results show that guidance, previously confined to diffusion policies, can be adapted to standard on-policy methods, opening new directions for controllable online reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。