通过能量场统一生成动作与逆强化学习，无需对抗训练即可恢复隐藏奖励。
Recovering Hidden Reward in Diffusion-Based Policies
- 用标量能量函数的梯度表示去噪场，统一生成建模与逆强化学习
- 在最大熵最优下，得分函数可恢复专家软Q函数梯度，实现无对抗奖励提取
- 结构约束提升泛化能力，适合需要可解释奖励信号的强化学习任务
本文提出EnergyFlow框架，将生成式动作建模与逆强化学习统一为参数化标量能量函数，其梯度即为去噪场。在最大熵最优性下，通过去噪得分匹配学习的得分函数可恢复专家软Q函数的梯度，实现无需对抗训练的奖励提取。我们证明，约束学习场为保守场可降低假设复杂度，并收紧分布外泛化边界。进一步分析了恢复奖励的可识别性，并界定了得分估计误差向动作偏好传播的范围。实验表明，EnergyFlow在多种操作任务上达到领先模仿学习性能，且提供的奖励信号优于对抗式逆强化学习和基于似然的方法。结果表明，奖励提取所需的结构约束同时作为有益归纳偏置，促进策略泛化。代码已开源：https://github.com/sotaagi/EnergyFlow。
原文摘要 · Abstract (English)
This paper introduces EnergyFlow, a framework that unifies generative action modeling with inverse reinforcement learning by parameterizing a scalar energy function whose gradient is the denoising field. We establish that under maximum-entropy optimality, the score function learned via denoising score matching recovers the gradient of the expert's soft Q-function, enabling reward extraction without adversarial training. Formally, we prove that constraining the learned field to be conservative reduces hypothesis complexity and tightens out-of-distribution generalization bounds. We further characterize the identifiability of recovered rewards and bound how score estimation errors propagate to action preferences. Empirically, EnergyFlow achieves state-of-the-art imitation performance on various manipulation tasks while providing an effective reward signal for downstream reinforcement learning that outperforms both adversarial IRL methods and likelihood-based alternatives. These results show that the structural constraints required for valid reward extraction simultaneously serve as beneficial inductive biases for policy generalization. The code is available at https://github.com/sotaagi/EnergyFlow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。