让强化学习的值函数同时匹配目标值和梯度，提升训练稳定性和收敛速度。
First-order Sobolev Reinforcement Learning
- 通过一阶贝尔曼一致性约束值函数与目标函数在值和梯度上同步匹配。
- 在DDPG、SAC等算法中使用时，可加速评论家收敛并提升策略梯度稳定性。
- 无需改变原有框架，即可实现更平滑的值函数几何对齐，适合追求高效训练的从业者。
我们提出一种时序差分学习的改进方法，强制要求学习到的值函数在状态和动作上不仅匹配贝尔曼目标的值，还匹配其导数。通过可微分动力学对贝尔曼备份进行微分，获得解析一致的梯度目标。将这些目标引入评论家目标函数，采用类似Sobolev的损失，促使评论家同时对齐目标函数的值和局部几何结构。该一阶TD匹配原则可无缝集成至现有算法（如Q-learning或演员-评论家方法，例如DDPG、SAC），有望在不改变整体结构的前提下，加快评论家收敛速度，并提升策略梯度的稳定性。
原文摘要 · Abstract (English)
We propose a refinement of temporal-difference learning that enforces first-order Bellman consistency: the learned value function is trained to match not only the Bellman targets in value but also their derivatives with respect to states and actions. By differentiating the Bellman backup through differentiable dynamics, we obtain analytically consistent gradient targets. Incorporating these into the critic objective using a Sobolev-type loss encourages the critic to align with both the value and local geometry of the target function. This first-order TD matching principle can be seamlessly integrated into existing algorithms, such as Q-learning or actor-critic methods (e.g., DDPG, SAC), potentially leading to faster critic convergence and more stable policy gradients without altering their overall structure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。