提出全梯度方法提升继承特征表示的稳定性和迁移效率。
Full-Gradient Successor Feature Representations
- 用全梯度优化继承特征，同时更新在线与目标网络参数
- 在离散和连续任务中均实现更优样本效率与迁移性能
- 适合需要稳定多任务学习的强化学习场景
继承特征（SF）结合广义策略改进（GPI）可将环境动态与奖励函数解耦，为强化学习中的迁移学习提供稳健框架。然而，标准的SF学习方法通常依赖半梯度时序差分（TD）更新。当使用非线性函数逼近时，半梯度方法缺乏可靠的收敛保证，易导致不稳定性，尤其在多任务设置中，准确的特征估计对有效GPI至关重要。受全梯度DQN启发，我们提出全梯度继承特征表示Q学习（FG-SFRQL），通过最小化完整的均方贝尔曼误差来优化继承特征。与标准方法不同，该方法同时计算在线网络和目标网络参数的梯度。我们提供了FG-SFRQL几乎必然收敛的理论证明，并在实验中验证：最小化完整残差可显著提升样本效率与迁移性能，优于半梯度基线方法，涵盖离散与连续域。
原文摘要 · Abstract (English)
Successor Features (SF) combined with Generalized Policy Improvement (GPI) provide a robust framework for transfer learning in Reinforcement Learning (RL) by decoupling environment dynamics from reward functions. However, standard SF learning methods typically rely on semi-gradient Temporal Difference (TD) updates. When combined with non-linear function approximation, semi-gradient methods lack robust convergence guarantees and can lead to instability, particularly in the multi-task setting where accurate feature estimation is critical for effective GPI. Inspired by Full Gradient DQN, we propose Full-Gradient Successor Feature Representations Q-Learning (FG-SFRQL), an algorithm that optimizes the successor features by minimizing the full Mean Squared Bellman Error. Unlike standard approaches, our method computes gradients with respect to parameters in both the online and target networks. We provide a theoretical proof of almost-sure convergence for FG-SFRQL and demonstrate empirically that minimizing the full residual leads to superior sample efficiency and transfer performance compared to semi-gradient baselines in both discrete and continuous domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。