用继承状态预测提升DQN的值函数更新稳定性
Enhancing Q-Value Updates in Deep Q-Learning via Successor-State Prediction
- 引入继承状态分布建模环境动态，优化价值估计
- 在标准任务上提升学习效率与训练稳定性
- 适合关注值函数更新机制改进的研究者
深度Q网络（DQN）通过从回放缓冲区采样的转移数据学习未来回报。然而，DQN中的目标更新通常依赖于过去策略产生的下一状态，这些状态可能不具信息量，导致更新过程方差高。当采样转移与当前策略对齐度低时，该问题更为严重。为此，我们提出继承状态聚合深度Q网络（SADQ），显式使用随机转移模型建模环境动态，将继承状态分布融入Q值估计过程，实现更稳定且与策略一致的价值更新。同时，基于建模的转移结构探索更高效的动作选择策略。我们提供了理论保证，证明SADQ在保持无偏价值估计的同时降低训练方差。在标准强化学习基准和真实世界向量控制任务上的大量实验表明，SADQ在稳定性和学习效率方面持续优于DQN变体。
原文摘要 · Abstract (English)
Deep Q-Networks (DQNs) estimate future returns by learning from transitions sampled from a replay buffer. However, the target updates in DQN often rely on next states generated by actions from past, potentially suboptimal, policy. As a result, these states may not provide informative learning signals, causing high variance into the update process. This issue is exacerbated when the sampled transitions are poorly aligned with the agent's current policy. To address this limitation, we propose the Successor-state Aggregation Deep Q-Network (SADQ), which explicitly models environment dynamics using a stochastic transition model. SADQ integrates successor-state distributions into the Q-value estimation process, enabling more stable and policy-aligned value updates. Additionally, it explores a more efficient action selection strategy with the modeled transition structure. We provide theoretical guarantees that SADQ maintains unbiased value estimates while reducing training variance. Our extensive empirical results across standard RL benchmarks and real-world vector-based control tasks demonstrate that SADQ consistently outperforms DQN variants in both stability and learning efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。