通过融合历史参数提升DQN目标网络稳定性,改善长期学习表现。
Memory Merge DQN: Sensitivity Weighted Target Updates for Stable Value Learning

- 用敏感度加权融合近期网络参数,替代传统硬拷贝更新
- 在多个Atari游戏上超越DQN等基线方法,部分游戏性能提升显著
- 适合关注值函数结构保持与长期训练稳定性的强化学习研究者
深度Q网络使用目标网络来稳定自举式值函数学习,但标准的硬拷贝更新引入了权衡。固定目标网络虽提升短期稳定性,但每次硬更新会突然用最新在线网络替换目标参数,丢弃近期参数历史,可能引发目标值突变,并移除后续训练仍有价值的值函数结构。本文提出Memory Merge DQN,一种目标网络更新机制,保留近期在线网络副本的短期记忆,并基于Q值敏感度而非仅复制最新网络,融合参数构建目标网络。该方法赋予对当前Q值行为仍具局部重要性的参数更大权重,同时以时效性先验确保融合后目标接近最新在线参数。该方法受费希尔权重合并启发,但采用Q值敏感度作为加权信号。在Atari环境上对比DQN、平均DQN、带层归一化的DQN及带有梯度裁剪的PQN,结果表明Memory Merge DQN具有高度竞争力,在所评估方法中获得最多的最终性能第一名,优于DQN、平均DQN和PQN,且在部分游戏中取得显著提升。结果表明,选择性地融合近期权重与历史信息可增强DQN代理的稳定性与最终性能,目标网络设计是长期值学习中保持有用值函数结构的关键机制。
原文摘要 · Abstract (English)
Deep Q-networks use target networks to stabilise bootstrapped value learning, but the standard hard copy update also introduces a tradeoff. Holding the target network fixed, improves short term stability, yet each hard update abruptly replaces the target parameters with the newest online network and discards recent parameter history. This can produce sudden changes in the bootstrap target and may remove value function structure that remains useful later in training. This paper introduces Memory Merge DQN, a target network update mechanism that maintains a short memory of recent historical online network copies and constructs the target network by merging network parameters based on the Q-value sensitivity rather than copying only the newest online network. Memory Merge gives greater influence to parameters that remain locally important for current Q-value behaviour, while using a recency prior to keep the merged target close to the latest online parameters. The method is inspired by Fisher Weight Model Merging, but uses Q-value sensitivity rather than Fisher information as the weighting signal. This paper evaluates Memory Merge DQN on Atari environments against DQN, Averaged DQN, DQN with layer normalisation, and PQN (with gradient clipping). The results show that Memory Merge DQN is highly competitive and it achieves the largest number of first place final performance results among the evaluated methods, beats DQN, Averaged DQN, and PQN (with gradient clipping), and produces substantial gains in several games where preserving useful value-function parameters appears beneficial. These findings suggest that selectively merging recent parameter weights and history can improve the stability and final performance of DQN agents, and that target network design is an important mechanism for preserving useful value function structure during long horizon value learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。