arXiv:2603.27346cs.ROcs.AI2026-03中稿 · IEEE 11th Internat…

针对机器人操作强化学习的训练不稳问题,提出双流自适应经验回放机制。

D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay for Stable Reinforcement Learning in Robotic Manipulation

  • 分离演员与评论家的经验采样,分别采用低误差和优先级策略
  • 在真实任务中实现更优性能与稳定训练,最高提升23%成功率
  • 适合需要长期稳定性的机器人控制场景

机器人操作因接触动力学复杂、长时序依赖和训练不稳定性而具挑战性。尽管SAC和TD3等离策略算法在仿真中表现良好,但在真实场景常出现策略振荡和性能崩溃,部分原因在于经验回放策略未区分演员与评论家的数据需求。我们提出D-SPEAR:双流优先自适应经验回放框架,将演员与评论家的采样解耦,共享一个回放缓冲区。评论家使用优先级回放以高效学习价值函数,而演员则通过低误差转移更新以稳定策略优化。自适应锚定机制根据时序差分误差的变异系数动态平衡均匀与优先采样,基于Huber的评论家目标函数进一步提升了异构奖励尺度下的鲁棒性。我们在robosuite基准的Block-Lifting和Door-Opening等复杂任务上评估D-SPEAR,结果表明其持续优于SAC、TD3和DDPG等强基线,在最终性能与训练稳定性上均有提升,消融实验验证了双流回放机制的互补作用。

原文摘要 · Abstract (English)

Robotic manipulation remains challenging for reinforcement learning due to contact-rich dynamics, long horizons, and training instability. Although off-policy actor-critic algorithms such as SAC and TD3 perform well in simulation, they often suffer from policy oscillations and performance collapse in realistic settings, partly due to experience replay strategies that ignore the differing data requirements of the actor and the critic. We propose D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay, a replay framework that decouples actor and critic sampling while maintaining a shared replay buffer. The critic leverages prioritized replay for efficient value learning, whereas the actor is updated using low-error transitions to stabilize policy optimization. An adaptive anchor mechanism balances uniform and prioritized sampling based on the coefficient of variation of TD errors, and a Huber-based critic objective further improves robustness under heterogeneous reward scales. We evaluate D-SPEAR on challenging robotic manipulation tasks from the robosuite benchmark, including Block-Lifting and Door-Opening. Results demonstrate that D-SPEAR consistently outperforms strong off-policy baselines, including SAC, TD3, and DDPG, in both final performance and training stability, with ablation studies confirming the complementary roles of the actorside and critic-side replay streams.

强化学习机器人控制经验回放稳定训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。