提出新方法应对感知延迟，让智能体在延迟观测下仍能稳定表现。
Model-Based Reinforcement Learning under Random Observation Delays
- 基于模型构建过滤机制，动态更新信念状态以处理乱序观测。
- 在模拟机器人任务中显著优于传统延迟处理方法，且对延迟分布变化鲁棒。
- 适合真实场景中存在随机感知延迟的强化学习应用。
现实环境中常出现传感器延迟，但标准强化学习算法通常假设环境感知是即时的。本文研究了部分可观测马尔可夫决策过程（POMDP）中的随机观测延迟问题，其中观测可能乱序到达，这是此前强化学习未被充分探索的场景。我们分析了此类延迟的结构特性，发现简单的过去观测堆叠策略无法保证可靠性能。为此，提出一种基于模型的过滤流程，根据持续到达的观测流逐步更新信念状态。进一步设计了一种延迟感知的框架，将该思想融入基于模型的强化学习，使智能体能够有效应对随机延迟。将该框架应用于Dreamer世界模型方案，在多个任务中持续优于针对MDP设计的延迟感知基线，并在部署时表现出对延迟分布变化的鲁棒性。此外，在模拟机器人任务上的实验对比了常见启发式方法，强调了显式建模观测延迟的重要性。
原文摘要 · Abstract (English)
Delays frequently occur in real-world environments, yet standard reinforcement learning (RL) algorithms often assume instantaneous perception of the environment. We study random sensor delays in POMDPs, where observations may arrive out-of-sequence, a setting that has not been previously addressed in RL. We analyze the structure of such delays and demonstrate that naive approaches, such as stacking past observations, are insufficient for reliable performance. To address this, we propose a model-based filtering process that sequentially updates the belief state based on an incoming stream of observations. We then introduce a simple delay-aware framework that incorporates this idea into model-based RL, enabling agents to effectively handle random delays. Applying this framework to the Dreamer world-modeling scheme, our method consistently outperforms delay-aware baselines developed for MDPs and demonstrates robustness to delay distribution shifts during deployment. Additionally, we present experiments on simulated robotic tasks, comparing our method to common practical heuristics and emphasizing the importance of explicitly modeling observation delays.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。