用时序模型增强PPO,让机器人在传感器失效时仍能稳定控制
When Sensors Fail: Temporal Sequence Models for Robust PPO under Sensor Drift
- 用Transformer和状态空间模型从历史数据推断缺失感知信息
- 传感器丢失超70%时,新方法仍保持90%以上原始性能
- 适合部署在真实环境中的机器人控制,尤其传感器易漂移的场景
现实世界的强化学习系统必须应对观测流中的分布漂移,但多数策略架构假设状态完全可观测且无噪声。本文研究近端策略优化(PPO)在具有时间持续性的传感器故障下的鲁棒性,这类故障导致部分可观测与表征偏移。为应对漂移,我们引入时序序列模型(包括Transformer和状态空间模型,SSM)增强PPO,使策略能基于历史信息推断缺失状态,维持性能。在随机传感器故障下,我们证明了无限时域奖励退化具有高概率上界,量化了鲁棒性与策略平滑性及故障持续时间的关系。在MuJoCo连续控制基准测试中,当传感器丢包严重时,基于Transformer的序列策略显著优于MLP、RNN和SSM基线,在传感器丢失超过70%的情况下仍保持接近原始性能的90%以上回报。
原文摘要 · Abstract (English)
Real-world reinforcement learning systems must operate under distributional drift in their observation streams, yet most policy architectures implicitly assume fully observed and noise-free states. We study robustness of Proximal Policy Optimization (PPO) under temporally persistent sensor failures that induce partial observability and representation shift. To respond to this drift, we augment PPO with temporal sequence models, including Transformers and State Space Models (SSMs), to enable policies to infer missing information from history and maintain performance. Under a stochastic sensor failure process, we prove a high-probability bound on infinite-horizon reward degradation that quantifies how robustness depends on policy smoothness and failure persistence. Empirically, on MuJoCo continuous-control benchmarks with severe sensor dropout, we show Transformer-based sequence policies substantially outperform MLP, RNN, and SSM baselines in robustness, maintaining high returns even when large fractions of sensors are unavailable. These results demonstrate that temporal sequence reasoning provides a principled and practical mechanism for reliable operation under observation drift caused by sensor unreliability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。