用世界模型监测异常，让强化学习控制器安全运行。
World Models for Anomaly Detection during Model-Based Reinforcement Learning Inference
- 推理时持续比对模型预测与实际行为差异
- 可识别局部几何与重力变化等异常
- 无需任务知识，适合各类部署场景
基于学习的控制器常因安全与可靠性担忧而无法投入真实应用。本文探索在基于模型的强化学习中，如何将先进的世界模型用于训练之外，在部署阶段确保策略仅在充分熟悉的状态空间内运行。通过在推理过程中持续监测世界模型预测与系统实际行为之间的偏差，一旦超过阈值即可触发紧急停止等措施。该方法无需任务特定知识，具有普遍适用性。仿真实验在典型机器人控制任务中验证了其有效性，可识别局部机器人几何变化和全局重力大小变化。真实世界实验使用敏捷四旋翼飞行器进一步证明，该方法能检测到作用于机体的意外外力。结果表明，即使在新环境或恶劣条件下，也能实现原本不可预测的学习型控制器的安全可靠运行。
原文摘要 · Abstract (English)
Learning-based controllers are often purposefully kept out of real-world applications due to concerns about their safety and reliability. We explore how state-of-the-art world models in Model-Based Reinforcement Learning can be utilized beyond the training phase to ensure a deployed policy only operates within regions of the state-space it is sufficiently familiar with. This is achieved by continuously monitoring discrepancies between a world model's predictions and observed system behavior during inference. It allows for triggering appropriate measures, such as an emergency stop, once an error threshold is surpassed. This does not require any task-specific knowledge and is thus universally applicable. Simulated experiments on established robot control tasks show the effectiveness of this method, recognizing changes in local robot geometry and global gravitational magnitude. Real-world experiments using an agile quadcopter further demonstrate the benefits of this approach by detecting unexpected forces acting on the vehicle. These results indicate how even in new and adverse conditions, safe and reliable operation of otherwise unpredictable learning-based controllers can be achieved.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。