arXiv:2606.03521cs.LGcs.AI2026-06

不重新训练模型,推理时提升强化学习的抗干扰能力。

Post-Hoc Robustness for Model-Based Reinforcement Learning

论文配图:Post-Hoc Robustness for Model-Based Reinforcement Learning
图 1 · 摘自论文原文
  • 用学习到的环境模型和原策略,在推理时通过对抗模拟优化策略。
  • 在MuJoCo环境中使鲁棒性显著提升,且无需额外训练神经网络。
  • 适合部署时需快速增强鲁棒性的实际应用,尤其计算资源有限场景。

为提升强化学习在现实世界的适用性,对抗鲁棒强化学习研究如何在对抗性环境扰动下训练智能体。在此设定中,主体智能体在对手制造的环境扰动下优化策略,形成零和马尔可夫博弈。当将对抗鲁棒强化学习与基于模型的强化学习结合时,对手可针对学习到的转移模型而非训练环境进行攻击。本文提出一种推理时的后处理鲁棒化方法:利用已学习的模型与训练好的正常策略,执行一次鲁棒策略改进步骤,无需重新训练神经网络。具体地,采用在有界不确定性集内通过投影梯度下降近似对抗轨迹的模型预测控制。此外,该方法在离线滚动过程中考虑并缓解了分布外问题。实验验证了在受扰的Gymnasium MuJoCo环境中,该方法显著提升了鲁棒性,同时满足推理阶段的计算限制。

原文摘要 · Abstract (English)

To improve the real-world applicability of reinforcement learning (RL), the field of adversarially robust RL studies how to train agents under adversarial environment perturbations. In this setting, a protagonist agent optimizes a policy under environmental perturbations from an adversary, resulting in a zero-sum Markov game. When adversarially robust RL is combined with model-based RL, the adversary can target a learned transition model instead of the training environment. Extending this idea, this work introduces post-hoc robustification of deep RL agents at inference time. By using the learned model in combination with a trained nominal policy, our approach performs a robust policy improvement step. The goal is to improve robustness without any additional training of neural networks. Specifically, we utilize model-predictive control under adversarial rollouts, which are approximated via projected gradient descent within a bounded uncertainty set. Furthermore, these offline rollouts are performed while considering and mitigating out-of-distribution issues. The proposed methodology is validated by demonstrating significant improvements in robustness when the algorithm is evaluated in perturbed Gymnasium MuJoCo environments, while considering the computational limitations of the post-hoc inference setting.

强化学习鲁棒性模型预测推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。