针对复杂扰动场景，提升强化学习的安全性与鲁棒性。
TCRL: Temporal-Coupled Adversarial Training for Robust Constrained Reinforcement Learning in Worst-Case Scenarios
- 设计时序耦合对抗训练框架，无需建模攻击者即可评估最坏情况下的安全代价。
- 通过双重约束机制在保持奖励不可预测性的同时防御时序关联攻击。
- 在多种约束强化学习任务中表现优于现有方法，适合高安全需求场景。
约束强化学习（CRL）旨在约束条件下优化决策策略，广泛应用于自动驾驶、机器人和电网管理等安全关键领域。然而，现有鲁棒CRL方法主要关注单步扰动和时序独立的对抗模型，缺乏对时序耦合扰动的显式建模。为此，本文提出TCRL——一种面向最坏情况下的时序耦合对抗训练框架。首先，TCRL引入一种感知最坏情况的成本约束函数，无需显式建模对抗攻击者即可估计时序耦合扰动下的安全成本。其次，建立双约束防御机制，在对抗时序耦合攻击的同时保持奖励的不可预测性。实验表明，TCRL在多种CRL任务中均能持续优于现有方法，展现出对时序耦合扰动攻击更强的鲁棒性。
原文摘要 · Abstract (English)
Constrained Reinforcement Learning (CRL) aims to optimize decision-making policies under constraint conditions, making it highly applicable to safety-critical domains such as autonomous driving, robotics, and power grid management. However, existing robust CRL approaches predominantly focus on single-step perturbations and temporally independent adversarial models, lacking explicit modeling of robustness against temporally coupled perturbations. To tackle these challenges, we propose TCRL, a novel temporal-coupled adversarial training framework for robust constrained reinforcement learning (TCRL) in worst-case scenarios. First, TCRL introduces a worst-case-perceived cost constraint function that estimates safety costs under temporally coupled perturbations without the need to explicitly model adversarial attackers. Second, TCRL establishes a dual-constraint defense mechanism on the reward to counter temporally coupled adversaries while maintaining reward unpredictability. Experimental results demonstrate that TCRL consistently outperforms existing methods in terms of robustness against temporally coupled perturbation attacks across a variety of CRL tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。