自适应调整鲁棒性预算，让强化学习更稳定且抗干扰。
Distributionally Robust Self Paced Curriculum Reinforcement Learning
- 将鲁棒性预算作为动态课程，随智能体进展自动调整。
- 在多种扰动下平均提升11.8%的回合回报，性能比基准高1.9倍。
- 适合需要稳定部署的现实场景，如机器人控制与自动驾驶。
强化学习中，于受控环境训练的策略常在真实世界部署时因分布偏移而失效。分布鲁棒强化学习（DRRL）通过在由鲁棒性预算ε定义的不确定性集内优化最差情况性能来应对此问题。然而，固定ε会带来性能与鲁棒性的权衡：小ε导致高正常性能但弱鲁棒性，大ε可能引发训练不稳定或过于保守的策略。本文提出分布鲁棒自适应课程强化学习（DR-SPCRL），将ε视为连续课程，根据智能体进展自适应调度鲁棒性预算，实现正常性能与鲁棒性的平衡。实验表明，DR-SPCRL不仅稳定了训练过程，还在多种环境中实现了更优的性能-鲁棒性权衡：相较于固定或启发式调度策略，平均提升11.8%的回合回报；其性能约为对应常规强化学习算法的1.9倍。
原文摘要 · Abstract (English)
A central challenge in reinforcement learning is that policies trained in controlled environments often fail under distribution shifts at deployment into real-world environments. Distributionally Robust Reinforcement Learning (DRRL) addresses this by optimizing for worst-case performance within an uncertainty set defined by a robustness budget $ε$. However, fixing $ε$ results in a tradeoff between performance and robustness: small values yield high nominal performance but weak robustness, while large values can result in instability and overly conservative policies. We propose Distributionally Robust Self-Paced Curriculum Reinforcement Learning (DR-SPCRL), a method that overcomes this limitation by treating $ε$ as a continuous curriculum. DR-SPCRL adaptively schedules the robustness budget according to the agent's progress, enabling a balance between nominal and robust performance. Empirical results across multiple environments demonstrate that DR-SPCRL not only stabilizes training but also achieves a superior robustness-performance trade-off, yielding an average 11.8\% increase in episodic return under varying perturbations compared to fixed or heuristic scheduling strategies, and achieving approximately 1.9$\times$ the performance of the corresponding nominal RL algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。