用分层继承特征近似提升强化学习安全性，兼顾多目标与实时保护。
Stacked Universal Successor Feature Approximators for Safety in Reinforcement Learning
- 构建分层连续控制的通用继承特征近似框架，适配SAC算法
- 在复杂任务中实现比基线更高的次级目标表现，且安全控制器有效干预
- 适合高风险场景下需多重保障的强化学习系统设计
现实问题常涉及复杂的多目标结构，难以简化为单一目标的强化学习环境。需平衡操作成本与多维任务表现，并考虑终态对后续可用性的影响，同时确保环境中其他智能体及强化学习代理自身的安全性。通过引入二级备用控制器实现系统冗余，已被证明是高风险场景下保障安全的有效手段。本文研究一种适用于连续控制的分层通用继承特征近似（USFA）变体，适配软演员-评论家（SAC）算法，并集成一组二级安全控制器，称为用于安全的分层USFA（SUSFAS）。实验表明，该方法在引入二级控制器（如运行时保证控制器，RTA）的情况下，相比SAC基线，在次级目标性能上显著提升。
原文摘要 · Abstract (English)
Real-world problems often involve complex objective structures that resist distillation into reinforcement learning environments with a single objective. Operation costs must be balanced with multi-dimensional task performance and end-states' effects on future availability, all while ensuring safety for other agents in the environment and the reinforcement learning agent itself. System redundancy through secondary backup controllers has proven to be an effective method to ensure safety in real-world applications where the risk of violating constraints is extremely high. In this work, we investigate the utility of a stacked, continuous-control variation of universal successor feature approximation (USFA) adapted for soft actor-critic (SAC) and coupled with a suite of secondary safety controllers, which we call stacked USFA for safety (SUSFAS). Our method improves performance on secondary objectives compared to SAC baselines using an intervening secondary controller such as a runtime assurance (RTA) controller.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。