为多智能体系统设计可扩展的安全防护机制，提升强化学习安全性与效率
Compositional Shielding and Reinforcement Learning for Multi-Agent Systems
- 为每个智能体独立计算局部防护，通过假设-保证推理将全局安全规范分解
- 在两个案例中将计算时间从小时级降至秒级，加速学习收敛
- 适合需要高安全性的多智能体场景，如自动驾驶、机器人协作
深度强化学习已成为获取高性能策略的强大工具，但其安全性始终是长期难题。一种有前景的保障安全方法是使用防护机制（shield），阻止策略执行不安全动作。然而，传统防护的计算复杂度随状态变量数量呈指数增长，在拥有多个智能体的系统中尤为严重。本文提出一种新型多智能体防护方法：通过为每个智能体分别计算独立防护来解决可扩展性问题。挑战在于典型安全规范为全局性质，而个体防护仅能保证局部性质。本文的关键突破是引入假设-保证推理，提出一个可靠证明规则，将（全局、复杂）安全规范分解为（局部、简单）防护责任。此外，实验表明在强化学习过程中应用防护机制能显著提升给定训练预算下的策略质量。我们在两个案例中验证了该框架的有效性与可扩展性，将计算时间从数小时缩短至数秒，并实现快速学习收敛。
原文摘要 · Abstract (English)
Deep reinforcement learning has emerged as a powerful tool for obtaining high-performance policies. However, the safety of these policies has been a long-standing issue. One promising paradigm to guarantee safety is a shield, which shields a policy from making unsafe actions. However, computing a shield scales exponentially in the number of state variables. This is a particular concern in multi-agent systems with many agents. In this work, we propose a novel approach for multi-agent shielding. We address scalability by computing individual shields for each agent. The challenge is that typical safety specifications are global properties, but the shields of individual agents only ensure local properties. Our key to overcome this challenge is to apply assume-guarantee reasoning. Specifically, we present a sound proof rule that decomposes a (global, complex) safety specification into (local, simple) obligations for the shields of the individual agents. Moreover, we show that applying the shields during reinforcement learning significantly improves the quality of the policies obtained for a given training budget. We demonstrate the effectiveness and scalability of our multi-agent shielding framework in two case studies, reducing the computation time from hours to seconds and achieving fast learning convergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。