为基于RNN的强化学习策略提供概率化验证，估计异常行为发生概率。
Probabilistic Verification of Recurrent Neural Networks for Single and Multi-Agent Reinforcement Learning
- 通过策略驱动采样估算可行隐藏状态集
- 给出高置信度的异常行为概率上界
- 适用于单智能体与多智能体场景
基于循环神经网络（RNN）的历史依赖策略依赖于潜在隐状态动态，使得在部分可观测强化学习中验证变得困难。现有RNN验证工具通常依赖严格建模假设或对隐状态空间的粗略上界近似，导致结果过于保守或无法得出结论。本文提出一种概率框架RNN-ProVe，通过估算不期望行为的发生概率来实现验证。RNN-ProVe利用策略驱动采样近似训练策略下可行的隐状态集合，并推导统计误差边界，从而获得有界误差、高置信度的行为违规估计。在部分可观测的单智能体与协作多智能体任务上的实验表明，相比现有工具，RNN-ProVe能提供更定量、更注重可行性概率保证，同时可扩展至循环与多智能体设置。
原文摘要 · Abstract (English)
History-dependent policies induced by recurrent neural networks (RNNs) rely on latent hidden state dynamics, making verification in partially observable reinforcement learning (RL) challenging. Existing RNN verification tools typically rely on restrictive modeling assumptions or coarse over-approximations of the hidden state space, which can lead to overly conservative or inconclusive results. We propose $\textbf{RNN}$ $\textbf{Pro}$babilistic $\textbf{Ve}$rification ($\texttt{RNN-ProVe}$), a probabilistic framework that $\textit{estimates the likelihood}$ of undesired behaviors in RNN-based policies. $\texttt{RNN-ProVe}$ uses policy-driven sampling to approximate the set of hidden states that are feasible under a trained policy, and derives statistical error bounds to produce bounded-error, high-confidence estimates of behavioral violations. Experiments on partially observable single-agent and cooperative multi-agent tasks show that $\texttt{RNN-ProVe}$ yields more quantitative, feasibility-aware probabilistic guarantees than existing tools, while scaling to recurrent and multi-agent settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。