用概率预测提前预警大模型代理的安全风险,避免事故发生。
ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Prediction
- 将代理行为抽象为符号状态,构建马尔可夫链模型预测安全概率。
- 在自动驾驶和家庭机器人任务中,提前15.84秒预警违规,误报为零。
- 支持动态干预或暂停,适合高安全需求场景使用。
大型语言模型(LLM)代理正广泛应用于机器人、虚拟助手和网络自动化等领域。然而,其随机决策机制带来了难以预见的安全风险。现有运行时监控框架如AgentSpec主要依赖反应式安全规则,仅在危险行为即将发生或已发生时检测,难以应对长时序依赖。本文提出ProbGuard,一种基于概率预测的主动运行时监控框架,可提前预判安全违规。ProbGuard将代理执行抽象为符号状态,并从执行轨迹中学习离散时间马尔可夫链(DTMC)以建模行为动态。运行时,监控器估计从当前状态继续安全执行的概率,当该概率低于用户设定阈值时触发干预。为增强鲁棒性,ProbGuard在抽象中引入语义有效性约束,并提供类似PAC的分析,刻画在标准假设下认证学习模型所需的样本复杂度。我们在自动驾驶和具身家庭代理两个关键安全领域评估ProbGuard。在各类场景中,其能持续提前预测交通法规违规和碰撞,预警时间最长达15.84秒(无误报),在更严格阈值下可达38.66秒。在具身代理任务中,采用重提示干预模式使不安全行为减少65.37%,同时保留80.4%的任务完成率;更严格的终止配置则将不安全行为降低93.60%,但任务完成率下降更多。
原文摘要 · Abstract (English)
Large Language Model (LLM) agents increasingly operate across domains such as robotics, virtual assistants, and web automation. However, their stochastic decision-making introduces safety risks that are difficult to anticipate during execution. Existing runtime monitoring frameworks, such as AgentSpec, primarily rely on reactive safety rules that detect violations only when unsafe behavior is imminent or has already occurred, limiting their ability to handle long-horizon dependencies. We present ProbGuard, a proactive runtime monitoring framework for LLM agents that anticipates safety violations through probabilistic risk prediction. ProbGuard abstracts agent executions into symbolic states and learns a Discrete-Time Markov Chain (DTMC) from execution traces to model behavioral dynamics. At runtime, the monitor estimates the probability that execution will remain safe from the current state, and triggers an intervention when this probability falls below a user-defined threshold. To improve robustness, ProbGuard incorporates semantic validity constraints in the abstraction and admits a PAC-style analysis that characterizes the sample complexity required to certify the learned model under standard assumptions. We evaluate ProbGuard in two safety-critical domains: autonomous driving and embodied household agents. Across evaluated scenarios, ProbGuard consistently predicts traffic law violations and collisions in advance, with warnings up to 15.84 seconds at a threshold yielding no false alarms, and up to 38.66 seconds at stricter thresholds. In embodied agent tasks, ProbGuard's re-prompting intervention mode reduces unsafe behavior by 65.37% relative to the unmonitored baseline while retaining 80.4% of the baseline task completion; a stricter halting configuration reduces unsafe behavior by 93.60% at a larger cost in completion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。