arXiv:2603.27148cs.CRcs.AI2026-03被引 3

提前预测AI代理何时会因多步安全行为累积而越界

SafetyDrift: Predicting When AI Agents Cross the Line Before They Actually Do

  • 将代理行为建模为吸收马尔可夫链,用闭式分析预测违规概率
  • 通信任务中5步内违规概率达85%,技术任务则低于5%
  • 轻量级监控器提前3.7步预警,误报率远低于现有方法

当大语言模型代理依次读取机密文件、生成摘要并外部发送时,单个步骤均不违规,但序列组合构成数据泄露。我们称此为安全漂移:看似安全的行为累积导致违规。本文提出安全漂移模型,将代理安全轨迹建模为吸收马尔可夫链,通过闭式吸收分析计算在给定步数内达到违规状态的概率。单调状态设计导致所有代理在无监督下终将违规(所有状态吸收概率为1.0),因此实际问题不是是否违规,而是何时违规。在40个真实任务、四个类别共357条轨迹上发现,‘不可逆点’高度依赖任务类型:通信任务中,即使进入轻微风险状态,5步内违规概率达85%;技术任务中,任何状态下违规概率均低于5%。基于该模型的轻量级监控器以极低计算开销实现94.7%违规检测率,提前3.7步预警,显著优于关键词匹配(44.7%检测率,55.9%误报率)和逐步大模型判断(52.6%检测率,38.2%误报率),且运行速度超前者60,000倍。

原文摘要 · Abstract (English)

When an LLM agent reads a confidential file, then writes a summary, then emails it externally, no single step is unsafe, but the sequence is a data leak. We call this safety drift: individually safe actions compounding into violations. Prior work has measured this problem; we predict it. SafetyDrift models agent safety trajectories as absorbing Markov chains, computing the probability that a trajectory will reach a violation within a given number of steps via closed form absorption analysis. A consequence of the monotonic state design is that every agent will eventually violate safety if left unsupervised (absorption probability 1.0 from all states), making the practical question not if but when, and motivating our focus on finite horizon prediction. Across 357 traces spanning 40 realistic tasks in four categories, we discover that "points of no return" are sharply task dependent: in communication tasks, agents that reach even a mild risk state have an 85% chance of violating safety within five steps, while in technical tasks the probability stays below 5% from any state. A lightweight monitor built on these models detects 94.7% of violations with 3.7 steps of advance warning at negligible computational cost, outperforming both keyword matching (44.7% detection, 55.9% false positive rate) and per step LLM judges (52.6% detection, 38.2% false positive rate) while running over 60,000x faster.

AI安全风险预测监控系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。