用轨迹自动生成轻量级在线预警器，提前发现大模型代理执行中的失败风险。
PrefixGuard: From LLM-Agent Traces to Online Failure-Warning Monitors
- 通过离线归纳轨迹中确定性步骤，生成可监督训练的监控器。
- 在多个基准上达到最高0.900 AUPRC，比原始文本方法提升0.137平均得分。
- 提供可解释诊断，帮助判断预警信号是否真正可操作。
大型语言模型（LLM）代理现在执行长序列、使用工具的任务，最终结果验证可能太晚而无法干预。在线预警需要轻量级前缀监控器来处理异构轨迹，但手工定义事件模式脆弱，部署时由LLM判断成本高。我们提出PrefixGuard，一种从轨迹生成监控器的框架：先离线进行StepView归纳，再监督训练监控器。StepView从原始轨迹样本中提取确定性的类型化步骤适配器，监控器则学习事件抽象与前缀风险评分。在WebArena、τ²-Bench、SkillsBench和TerminalBench上，最强的PrefixGuard监控器分别达到0.900/0.710/0.533/0.557 AUPRC。在每种表示中使用最强后端，其平均比原始文本控制提升+0.137 AUPRC。LLM判断者在相同前缀预警协议下仍表现较差。我们还推导出基于评分的精确率-召回率曲线下面积（AUPRC）的可观测性上限，将监控误差与无证据可观察的失败区分开。对于有限状态审计，事后确定性有限自动机（DFA）提取在WebArena和τ²-Bench上保持紧凑（29和20状态），但在SkillsBench和TerminalBench扩展至151和187状态。最后，首次警报诊断显示，强排序不等于部署实用性：WebArena排名高却无法支持低误报警报，而τ²-Bench和TerminalBench保留更多可行动的早期警报。这些结果共同将PrefixGuard定位为一种实用的监控合成方案，并带有明确诊断，以判断前缀预警能否转化为可操作干预。
原文摘要 · Abstract (English)
Large language model (LLM) agents now execute long, tool-using tasks where final outcome checks can arrive too late for intervention. Online warning requires lightweight prefix monitors over heterogeneous traces, but hand-authored event schemas are brittle and deployment-time LLM judging is costly. We introduce PrefixGuard, a trace-to-monitor framework with an offline StepView induction step followed by supervised monitor training. StepView induces deterministic typed-step adapters from raw trace samples, and the monitor learns an event abstraction and prefix-risk scorer from terminal outcomes. Across WebArena, $τ^2$-Bench, SkillsBench, and TerminalBench, the strongest PrefixGuard monitors reach 0.900/0.710/0.533/0.557 AUPRC. Using the strongest backend within each representation, they improve over raw-text controls by an average of +0.137 AUPRC. LLM judges remain substantially weaker under the same prefix-warning protocol. We also derive an observability ceiling on score-based area under the precision-recall curve (AUPRC) that separates monitor error from failures lacking evidence in the observed prefix. For finite-state audit, post-hoc deterministic finite automaton (DFA) extraction remains compact on WebArena and $τ^2$-Bench (29 and 20 states) but expands to 151 and 187 states on SkillsBench and TerminalBench. Finally, first-alert diagnostics show that strong ranking does not imply deployment utility: WebArena ranks well yet fails to support low-false-alarm alerts, whereas $τ^2$-Bench and TerminalBench retain more actionable early alerts. Together, these results position PrefixGuard as a practical monitor-synthesis recipe with explicit diagnostics for when prefix warnings translate into actionable interventions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。