arXiv:2608.02464cs.AIcs.LG2026-08

用微秒级监控实时发现并修复大模型智能体的运行错误

Real-Time Detection and Repair of LLM Agent Failures

论文配图:Real-Time Detection and Repair of LLM Agent Failures
图 1 · 摘自论文原文
  • 仅通过健康运行数据训练的时序监测器,实现毫秒级故障检测
  • 检测准确率达71%(误报率5%),修复后任务成功率从52%提升至73%
  • 结合确定性验证可零误报修复,适合部署在资源受限场景

大模型智能体在执行过程中会陷入循环、工具链错误蔓延、目标偏离、虚构结果或无声吸收污染内容,而传统逐步评估方法成本高于智能体本身。本文探索仅基于可观测步骤日志能否实现高效故障检测,采用每步耗时微秒级的监测器,仅使用正常运行数据训练。在三个框架共2,823个智能体运行实例上,基于qwen2.5 7b/3b、llama3.1 8b及gemini-2.5-flash的单类回声状态网络集成模型,在5%误报率预算下实现0.71的故障检出率(AUROC 0.872)。其性能随故障发生后时间窗口增加而单调提升(≤3步+0.09,≥9步+0.40),并在AFTraj-2K上外推预测自身失效区域。无需重新训练即可迁移至其他数据集(AFTraj-2K: 0.745, ATBench: 0.779)。监测器存在两个负担:每次部署需健康基线(冷启动时AUROC 0.527,校准后达0.885)和残余误报率。为此引入确定性验证层,通过重算工具实际返回结果与声明总和,并验证所有必要调用是否执行。该方法在对比中捕获60%故障(配合覆盖检查达96%),且零误报(对比监测器54%检出率,17个误报),在llama3.1:8b上保持不变(110/110成功,0/10误报),在1825个健康样本中无误触发。最终将检测闭环至修复:对每个标记异常的运行进行回滚并实时重试,恢复45%故障(对照组仅16%,p=0.0005),使任务成功率从52%提升至73%,平均每轮仅额外调用一次模型。系统每步耗时约200微秒,比判别调用快三个数量级。代码、轨迹与结果均已公开。

原文摘要 · Abstract (English)

LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself. We ask how much detection is achievable from observable step telemetry alone, using monitors costing microseconds per step and trained only on healthy runs. On 2,823 committed agent episodes across three frameworks, three local models (qwen2.5 7b/3b, llama3.1 8b) and a commercial API (gemini-2.5-flash), a one-class echo-state-network ensemble with CUSUM alarms detects 0.71 of failures at a 5% false-alarm budget (AUROC 0.872). Its advantage over a memoryless baseline is a monotone function of post-onset horizon (+0.09 at <=3 steps, +0.40 at >=9), predicting its own failure region out of sample on AFTraj-2K. Ranking transfers with no retraining to two corpora from other groups (AFTraj-2K 0.745, ATBench 0.779). Monitors carry two burdens: a per-deployment healthy null (they do not transfer -- AUROC 0.527 cold against 0.885 recalibrated) and a residual false-alarm rate. We add a layer carrying neither: deterministic verification, which recomputes a run's stated total from the tool results it actually received and confirms every required call was made. Head-to-head it catches 60% of failures (96% with the coverage check) at 0 of 63 false positives against the monitor's 54% at 17%, transfers unchanged to llama3.1:8b (110 of 110 at 0 of 10), and trips on 0 of 1825 healthy episodes. Detection is then closed into repair: each flagged run is rolled back and re-run live, recovering 45% of failures against a 16% resampling control (p=0.0005) and lifting task success from 52% to 73% for about one extra model call per run. The system runs at ~200 microseconds per step, three orders of magnitude below a judge call. Code, traces and results are released.

大模型代理实时监控故障修复低延迟

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。