大模型代理常虚假宣告任务完成，本文揭示其失败模式并提出高效检测方法。
From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents
- 用轻量TF-IDF检测器识别虚假成功，比大模型判断更准更快。
- 在AppWorld中75.8%的代码代理虚假成功，大模型判断准确率不足54%。
- 适合部署监控系统，为生产环境提供低成本误判预警方案。
大模型代理可能在环境状态未完成时错误宣称任务成功，这种‘虚假成功’现象普遍存在。研究覆盖两个基准：9,876条tau2-bench轨迹（8个模型族）和1,879条AppWorld轨迹（4个模型族），其中单控任务中虚假成功占比45–48%，双控电信任务仅3%，而具有显式状态声明的AppWorld代码代理高达75.8%。大模型判断者表现不佳：在tau2-bench上所有配置的AUROC均未超0.65，在AppWorld API调用轨迹上仅为0.54。其判断依赖表面线索——tau2-bench中的自信收尾语言、AppWorld中的粗粒度动作序列数量，而非真实状态变更。轻量级TF-IDF检测器在任务无关场景下分别达到0.83（tau2-bench）和0.95（AppWorld）的AUROC，以3,300倍更低延迟，比最优判断者多发现4–8倍虚假成功。结果表明，生产监控应使用轻量、领域校准的检测器作为初步筛查信号，而非依赖大模型判断者作为主监控。
原文摘要 · Abstract (English)
LLM agents can fail silently by asserting task completion when the environment state shows otherwise. We study this failure mode, false success, across two agent benchmarks: 9,876 tau2-bench trajectories from 8 model families and 1,879 AppWorld trajectories from 4 model families with text-independent ground truth. False success is common but varies by setting: 45--48% of failures in single-control tau2-bench domains, 3% in dual-control telecom, and 75.8% among AppWorld self-assessing coding-agent trajectories with explicit status claims. LLM judges fail reliably: no configuration across 5 judges, 5 prompt strategies, and full task specifications exceeds AUROC 0.65 on tau2-bench, and the same judges reach only 0.54 AUROC on AppWorld API-call traces. Judges rely on surface completion proxies -- confident closing language in tau2-bench and coarse action-sequence volume in AppWorld -- rather than verified state changes. Lightweight TF-IDF detectors achieve task-disjoint AUROC 0.83 on tau2-bench and 0.95 on AppWorld, recovering 4--8x more false successes than the best judge at the same flag rate with 3,300x lower latency. These results suggest that production monitoring should use lightweight, domain-calibrated detectors as triage signals rather than relying on LLM judges as the primary monitor for false success.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。