arXiv:2512.07497cs.AIcs.SE2025-12被引 7

分析大模型在自主任务中失败原因,发现模型大小不决定可靠性。

How Do LLMs Fail In Agentic Scenarios? A Qualitative Analysis of Success and Failure Scenarios of Various LLMs in Agentic Simulations

  • 通过细粒度行为分析,识别成功执行的策略与常见失败模式。
  • 三款模型在不确定任务中表现相近,深度强化学习提升可靠性。
  • 揭示四类典型失败:过早行动、过度帮助、干扰污染、负载脆弱。

我们研究大型语言模型(LLMs)在具备工具使用能力的自主代理场景中的失败机制。基于Kamiwaza Agentic Merit Index (KAMI) v0.1基准,对Granite 4 Small(32B)、Llama 4 Maverick(400B)和DeepSeek V3.1三款代表性模型在文件系统、文本提取、CSV分析和SQL场景下的900条执行轨迹进行分析。不关注总体得分,而是开展逐次任务的细粒度行为剖析,揭示实现多步工具操作成功的策略及反复出现的失败模式。结果表明,模型规模并不能预测代理鲁棒性:在某些依赖不确定性的任务中,Llama 4 Maverick(400B)仅略优于Granite 4 Small(32B);而DeepSeek V3.1的更高可靠性主要源于后训练强化学习,而非架构或规模。各模型中普遍存在四类失败原型:缺乏依据的过早行动、替代缺失实体的过度帮助、受干扰导致的上下文污染、高负载下的脆弱执行。这些模式强调需要更注重交互式溯源、恢复行为与环境自适应的评估方法,提示企业级部署不仅需更强模型,还需有意识地设计验证机制、约束发现与真实数据遵循能力。

原文摘要 · Abstract (English)

We investigate how large language models (LLMs) fail when operating as autonomous agents with tool-use capabilities. Using the Kamiwaza Agentic Merit Index (KAMI) v0.1 benchmark, we analyze 900 execution traces from three representative models - Granite 4 Small, Llama 4 Maverick, and DeepSeek V3.1 - across filesystem, text extraction, CSV analysis, and SQL scenarios. Rather than focusing on aggregate scores, we perform fine-grained, per-trial behavioral analysis to surface the strategies that enable successful multi-step tool execution and the recurrent failure modes that undermine reliability. Our findings show that model scale alone does not predict agentic robustness: Llama 4 Maverick (400B) performs only marginally better than Granite 4 Small (32B) in some uncertainty-driven tasks, while DeepSeek V3.1's superior reliability derives primarily from post-training reinforcement learning rather than architecture or size. Across models, we identify four recurring failure archetypes: premature action without grounding, over-helpfulness that substitutes missing entities, vulnerability to distractor-induced context pollution, and fragile execution under load. These patterns highlight the need for agentic evaluation methods that emphasize interactive grounding, recovery behavior, and environment-aware adaptation, suggesting that reliable enterprise deployment requires not just stronger models but deliberate training and design choices that reinforce verification, constraint discovery, and adherence to source-of-truth data.

大模型智能体失败分析工具使用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。