arXiv:2604.11978cs.AI2026-04被引 22

诊断大模型代理在长任务中失败的原因,发现其性能随任务长度呈系统性下降。

The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break

  • 构建跨领域基准HORIZON,系统生成长任务并追踪代理行为轨迹。
  • 在4个领域测试3100+轨迹,发现代理性能随任务长度显著退化。
  • 提出基于大模型判官的可复现故障归因方法,适合研究长程智能体的开发者。

大型语言模型(LLM)代理在短中等任务上表现优异,但在需要长期、相互依赖动作序列的长任务中常出现失效。尽管代理系统进展迅速,但长任务失败机制仍缺乏清晰刻画,阻碍了跨领域的系统性诊断与比较。为此,本文提出HORIZON——首个跨领域长任务诊断基准,用于系统构建任务并分析基于LLM代理的长任务失效行为。利用HORIZON,我们评估了多个模型家族(GPT-5变体和Claude模型)的最先进代理,在四个代表性代理领域收集超过3100条轨迹,研究任务时长相关的性能退化模式。此外,我们提出一种基于轨迹的大模型作为裁判(LLM-as-a-Judge)流水线,实现可扩展且可复现的故障归因,并通过人工标注验证,达到强一致性(人与人κ=0.61;人与模型κ=0.84)。研究结果为系统性跨领域分析长任务代理失败提供了初步方法论支持,并为构建更可靠的长任务代理提供实践指导。项目官网已发布于HORIZON Leaderboard,欢迎社区贡献。

原文摘要 · Abstract (English)

Large language model (LLM) agents perform strongly on short- and mid-horizon tasks, but often break down on long-horizon tasks that require extended, interdependent action sequences. Despite rapid progress in agentic systems, these long-horizon failures remain poorly characterized, hindering principled diagnosis and comparison across domains. To address this gap, we introduce HORIZON, an initial cross-domain diagnostic benchmark for systematically constructing tasks and analyzing long-horizon failure behaviors in LLM-based agents. Using HORIZON, we evaluate state-of-the-art (SOTA) agents from multiple model families (GPT-5 variants and Claude models), collecting 3100+ trajectories across four representative agentic domains to study horizon-dependent degradation patterns. We further propose a trajectory-grounded LLM-as-a-Judge pipeline for scalable and reproducible failure attribution, and validate it with human annotation on trajectories, achieving strong agreement (inter-annotator κ=0.61; human-judge κ=0.84). Our findings offer an initial methodological step toward systematic, cross-domain analysis of long-horizon agent failures and offer practical guidance for building more reliable long-horizon agents. We release our project website at \href{https://xwang2775.github.io/horizon-leaderboard/}{HORIZON Leaderboard} and welcome contributions from the community.

智能体长任务诊断评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。