arXiv:2606.17454cs.AIcs.LG2026-06被引 1

通过轨迹分析揭示模型与智能体执行间的偏差,提升系统级性能。

Dissecting model behavior through agent trajectories

  • 构建可定制的Simple Strands Agent框架,统一捕捉多模型共性行为
  • 在3个主流基准上复现或超越现有表现,关键指标达138k条轨迹数据支持
  • 用代码状态空间分析轨迹,发现模型解决问题的差异化努力分配模式

AI智能体的表现不仅是建模问题,更是系统性问题。模型的先进能力需通过智能体机制实现,因此模型假设与执行机制之间的偏差会阻碍能力转化。本文提出‘意图-执行’差距概念,即模型意图与实际执行间的不匹配。我们主张缩小该差距与工具设计、执行循环同等重要。为此,开发了名为Simple Strands Agent(SSA)的简易可定制框架,旨在捕捉不同模型族(如Claude、Gemini、GPT、Grok、Qwen)的通用行为模式及少量模型特异性偏好。主要贡献有二:(i) 在SWE-Pro、SWE-Verified和Terminal-Bench-2等主流智能体基准上复现或超越各模型厂商报告的pass@1性能;(ii) 基于生成的13.8万条轨迹分析,突破仅看pass@1指标的局限。通过将智能体轨迹映射至代码状态空间,观察到模型在解题行为上的差异。细粒度指标如编辑频率、测试活跃度与阶段跃迁现象,揭示了不同模型在自主求解各阶段的努力分配差异。

原文摘要 · Abstract (English)

AI agent performance is not just a modeling problem, it is fundamentally a systems problem. The advanced capabilities of models are realized through agent harnesses. Therefore, a gap between model assumptions and harness behavior can easily prevent the model's full capabilities from translating into agent performance. We formalize this as the `intent-execution' gap: the mismatch between what the model intends and what the harness executes, and vice versa. We argue that minimizing this intent-execution gap is as important as other aspects of harness design such as tools and execution loops. To illustrate the impact of this harness-model alignment, we develop a simple and customizable harness called `Simple Strands Agent' (SSA). SSA aims to find the bulk of common patterns which generalize across different model families (such as Claude, Gemini, GPT, Grok, Qwen), as well as a small number of model-specific preferences. We make two contributions: (i) we reproduce or improve on the pass@1 performance reported by diverse model-provider families on popular agentic benchmarks (SWE-Pro, SWE-Verified and Terminal-Bench-2), and (ii) building on an analysis of 138k trajectories generated by SSA, we look beyond the pass@1 numbers which tend to be relatively even across frontier models. By representing agent trajectories in code state-spaces, we observe model-level differences in problem-solving behavior. Finer-grained metrics such as edit frequency, testing activity, and phase-transitions reveal how individual models allocate effort across different stages of autonomous problem solving.

智能体轨迹分析模型对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。