arXiv:2605.18332cs.SEcs.AI2026-05被引 4

不同框架的代码生成代理,同一行为可能正反效果,需跨框架验证。

Same Signal, Different Semantics: A Cross-Framework Behavioral Analysis of Software Engineering Agents

  • 在126种配置中分层隔离框架与模型影响,分析行为与结果关系
  • 47个配置低错误率更优,48个却高错误率反而更好,方向相反
  • 框架差异比模型家族影响更大,行为信号意义可能完全相反

对基于大模型的软件工程代理进行行为研究,通常得出如‘测试步骤紧跟代码修改’、‘错误链短’等操作规则。这些规则多来自单一框架,其是否能在结构不同的代理设计间有效迁移尚无直接验证。本文在生态尺度上开展分析:涵盖126种代理配置、43种框架、共64,380次SWE-bench运行,每种配置由一个LLM与一个框架(如SWE-Agent、OpenHands)组合而成,后者提供工具与工作流。通过固定某一层次(模型或框架),分离两者影响,逐配置测量行为-结果关联,并比较一致性。结果显示,保持模型不变仅更换框架时,所有动作特征的行为差异显著;多数信号不仅幅度不同,甚至方向相反。以错误率为例:47种配置中错误率越低解决率越高,而48种配置则反之。此外,五项连续特征和七项二值模式中的三项也呈现方向性分歧。框架身份解释了64%的配置间差异,远超模型家族的10%。结论是:同一可观测行为信号在不同配置中可能具有相反含义。因此,任何单一框架所得行为结论,必须经过跨配置验证方可推广。

原文摘要 · Abstract (English)

Behavioral studies of LLM-based software engineering agents extract operational rules about which trajectory shapes correlate with higher resolution rates: that a test step follows a code modification, that error cascades are short, or that trajectories are compact. Each rule is typically derived from a single framework, and whether it transfers, in sign as well as magnitude, to structurally different agent designs has not been directly tested. We address this at ecosystem scale: 64,380 SWE-bench runs from 126 agent configurations spanning 43 frameworks, where each configuration pairs an LLM with a framework (e.g., SWE-Agent, OpenHands) that supplies its tools and workflow. We separate framework effects from LLM effects by holding each layer fixed in turn, then measure one behavior-outcome effect per configuration and examine how those effects agree or disagree. Swapping the framework while the LLM is held fixed produces large behavioral differences in every action feature. On most signals, configurations disagree not merely in magnitude but in direction. Error rate is the cleanest case: 47 configurations resolve more issues when their error rate is lower, while 48 resolve more when it is higher. Five other continuous features and three of seven binary patterns from prior SE literature show similar directional disagreement. Framework identity accounts for more of this variation than LLM family: for mean turns, framework explains 64% of the between-configuration variance against the LLM's 10%. The implication is that the same observable behavioral signal can carry opposite meaning for different agent configurations. Behavioral findings from any single framework therefore warrant cross-configuration validation before being claimed as general.

软件工程LLM代理行为分析框架差异

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。