厘清代码智能体评估中动作、任务与步骤三层次的差异
What Process Evaluation of Coding Agents Actually Measures: Action, Task, and Step Are Three Different Levels
- 构建分层评估框架，区分动作、任务、步骤三个执行层面
- 实验证明执行不确定性存在于任务层级而非步骤层级
- 发现现有评估易受语义相关性干扰，非真实因果贡献
代码智能体的评估正从结果导向转向过程评估，但现有方法常混淆动作预测、任务不确定性和步骤归因，导致评估目标模糊。本文提出一个分层评估框架，通过基于回放的因果归因方法SCAE（源自执行结构因果模型），结合前缀条件识别、回放干预估计和受控判别信息操控，在12个仓库共499个文件定位任务中验证。结果表明：下一步动作主要由执行溯源驱动，而非代码图转移；执行不确定性集中在任务层级而非步骤层级；全轨迹判别存在系统性碰撞偏差，说明当前过程评估多测量语义相关性，而非可认证的因果贡献。
原文摘要 · Abstract (English)
Coding agents are increasingly evaluated not only by whether they solve a task, but also by how they execute it. However, existing process-level evaluations often treat action prediction, task uncertainty, and step attribution as if they were the same problem, which makes it unclear what such evaluations actually measure. In this paper, we introduce a measurement framework for process evaluation in coding agents and instantiate step-level causal attribution with SCAE, a replay-based estimator derived from a structural causal model of agent execution. Our framework combines prefix-conditioned identification, replay/intervention-based estimation, and controlled judge-information manipulation to study process evaluation at the action, task, and step levels. Experiments on 499 file-localization episodes from 12 repositories show that next actions are driven primarily by execution provenance rather than code-graph transitions, execution uncertainty is structured at the task rather than step level, and full-trace judges exhibit systematic collider bias, suggesting that current process evaluation often measures semantic relevance rather than certified causal contribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。