arXiv:2606.16988cs.SEcs.LG2026-06

通过行为指纹分析代码生成智能体的执行路径,识别其独特习惯。

Agent trajectories as programs: fingerprinting and programming coding-agent behavior

论文配图:Agent trajectories as programs: fingerprinting and programming coding-agent behavior
图 1 · 摘自论文原文
  • 用程序化表示捕捉智能体解题过程中的行为特征,形成可区分的指纹。
  • 在未见轨迹上准确识别来源智能体达85.7%,排除任务泄露干扰。
  • 适用于模型路由、监控与成本分析,适合开发者调试智能体行为。

基准得分只能告诉你智能体是否正确完成任务,却无法揭示其如何完成。本文提出一种在不同模型、任务和方法下比较智能体程序行为的方法。我们对比了十种智能体,发现它们的行为习惯具有可识别性,称为行为指纹。通过探测这些程序签名,在控制任务泄露的前提下,可将未知轨迹正确归属到对应智能体,准确率达85.7%。我们开发了基于涌现词汇归纳的程序表示方法,实现最大程度压缩表面差异的同时,保留模型行为模式的独特性。在SWE-Bench数据集上应用该框架,发现来自相似发布周期或蒸馏关系的模型(如教师-学生对)行为最相似,其Jensen-Shannon散度为0.25,约为其他模型对的一半。随着模型性能趋近饱和,仅依赖成功率已不足以评估模型,需从更全面的程序维度探究行为。我们提出了ProcGrep库,支持自顶向下审计和评估智能体的执行策略。本工作可应用于任务感知模型路由、智能体监控及细粒度成本分析。

原文摘要 · Abstract (English)

Benchmark scores tell you what an agent got right; they do not tell you how it got there. In this work, we introduce methods for comparing agents procedurally in different contexts, where the model, tasks, and approaches vary. We compare ten agents and find that they are identifiable by their behavioral habits, which we define as fingerprints: a probe over these procedural signatures attributes an unseen trajectory to the correct agent at 85.7% accuracy, controlling for leakage across tasks. We develop procedural representations for agent problem-solving procedures with an emergent vocabulary induction technique that is meant to be maximally compressive to avoid surface-level variation while being expressive enough to unveil the quirks of the models' patterns. We apply our framework to the software engineering evaluation dataset SWE-Bench to study the structural distinctness of agent trajectories and find that behavior is most similar between models from similar release periods and those that are distilled from one another (e.g., a distilled student model and its teacher have a Jensen-Shannon divergence of 0.25, about half the distance between other model pairs). As more models saturate evaluations, we believe that it will be important to probe model behavior along more holistic dimensions than success rates alone. We introduce ProcGrep, a library for auditing and evaluating agents for how they approach tasks at a procedural level given their traces in a top-down fashion. We believe this work has a range of applications to help developers work with and program coding agents, such as task-aware model routing, agent monitoring, and finer-grained cost analysis.

智能体行为代码生成行为指纹程序分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。