用全流程轨迹评估代码智能体，让评测结果可读可诊断。
AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation
- 结合形式化验证与大模型撰写轨迹评述
- 每轮运行生成可解释的评分理由
- 适合产品迭代和故障排查场景
我们提出 AgentLens，一个面向交互式代码智能体的生产环境评估基准。现有代码智能体评测多简化为任务通过/失败单比特判断，但实际使用者关注的是完整执行轨迹：智能体如何理解指令、调用工具、自检纠错、与用户沟通等。AgentLens 评估整个轨迹表现，将形式化验证与大模型生成的轨迹评述结合，并支持并排对比，使每轮运行都能输出清晰的得分解释。该设计不仅用于模型排序，还可用于诊断模型行为、对比自身智能体版本差异，以及在每日流水线中发现产品回归问题。基准已开源，地址为 https://github.com/agent-lens/agent-lens-bench。
原文摘要 · Abstract (English)
We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the task pass? -- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is. This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline. We release the benchmark as open source at https://github.com/agent-lens/agent-lens-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。