用AI程序员评测51个智能体开发框架，发现文档易用性直接影响开发成本。
ADK Arena: Evaluating Agent Development Kits via LLM-as-a-Developer
- 让AI写代码并自动修复，用统一开发者对比不同框架表现。
- 开发成本相差5.6倍，但高成本不保证成功率，最高仅80%任务完成率。
- 文档、代码和模型知识可互相替代，单一依赖不是瓶颈。
Agent Development Kits（ADKs）的迅速发展缺乏对其性能影响的实证理解。本文提出LLM-as-a-Developer方法：用一个学习框架API的LLM编码代理，通过验证与反馈循环自动编写并修复代码，直至测试通过。该方法固定开发者变量，仅改变框架，生成成本成为API易用性的量化指标，生成的智能体则构成框架有效性的控制评估。我们构建了全自动化管道ADK Arena，包含框架级Docker隔离、三级验证流程及对SWE-bench、τ²-bench、Terminal-Bench和MCP-Atlas的适配器。评估51个主流Python ADK框架（共204个智能体-基准组合），结果表明：(1) 生成成功率为57%，成本在$0.6至$3.4之间波动，相差5.6倍，是API复杂度的量化指标，但成本不能预测成功率；(2) 无单一框架占优：最优框架在单个基准上解决率达80%，甚至以更低成本击败通用前沿编码代理，但中位数仅为32%；(3) 在信息源消融实验中，真实框架使用率维持在28%–40%区间（原始源码访问时最高，无参考材料时仍达33%），说明文档、源码与参数化知识可互换，无单一瓶颈。
原文摘要 · Abstract (English)
The rapid proliferation of Agent Development Kits (ADKs), SDK-level frameworks for building LLM-powered autonomous agents, has outpaced any empirical understanding of how framework choice affects agent performance. We propose \textbf{LLM-as-a-Developer}, a methodology that replaces human developers with an LLM coding agent that learns each framework's API from documentation, writes agent code, and iteratively repairs it through a validate-and-feedback loop until tests pass. By holding the developer constant and varying only the framework, generation effort becomes a quantitative proxy for API usability and the resulting agents provide a controlled measure of framework effectiveness. We implement this in \textbf{ADK Arena}, a fully automated pipeline with per-framework Docker isolation, a three-level validation pipeline, and benchmark adapters for SWE-bench, $τ^2$-bench, Terminal-Bench, and MCP-Atlas. Evaluating all 51 popular Python ADK frameworks (204 agent--benchmark pairs), we find that: (1)~generation succeeds for 57\% of runs, and its cost varies 5.6$\times$ across frameworks (\$0.6 to \$3.4 per agent), a quantitative proxy for API complexity, though cost alone does not predict success; (2)~no single framework dominates: the best single-benchmark ADK agents resolve up to 80\% of tasks and can even \emph{beat} general-purpose frontier coding agents at a fraction of the cost, yet the median framework resolves only 32\%; (3)~across information-source ablations, genuine framework usage stays within a narrow 28--40\% band (highest with raw source access and still 33\% with no reference material at all), indicating that documentation, source code, and parametric knowledge are largely substitutable rather than any one being a hard bottleneck.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。