量化编程智能体的效率与成本，揭示复杂设计未必更好。
Taming Scylla: Understanding the multi-headed agentic daemon of the coding seas
- 构建七层测试框架，逐步增加难度以隔离影响因素。
- 提出成本通过率（CoP）指标，精确衡量每解一题的预期开销。
- 验证复杂架构不必然提升质量，适合评估代码生成工具的开发者。
基于大模型的开发工具正快速自动化软件任务，但缺乏严谨方法评估不同架构选择（提示、技能、工具、多智能体配置）对能力与成本的实际影响。本文提出Scylla评估框架，通过七级递进测试（T0-T6）进行结构化消融实验，分离出直接影响结果的因素。核心指标为成本通过率（CoP）：获得一个正确解的预期美元成本，直接量化复杂性与效率的权衡。该框架与模型无关，可适配任意命令行工具；本研究以Claude Sonnet 4.5为被测工具，使用同一厂商的多个LLM裁判（Opus 4.5、Sonnet 4.5、Haiku 4.5）进行评估共识，评分方式包括直接测试、人工设计的LLM评估标准及定性分析。结果表明，该框架可复现地量化智能体复杂性与实际产出之间的权衡，揭示架构复杂性并不总是提升质量。
原文摘要 · Abstract (English)
LLM-based tools are automating more software development tasks at a rapid pace, but there is no rigorous way to evaluate how different architectural choices -- prompts, skills, tools, multi-agent setups -- materially affect both capability and cost. This paper introduces Scylla, an evaluation framework for benchmarking agentic coding tools through structured ablation studies that uses seven testing tiers (T0-T6) progressively adding complexity to isolate what directly influences results and how. The key metric is Cost-of-Pass (CoP): the expected dollar cost to get one correct solution, which directly quantifies the trade-off between complexity and efficiency. The framework is model-agnostic, designed to work with any CLI tool; this paper demonstrates it with Claude Sonnet 4.5, using multiple LLM judges (Opus 4.5, Sonnet 4.5, Haiku 4.5) from the same vendor for evaluation consensus, where judges score results using direct tests, human-designed LLM-evaluated rubrics, and qualitative assessment. The result is a reproducible framework that quantifies trade-offs between agent complexity and actual outcomes, suggesting that architectural complexity does not always improve quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。