提出衡量框架AI辅助能力的新指标，发现规范对齐比声明式设计更重要。
Declarative by Design, Assistable Only by Convention: Benchmarking Multi-Agent Frameworks for AI-Assistability
- 用结构对齐与正确率结合定义AI辅助能力指标
- 规范对齐的Agno得分最高（0.55），声明式框架DSPy最低（0.07）
- 适合关注AI编程助手效率的研究者和框架设计者
多智能体框架（MAFs）有望简化基于大模型的软件开发，但缺乏能衡量AI编码助手生成正确、框架特定代码能力的系统性指标。本文提出 extit{AI-assistability}（$/mathcal{AI}$）这一综合指标，通过结构对齐（$arσ$）与功能正确率（pass@1）的结合，量化框架对AI辅助开发的适应性。为在受控环境下评估该指标，我们设计了DDL2PropBank新基准任务，将关系型数据库模式映射到PropBank语义角色集，并在十种框架中使用相同的代理逻辑实现，采用“代理即工具”模式。结果挑战了声明式设计天然更利于AI辅助的直觉：尽管Agno是单一规范模式且接口符合惯例，其$/mathcal{AI}$得分为0.55，表现最佳；而最声明式的DSPy框架得分仅0.07，因其新颖抽象在训练数据中代表性不足。分析显示，规范对齐是驱动AI辅助能力的核心因素（$arσ$与pass@1相关系数$r=0.576$）。所有代码、数据与实现均开源于https://github.com/ahmeshaf/ddl2propbank。
原文摘要 · Abstract (English)
Multi-agent frameworks (MAFs) promise to simplify LLM-driven software development, yet no principled metric captures how well AI coding assistants can generate correct, framework-specific code. We introduce \textit{AI-assistability} ($\mathcal{AI}$), a composite metric that quantifies a framework's amenability to AI-assisted development by combining structural alignment ($\barσ$) with functional correctness (pass@1). To evaluate this metric in a controlled setting, we design DDL2PropBank, a novel benchmark task that maps relational database schemas to PropBank semantic rolesets, and implement identical agent logic across ten frameworks using the Agent-as-a-Tool pattern. Our results challenge the intuition that declarative framework design guarantees AI-assistability: Agno, with a single canonical pattern and convention-aligned API, achieves the highest $\mathcal{AI}$ score (0.55), while DSPy -- the most declarative framework by design -- scores lowest (0.07), as its novel abstractions are insufficiently represented in AI training data. We find that convention alignment, not declarative design alone, is the primary driver of AI-assistability ($r = 0.576$ between $\barσ$ and pass@1). All artifacts -- DDL2PropBank, PropBank MCP server, and all implementations -- are available at https://github.com/ahmeshaf/ddl2propbank
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。