评估大模型智能体系统时，框架选择与模型同样关键。
MASEval: Extending Multi-Agent Evaluation from Models to Systems
- 将整个智能体系统作为评估单位，而非仅聚焦模型
- 在3个基准、3个模型、3个框架下验证框架影响等同于模型影响
- 开源工具支持研究者和开发者优化系统设计与落地
大语言模型驱动的智能体系统快速发展，催生了众多框架（如smolagents、LangGraph、AutoGen、CAMEL、LlamaIndex等）。然而现有评估基准仍以模型为中心：固定智能体配置，无法比较系统其他组件。我们指出，拓扑结构、编排逻辑、错误处理等实现决策显著影响性能。MASEval提出一个框架无关的评估库，将整个系统作为分析单元。通过在3个基准、3个模型、3个框架上的系统级对比，发现框架选择对性能的影响与模型选择相当。该工具使研究者可全面探索智能体系统的各组件，推动系统化设计；也帮助从业者为特定场景选择最优实现。MASEval已开源，许可协议为MIT，代码见https://github.com/parameterlab/MASEval。
原文摘要 · Abstract (English)
The rapid adoption of LLM-based agentic systems has produced a rich ecosystem of frameworks (smolagents, LangGraph, AutoGen, CAMEL, LlamaIndex, i.a.). Yet existing benchmarks are model-centric: they fix the agentic setup and do not compare other system components. We argue that implementation decisions substantially impact performance, including choices such as topology, orchestration logic, and error handling. MASEval addresses this evaluation gap with a framework-agnostic library that treats the entire system as the unit of analysis. Through a systematic system-level comparison across 3 benchmarks, 3 models, and 3 frameworks, we find that framework choice matters as much as model choice. MASEval allows researchers to explore all components of agentic systems, opening new avenues for principled system design, and practitioners to identify the best implementation for their use case. MASEval is available under the MIT licence https://github.com/parameterlab/MASEval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。