构建统一图框架,让多智能体协作效果可比可测。
ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration

- 将不同方法的协作过程转为统一协作图,实现公平对比。
- 从7个数据集筛选844个需协作的任务,每题预设10个成功参考图。
- 评估仅需毫秒级计算,无需额外大模型推理,适合研究者快速验证。
基于大语言模型的多智能体系统(MAS)发展迅速,但其执行轨迹异构,缺乏统一评估基础。仅看结果的基准忽略协作过程,而使用大模型评分则依赖额外推理且结果受模型和标准影响。本文提出通用评估框架,将原生MAS轨迹映射为统一协作图,实现方法间在相同表示、参考集和指标下的比较。候选图与特定查询的参考森林比对,每个森林包含10个经验证成功的任务完成路径,涵盖多种有效协作方式而非唯一最优解。基于此构建ForestBench,从7个公开数据集筛选844个需协作的查询,预生成每题10个目标条件化的参考图,并评估6个代表性MAS框架。控制实验验证了评估的稳定性与覆盖范围。一旦参考森林建立,只需毫秒即可评分,无需再进行大模型推理,为多样式协作轨迹提供可复用的结构化评估基础。
原文摘要 · Abstract (English)
Multi-agent systems (MAS) built on Large Language Models (LLMs) are proliferating rapidly, but their heterogeneous execution traces provide no common basis for evaluation across methods. Outcome-only benchmarks discard collaborations, whereas LLM-as-Judge evaluation requires additional, model-dependent inference and can vary with the LLM and rubric. We introduce a generalizable evaluation framework that maps native MAS traces into a shared space of unified collaboration graphs, enabling different methods to be evaluated under the same representation, reference set, and metric panel. Candidate graphs are compared with a query-specific reference forest. Each forest is a benchmark-provided collection of verified-success graphs: it records diverse ways in which representative MAS methods can complete the task, rather than prescribing a unique optimal process. Instantiating the framework as ForestBench, we filter $844$ collaboration-necessary queries from seven public datasets, precompute ten successful target-conditioned reference graphs per query, and evaluate six representative MAS frameworks. Controlled backbone, reference-construction, and perturbation studies test the stability and scope of evaluation. Once the benchmark forests are built, ForestBench scores a trace in milliseconds without further LLM inference, providing a reusable structural basis for comparing diverse MAS collaboration traces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。