构建跨领域多模态的多智能体辩论评估基准,统一评测标准。
M3MAD-Bench: Multi-Dimensional Evaluation of Multi-Agent Debate Across Domains and Modalities
- 设计统一协议,覆盖5大领域13个数据集,支持文本与视觉语言输入
- 验证多智能体辩论在复杂推理任务中效果优于单一模型,但效率成本高
- 提供准确率、耗时、令牌消耗等多维指标,适合实际系统设计参考
多智能体辩论(MAD)通过结构化辩论提升答案质量与复杂推理能力。然而现有研究存在评价标准碎片化、仅限纯文本场景的问题。为此,我们提出M3MAD-Bench,一个面向多领域、多模态和多维度的统一评估基准。该基准覆盖知识、数学、医学、自然科学和复杂推理五大领域,包含13个数据集,涵盖纯文本与视觉-语言数据,支持可控的跨模态对比。我们在9个不同架构、规模与模态能力的基础模型上评估MAD方法。除准确率外,还引入令牌消耗和推理时间等效率指标,全面刻画性能-成本权衡。实验揭示九项关键洞见:协作式方法在推理密集型及多模态任务中更鲁棒,但通常伴随显著效率开销。这些发现为真实场景中的MAD策略选择与设计提供实用指导。代码已开源。
原文摘要 · Abstract (English)
As an agent-level reasoning and coordination paradigm, Multi-Agent Debate (MAD) orchestrates multiple agents through structured debate to improve answer quality and support complex reasoning. However, existing research on MAD suffers from two fundamental limitations: evaluations are conducted under fragmented and inconsistent settings, making fair comparison difficult, and are largely confined to text-only scenarios, leaving its effectiveness in multimodal settings underexplored. To address these gaps, we introduce M3MAD-Bench, a unified and extensible benchmark for evaluating MAD methods across Multi-domain tasks, Multi-modal inputs, and Multi-dimensional metrics. M3MAD-Bench establishes standardized protocols over five core task domains, including Knowledge, Mathematics, Medicine, Natural Sciences, and Complex Reasoning, covering a total of 13 datasets, and systematically includes both pure text and vision-language data, enabling controlled cross-modality comparison. We evaluate MAD methods on 9 base models spanning different architectures, scales, and modality capabilities. Beyond accuracy, M3MAD-Bench incorporates efficiency-oriented metrics such as token consumption and inference time, providing a holistic view of performance--cost trade-offs. Through extensive experiments, we derive nine key insights, revealing that MAD is not uniformly effective: collaborative methods are generally more robust than adversarial ones, especially on reasoning-intensive and multimodal tasks, but often incur substantial efficiency costs. These findings provide practical guidance for selecting and designing MAD strategies in real-world applications. We believe M3MAD-Bench offers a reliable foundation for future research on standardized and reproducible MAD evaluation. The code is available at https://github.com/liaolea/M3MAD-Bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。