评测多模态模型理解与生成的协同能力,发现现有模型短板。
Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation

- 构建联合任务数据集,评估理解与生成的交互能力
- 2169个高质量样本,涵盖四种协同维度
- 引入对齐人类判断的评估模型,提升评测可靠性
统一多模态模型在理解与生成方面已取得显著进展。然而,尽管设计统一,现有评估通常将理解与生成能力分开测试,忽视二者之间的协同效应。为此,我们提出Unison,一个包含2,169个高质量统一任务样本的综合性基准,用于评估统一多模态模型在理解与生成方面的协同能力。Unison具备三大优势:1)多维覆盖:涵盖内部一致性、理解引导生成、生成引导理解及相互增强,实现全面评估;2)诊断性评价:提供统一与解耦双轨道,可细粒度定位失败模式并量化统一建模带来的收益;3)人类对齐:引入与人类判断高度一致的Unison-Judge评估模型,确保评估可靠性。基于对主流模型在Unison上的系统评估,我们揭示了当前统一多模态系统的关键局限,并指明未来研究方向。代码、Unison数据集及Unison-Judge模型已公开于https://github.com/FudanCVL/Unison。
原文摘要 · Abstract (English)
Unified multimodal models capable of both understanding and generation have achieved remarkable strides. However, despite their unified designs, existing evaluations typically assess understanding and generation capabilities in isolation, overlooking the synergy between comprehension and generation. To bridge this gap, we introduce Unison, a comprehensive benchmark comprising 2,169 high-quality unified task samples, designed to evaluate joint understanding and generation in unified multimodal models. Unison offers three key strengths: 1) Comprehensive Dimensions: Unison encompasses internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement to enable holistic evaluation. 2) Diagnostic Evaluation: it provides both unified and decoupled tracks for understanding and generation, allowing fine-grained attribution of failure modes and quantitative analysis of the gains from unified modeling. 3) Human Alignment: we also introduce Unison-Judge, an evaluation model well aligned with human judgments to ensure reliable assessment. Based on systematic evaluations of state-of-the-art models on Unison, we uncover critical limitations in current unified multimodal systems and highlight promising directions for future research. Codes, Unison and Unison-Judge are publicly available at https://github.com/FudanCVL/Unison.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。