首个同步评估多模态大模型公平性的基准，揭示生成与理解中的系统性偏见。
Fair in Mind, Fair in Action? A Synchronous Benchmark for Understanding and Generation in UMLLMs
- 构建统一公平性空间，融合60项指标同步评测理解与生成任务。
- 发现生成环节存在'代际差距'、'个性分裂'等系统性偏见现象。
- 适合关注AI公平性、模型可解释性与伦理对齐的研究者使用。
随着人工智能在各领域广泛应用，公平性已成为核心挑战。然而，该领域面临‘巴别塔’困境：公平性度量众多，但其哲学假设常相互冲突，阻碍统一范式，尤其在统一多模态大语言模型（UMLLMs）中，偏见会跨任务系统性传播。为此，我们提出IRIS基准，据知是首个同步评估UMLLMs理解与生成公平性的基准。依托我们的性别/族裔分类器ARES及四个大规模数据集，该基准能将任意度量标准化并聚合至高维‘公平性空间’，整合60项细粒度指标，涵盖理想公平性、现实保真度、偏见惯性与可调控性（IRIS）。通过该基准，我们对主流UMLLMs的评估揭示了系统性现象如‘生成差距’、个体不一致性如‘个性分裂’，以及‘反刻板印象奖励’，并提供诊断工具以优化模型公平性。其新颖且可扩展的框架可集成不断演进的公平性度量，最终助力破解‘巴别塔’困局。项目页：https://iris-benchmark-web.vercel.app/
原文摘要 · Abstract (English)
As artificial intelligence (AI) is increasingly deployed across domains, ensuring fairness has become a core challenge. However, the field faces a "Tower of Babel'' dilemma: fairness metrics abound, yet their underlying philosophical assumptions often conflict, hindering unified paradigms-particularly in unified Multimodal Large Language Models (UMLLMs), where biases propagate systemically across tasks. To address this, we introduce the IRIS Benchmark, to our knowledge the first benchmark designed to synchronously evaluate the fairness of both understanding and generation tasks in UMLLMs. Enabled by our demographic classifier, ARES, and four supporting large-scale datasets, the benchmark is designed to normalize and aggregate arbitrary metrics into a high-dimensional "fairness space'', integrating 60 granular metrics across three dimensions-Ideal Fairness, Real-world Fidelity, and Bias Inertia & Steerability (IRIS). Through this benchmark, our evaluation of leading UMLLMs uncovers systemic phenomena such as the "generation gap'', individual inconsistencies like "personality splits'', and the "counter-stereotype reward'', while offering diagnostics to guide the optimization of their fairness capabilities. With its novel and extensible framework, the IRIS benchmark is capable of integrating evolving fairness metrics, ultimately helping to resolve the "Tower of Babel'' impasse. Project Page: https://iris-benchmark-web.vercel.app/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。