用物理模型揭示多模态模型中的偏见机制,助力算法公平性研究。
Physics-based phenomenological characterization of cross-modal bias in multimodal models
- 构建物理启发的代理模型,分析多模态注意力动态与偏见演化。
- 实验证明多模态输入会强化模态主导而非缓解偏见,出现结构化错误吸引子。
- 适合关注多模态公平性、模型可解释性的研究人员参考。
算法公平性用于评估人工智能模型在比较性(如‘类似情况应同等对待’)与非比较性(由模型不准确、随意或不可解释引发)情境下的公正性。尽管多模态大语言模型(MLLMs)在多模态理解、推理与生成方面取得进展,但我们认为复杂的多模态交互动态可能引发隐蔽偏差。本文旨在两方面:一是介绍基于物理实体的经验式可解释方法,替代传统符号主义或形而上学视角;二是主张该现象学框架对解决MLLMs中的算法公平问题具有实际价值。我们提出一个基于物理的代理模型,刻画变换器的动态行为(即语义网络结构与自/交叉注意力),以分析传统嵌入或表示层面难以捕捉的跨模态偏差。通过多输入诊断实验支持:1)使用Qwen2.5-Omni和Gemma 3n进行情绪分类的扰动分析;2)利用物理代理模型对洛伦兹混沌时间序列预测的动力学分析。在两种架构不同的MLLM中,均发现多模态输入强化模态主导而非缓解偏见,表现为系统标签扰动下的结构化误差吸引子模式,辅以动力学分析验证。
原文摘要 · Abstract (English)
The term 'algorithmic fairness' is used to evaluate whether AI models operate fairly in both comparative (where fairness is understood as formal equality, such as "treat like cases as like") and non-comparative (where unfairness arises from the model's inaccuracy, arbitrariness, or inscrutability) contexts. Recent advances in multimodal large language models (MLLMs) are breaking new ground in multimodal understanding, reasoning, and generation; however, we argue that inconspicuous distortions arising from complex multimodal interaction dynamics can lead to systematic bias. The purpose of this position paper is twofold: first, it is intended to acquaint AI researchers with phenomenological explainable approaches that rely on the physical entities that the machine experiences during training/inference, as opposed to the traditional cognitivist symbolic account or metaphysical approaches; second, it is to state that this phenomenological doctrine will be practically useful for tackling algorithmic fairness issues in MLLMs. We develop a surrogate physics-based model that describes transformer dynamics (i.e., semantic network structure and self-/cross-attention) to analyze the dynamics of cross-modal bias in MLLM, which are not fully captured by conventional embedding- or representation-level analyses. We support this position through multi-input diagnostic experiments: 1) perturbation-based analyses of emotion classification using Qwen2.5-Omni and Gemma 3n, and 2) dynamical analysis of Lorenz chaotic time-series prediction through the physical surrogate. Across two architecturally distinct MLLMs, we show that multimodal inputs can reinforce modality dominance rather than mitigate it, as revealed by structured error-attractor patterns under systematic label perturbation, complemented by dynamical analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。