解决多模态模型文本偏好问题，从注意力空间对齐入手。
MaLoRA: Gated Modality LoRA for Key-Space Alignment in Multimodal LLM Fine-Tuning
- 通过门控机制调节视觉与文本键向量的注意力权重
- 发现视觉键与文本键在注意力空间中分布差异显著
- 适合研究多模态模型对齐与注意力机制的学者
多模态大语言模型在处理视觉-语言数据时表现出明显的文本偏好，限制了其基于视觉证据的有效推理能力。不同于以往将此偏见归因于数据不平衡或指令微调等外部因素，我们提出该偏见源于模型内部架构。具体而言，假设视觉键向量（Visual Keys）相对于语言预训练中学习到的文本键空间处于分布外（OOD）状态。因此，这些视觉键在注意力计算中获得系统性更低的相似度得分，导致其在上下文表示中被低估。为验证此假设，我们从LLaVA和Qwen2.5-VL中提取键向量，使用t-SNE（定性）和Jensen-Shannon散度（定量）分析其分布结构。结果表明，视觉与文本键在注意力空间中占据明显不同的子空间，跨模态差异在统计上显著，超过模内变异几个数量级。这揭示出文本偏见源于注意力键空间内的内在错位，而非仅由外部数据因素造成。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) exhibit a pronounced preference for textual inputs when processing vision-language data, limiting their ability to reason effectively from visual evidence. Unlike prior studies that attribute this text bias to external factors such as data imbalance or instruction tuning, we propose that the bias originates from the model's internal architecture. Specifically, we hypothesize that visual key vectors (Visual Keys) are out-of-distribution (OOD) relative to the text key space learned during language-only pretraining. Consequently, these visual keys receive systematically lower similarity scores during attention computation, leading to their under-utilization in the context representation. To validate this hypothesis, we extract key vectors from LLaVA and Qwen2.5-VL and analyze their distributional structures using qualitative (t-SNE) and quantitative (Jensen-Shannon divergence) methods. The results provide direct evidence that visual and textual keys occupy markedly distinct subspaces within the attention space. The inter-modal divergence is statistically significant, exceeding intra-modal variation by several orders of magnitude. These findings reveal that text bias arises from an intrinsic misalignment within the attention key space rather than solely from external data factors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。