通过逐注意力头对齐视觉与语言表征,提升多模态大模型的视觉理解能力。
Mind the Heads: Topological Representation Alignment for Multimodal LLMs

- 按注意力头细粒度对齐跨模态表征,保留局部结构关系。
- 在18个基准上显著提升视觉主导任务性能,减少视觉幻觉。
- 反直觉发现:越不匹配的头越需对齐,效果越好,适合改进视觉依赖模型。
表示对齐已成为提升多模态大语言模型(MLLMs)的有效方法,通过将内部表征正则化至外部视觉编码器的表征。然而,现有方法通常仅对语言主干的固定层进行对齐,忽视了Transformer模型的细粒度结构。本文提出头级表示对齐(HeRA),在单个注意力头层面实现跨模态对齐。基于柏拉图表征假设,该方法聚焦于保持表征的拓扑结构(即局部邻域关系)。采用互K近邻(MKNN)对齐度量,引入可微分对比目标作为局部结构匹配的代理。HeRA在多模态训练中针对特定注意力头应用此目标,这些头根据MKNN度量的对齐得分选出。反直觉的是,对最不匹配的头进行对齐能带来最大收益。在多个MLLM和18个基准上的广泛评估表明,HeRA在挑战性的视觉主导任务中持续提升性能,并自然抑制对语言先验的过度依赖,有效缓解视觉幻觉。代码已公开。
原文摘要 · Abstract (English)
Representation alignment has emerged as an effective approach to improve Multimodal Large Language Models (MLLMs) by regularizing their internal representations toward those of an external vision encoder. However, existing methods typically align a fixed layer of the language backbone, overlooking the fine-grained structure of Transformer models. In this work, we propose Head-Wise Representation Alignment (HeRA), a method that enforces cross-modal alignment at the level of individual attention heads. Our approach is grounded in the Platonic Representation Hypothesis, focusing on preserving the topological structure of representations (i.e., their local neighborhood relationships) across modalities. Following the Mutual K-Nearest Neighbor (MKNN) alignment metric, we introduce a contrastive objective that acts as a differentiable proxy for matching local structures. HeRA applies this objective during multimodal training to specific attention heads in the LLM, selected by their alignment score according to the MKNN metric. Counterintuitively, we find that aligning the least aligned heads yields the largest gains. Extensive evaluations across multiple MLLMs and 18 benchmarks demonstrate that HeRA consistently improves performance on challenging vision-centric tasks and serves as an effective regularizer against visual hallucinations by naturally curbing the over-reliance on linguistic priors. Our code is publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。