预归一化架构导致视觉信息丢失,提出简单方案修复
The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss
- 发现视觉与文本令牌的范数差异引发更新不对称
- 插入一层归一化层可显著提升多模态与纯文本任务表现
- 适合关注模型架构公平性与跨模态融合的研究者
多模态大语言模型(MLLMs)结合预训练视觉编码器与语言模型,展现出强大能力。然而,其普遍采用的预归一化(Pre-Norm)架构引入了关键缺陷:视觉令牌高范数与文本令牌低范数之间存在严重范数差异。本文通过理论分析表明,这种不平衡并非静态问题,而是导致‘非对称更新动态’——高范数视觉令牌表现出‘表征惯性’,语义变换速度远慢于文本令牌,从根本上阻碍跨模态特征融合。在多个主流MLLM上实证验证了该现象普遍存在。基于此,我们提出简单有效方案:在视觉投影层后加入一个精心初始化的层归一化(LayerNorm)层以实现范数对齐。在LLaVA-1.5架构上的实验显示,该干预不仅显著提升多项多模态基准表现,更在纯文本评估(如MMLU)中取得明显进步,表明修复架构失衡可带来更全面的模型能力。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs), which couple pre-trained vision encoders and language models, have shown remarkable capabilities. However, their reliance on the ubiquitous Pre-Norm architecture introduces a subtle yet critical flaw: a severe norm disparity between the high-norm visual tokens and the low-norm text tokens. In this work, we present a formal theoretical analysis demonstrating that this imbalance is not a static issue. Instead, it induces an ``asymmetric update dynamic,'' where high-norm visual tokens exhibit a ``representational inertia,'' causing them to transform semantically much slower than their textual counterparts. This fundamentally impairs effective cross-modal feature fusion. Our empirical validation across a range of mainstream MLLMs confirms that this theoretical dynamic -- the persistence of norm disparity and the resulting asymmetric update rates -- is a prevalent phenomenon. Based on this insight, we propose a remarkably simple yet effective solution: inserting a single, carefully initialized LayerNorm layer after the visual projector to enforce norm alignment. Experiments conducted on the LLaVA-1.5 architecture show that this intervention yields significant performance gains not only on a wide suite of multimodal benchmarks but also, notably, on text-only evaluations such as MMLU, suggesting that resolving the architectural imbalance leads to a more holistically capable model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。