解决多模态模型长文本中视觉信息衰减问题
Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding
- 提出跨模态位置编码机制，分离模态内与跨模态位置关系
- 在长上下文场景下保持视觉信号稳定，避免视觉信息随距离衰减
- 适用于需要长期视觉依赖的多模态任务，如图文对话、视频理解
尽管多模态大语言模型表现出色，但在长上下文场景中仍存在视觉信息衰减问题。具体表现为：随着文本序列变长，模型对视觉标记的关注度下降，导致生成内容脱离视觉约束。我们归因于多模态RoPE固有的归纳偏置——随着视觉与文本标记间距离增加，跨模态注意力受到惩罚。为此，我们提出跨模态距离无关位置编码（DIPE），一种简单但有效的机制，通过解耦基于模态交互的位置编码实现。DIPE保留模态内交互的自然相对位置以维持局部结构，同时为跨模态交互引入锚定感知邻近性。该策略有效缓解了跨模态距离带来的惩罚，确保视觉信号在任意上下文长度下均保持感知一致性。实验表明，将DIPE与多模态RoPE结合后，模型在长上下文场景中保持稳定的视觉定位能力，显著缓解视觉衰减，同时在标准短上下文基准上性能不受影响。代码已开源。
原文摘要 · Abstract (English)
Despite the remarkable capabilities of Multimodal Large Language Models (MLLMs), they still suffer from visual fading in long-context scenarios. Specifically, the attention to visual tokens diminishes as the text sequence lengthens, leading to text generation detached from visual constraints. We attribute this degradation to the inherent inductive bias of Multimodal RoPE, which penalizes inter-modal attention as the distance between visual and text tokens increases. To address this, we propose inter-modal Distance Invariant Position Encoding (DIPE), a simple but effective mechanism that disentangles position encoding based on modality interactions. DIPE retains the natural relative positioning for intra-modal interactions to preserve local structure, while enforcing an anchored perceptual proximity for inter-modal interactions. This strategy effectively mitigates the inter-modal distance-based penalty, ensuring that visual signals remain perceptually consistent regardless of the context length. Experimental results demonstrate that by integrating DIPE with Multimodal RoPE, the model maintains stable visual grounding in long-context scenarios, significantly alleviating visual fading while preserving performance on standard short-context benchmarks. Code is available at https://github.com/lchen1019/DIPE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。