arXiv:2604.12537cs.CVcs.AI2026-04中稿 · CVPR被引 2

让视觉语言模型自动分配注意力,信息多的地方更精细,冗余区域压缩。

MODIX: A Training-Free Multimodal Information-Driven Positional Index Scaling for Vision-Language Models

  • 根据模态内密度和跨模态交互动态调整位置编码粒度。
  • 在多个模型和任务上提升推理能力,冗余区域注意力下降23%。
  • 无需训练或修改模型,适合希望优化现有VLM的开发者。

视觉语言模型(VLMs)在多模态理解方面取得显著进展,但其位置编码机制仍不完善。现有方法对所有标记统一分配位置索引,忽视了模态内及模态间的信息密度差异,导致注意力分配效率低下,冗余视觉区域占据主导而关键内容被弱化。我们识别出位置粒度是一种隐式资源,提出MODIX(多模态信息驱动的位置索引缩放),一种无需训练的框架,可基于模态特异性贡献动态调整位置步长。MODIX通过协方差熵建模模态内密度,并利用跨模态对齐捕捉模态间交互,生成统一评分以重标定位置索引,使信息丰富模态获得更细粒度表示,冗余部分则被压缩,且无需修改模型参数或架构。在多种架构与基准上的实验表明,MODIX持续提升多模态推理能力,并根据任务依赖的信息分布自适应重分配注意力,表明位置编码应被视为Transformer中可自适应的资源。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have achieved remarkable progress in multimodal understanding, yet their positional encoding mechanisms remain suboptimal. Existing approaches uniformly assign positional indices to all tokens, overlooking variations in information density within and across modalities, which leads to inefficient attention allocation where redundant visual regions dominate while informative content is underrepresented. We identify positional granularity as an implicit resource and propose MODIX (Multimodal Information-Driven Positional IndeX Scaling), a training-free framework that dynamically adapts positional strides based on modality-specific contributions. MODIX jointly models intra-modal density via covariance-based entropy and inter-modal interaction via cross-modal alignment to derive unified scores, which rescale positional indices to allocate finer granularity to informative modalities while compressing redundant ones, without requiring any modification to model parameters or architecture. Experiments across diverse architectures and benchmarks demonstrate that MODIX consistently improves multimodal reasoning and adaptively reallocates attention according to task-dependent information distributions, suggesting that positional encoding should be treated as an adaptive resource in Transformers for multimodal sequence modeling.

视觉语言模型位置编码注意力机制多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。