为视觉语言模型设计更精准的位置编码,提升多模态理解能力。
OMEGA: Optimized Multimodal Position Encoding Index Derivation with Global Adaptive Scaling for Vision-Language Models
- 针对图文差异设计独立位置编码,保留文本序列与图像空间结构。
- 在Qwen2.5-VL-3B上实现最高3.43%的性能提升,大模型亦有稳定增益。
- 适合关注视觉语言模型位置建模优化的研究者与开发者。
视觉语言模型(VLMs)在多模态任务中表现优异,其中位置编码对建模文本的序列结构和图像的空间结构至关重要。然而,现有VLM普遍采用统一的1D或2D位置索引策略,未区分文本与视觉令牌的结构性差异及连续性需求。为此,我们提出OMEGA,一种新型位置编码框架,通过模态特异性位置编码(MSPE)在独立坐标维度上分配位置索引,以保持各模态的内在结构。此外,为对齐多模态数据在位置索引空间中的信息密度,引入全局自适应编码步长缩放(GAESS),根据双模态嵌入熵动态调整视觉令牌的位置编码步长。实验表明,OMEGA在多种架构和VQA基准上均显著提升性能。在视觉密集型任务中,相较基线编码策略,Qwen2.5-VL-3B最高提升3.43%,大模型如Qwen2.5-VL-7B和LLaVA-v1.5-7B亦呈现一致增益。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have demonstrated strong performance across various multimodal tasks, where position encoding plays a vital role in modeling both the sequential structure of textual information and the spatial structure of visual information. However, current VLMs commonly adopt modality-unified 1D or 2D positional indexing strategies, which treat textual and visual tokens uniformly without accounting for their distinct structural properties and sequential continuity for text and spatial coherence for vision. To address this limitation, we propose OMEGA, a novel position encoding framework that employs Modality-Specific Position Encoding (MSPE) to assign positional indices while preserving the inherent structures of each modality across separate coordinate dimensions. Additionally, to align the information density of multimodal data in the positional index space, OMEGA introduces Global Adaptive Encoding Step Scaling (GAESS), which adaptively adjusts the position encoding step size of visual tokens based on the embedding entropy of both modalities. Experimental results demonstrate that OMEGA consistently enhances VLM performance across diverse architectures and VQA benchmarks. On visual-intensive tasks, OMEGA achieves up to 3.43% improvement over baseline position encoding strategies on Qwen2.5-VL-3B, with consistent gains observed across larger models including Qwen2.5-VL-7B and LLaVA-v1.5-7B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。