改进视觉语言模型的位置编码，提升多模态理解能力
Revisiting Multimodal Positional Encoding in Vision-Language Models
- 提出双组件分析框架，聚焦位置设计与频率分配
- 在多个基准上实现显著提升，细粒度理解能力增强
- 无需修改结构，可直接替换现有位置编码
多模态位置编码对视觉语言模型至关重要，但相关研究仍缺乏系统性。本文通过全面分析旋转位置编码（RoPE）的两个核心组成部分——位置设计与频率分配，提出三项关键准则：位置一致性、全频段利用和保留文本先验，以确保布局清晰、表征丰富且预训练语言模型知识有效迁移。基于此，我们提出两种简单即插即用的变体：多头RoPE（MHRoPE）与交错式RoPE（MRoPE-I），无需架构改动。大量实验表明，所提方法在多样化基准上持续优于现有方法，显著提升通用与细粒度多模态理解性能。代码将发布于 https://github.com/JJJYmmm/Multimodal-RoPEs。
原文摘要 · Abstract (English)
Multimodal position encoding is essential for vision-language models, yet there has been little systematic investigation into multimodal position encoding. We conduct a comprehensive analysis of multimodal Rotary Positional Embedding (RoPE) by examining its two core components: position design and frequency allocation. Through extensive experiments, we identify three key guidelines: positional coherence, full frequency utilization, and preservation of textual priors-ensuring unambiguous layout, rich representation, and faithful transfer from the pre-trained LLM. Based on these insights, we propose Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), two simple and plug-and-play variants that require no architectural changes. Our methods consistently outperform existing approaches across diverse benchmarks, with significant improvements in both general and fine-grained multimodal understanding. Code will be avaliable at https://github.com/JJJYmmm/Multimodal-RoPEs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。