改进视觉位置编码，让多模态模型更精准理解图像细节与整体关系。
Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encoding
- 采用金字塔式中心递进的位置编码，优化视觉标记的空间感知。
- 在多个规模的模型上显著提升跨粒度视觉理解能力，减少对锚点的依赖。
- 适合需要精细图像分析的多模态任务，如视觉问答与图文生成。
视觉语言模型（VLMs）在推动通用人工智能方面展现出卓越能力，但传统视觉位置编码在不同粒度层级上的不合理性仍限制其全面感知性能。本文提出金字塔下降式视觉位置编码（PyPE），通过从外围到中心逐步分配位置索引并增量扩展中心感受野，克服了传统光栅扫描方法的局限，并缓解了旋转位置编码（RoPE）带来的长期衰减效应。PyPE降低了相关视觉元素与指令标记间的相对距离，促进注意力权重的合理分配，实现多粒度视觉元素感知，削弱对锚点标记的过度依赖。大量实验表明，PyPE在多种规模的VLM中均持续提升其通用能力。代码已开源：https://github.com/SakuraTroyChen/PyPE。
原文摘要 · Abstract (English)
Vision-language Models (VLMs) have shown remarkable capabilities in advancing general artificial intelligence, yet the irrational encoding of visual positions persists in inhibiting the models' comprehensive perception performance across different levels of granularity. In this work, we propose Pyramid-descent Visual Position Encoding (PyPE), a novel approach designed to enhance the perception of visual tokens within VLMs. By assigning visual position indexes from the periphery to the center and expanding the central receptive field incrementally, PyPE addresses the limitations of traditional raster-scan methods and mitigates the long-term decay effects induced by Rotary Position Embedding (RoPE). Our method reduces the relative distance between interrelated visual elements and instruction tokens, promoting a more rational allocation of attention weights and allowing for a multi-granularity perception of visual elements and countering the over-reliance on anchor tokens. Extensive experimental evaluations demonstrate that PyPE consistently improves the general capabilities of VLMs across various sizes. Code is available at https://github.com/SakuraTroyChen/PyPE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。