通过重排位置编号，提升视觉语言模型对高分辨率图像的推理能力。
ID-Align: RoPE-Conscious Position Remapping for Dynamic High-Resolution Adaptation in Vision-Language Models
- 高分辨率图像令牌继承缩略图位置编号，避免位置索引过度扩展。
- 在MMBench上关系推理任务提升6.09%，多基准测试表现显著改善。
- 适合需要处理高清图像的视觉语言模型研究者使用。
当前提升视觉语言模型性能的常见方法是同时编码图像的高分辨率版本和缩略图。尽管有效，但该方法会产生大量图像令牌。当与广泛使用的旋转位置嵌入（RoPE）结合时，其长期衰减特性会阻碍高分辨率令牌与缩略图令牌之间、以及文本与图像之间的交互。为此，我们提出ID-Align，通过重新排序位置编号缓解上述问题。该方法中，高分辨率令牌继承其对应缩略图令牌的位置编号，同时限制位置索引的过度扩展。在LLaVA-Next框架下的实验表明，ID-Align在关系推理任务上取得6.09%的显著提升，并在多个基准测试中表现优异。代码已公开于https://github.com/zooblastlbz/ID-Align。
原文摘要 · Abstract (English)
Currently, a prevalent approach for enhancing Vision-Language Models (VLMs) performance is to encode both the high-resolution version and the thumbnail of an image simultaneously. While effective, this method generates a large number of image tokens. When combined with the widely used Rotary Position Embedding (RoPE), its long-term decay property hinders the interaction between high-resolution tokens and thumbnail tokens, as well as between text and image. To address these issues, we propose ID-Align, which alleviates these problems by reordering position IDs. In this method, high-resolution tokens inherit IDs from their corresponding thumbnail token while constraining the overexpansion of positional indices. Our experiments conducted within the LLaVA-Next framework demonstrate that ID-Align achieves significant improvements, including a 6.09% enhancement on MMBench's relation reasoning tasks and notable gains across multiple benchmarks. Our code is available at the following link: https://github.com/zooblastlbz/ID-Align.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。