解决视觉语言模型忽视中间图像内容的问题
CoMemo: LVLMs Need Image Context with Image Memory
- 双路径架构，分离上下文与记忆图像处理
- 在7个基准上优于传统模型，长序列理解更优
- 适合需要精准视觉细节理解的任务
基于大语言模型的大型视觉语言模型（LVLMs）主流采用将视觉特征对齐至语言模型表示的方法。然而，继承自语言模型的架构设计在多模态处理中存在不足：首先，注意力分配呈现双峰分布，导致上下文扩展时中间视觉内容被逐步忽略；其次，传统位置编码难以保留动态高分辨率图像中的关键二维结构关系。为此，我们提出CoMemo——一种双路径架构，包含上下文图像路径与图像记忆路径，有效缓解视觉信息遗漏问题。同时，引入新的位置编码方法RoPE-DHR，通过缩略图位置聚合保持二维空间感知，减轻长序列中的远距离衰减问题。在包括长上下文理解、多图像推理和视觉问答在内的七个基准上的评估表明，CoMemo性能显著优于传统LVLM架构。
原文摘要 · Abstract (English)
Recent advancements in Large Vision-Language Models built upon Large Language Models have established aligning visual features with LLM representations as the dominant paradigm. However, inherited LLM architectural designs introduce suboptimal characteristics for multimodal processing. First, LVLMs exhibit a bimodal distribution in attention allocation, leading to the progressive neglect of middle visual content as context expands. Second, conventional positional encoding schemes fail to preserve vital 2D structural relationships when processing dynamic high-resolution images. To address these limitations, we propose CoMemo - a dual-path architecture that combines a Context image path with an image Memory path for visual processing, effectively alleviating visual information neglect. Additionally, we introduce RoPE-DHR, a novel positional encoding mechanism that employs thumbnail-based positional aggregation to maintain 2D spatial awareness while mitigating remote decay in extended sequences. Evaluations across seven benchmarks,including long-context comprehension, multi-image reasoning, and visual question answering, demonstrate CoMemo's superior performance compared to conventional LVLM architectures. Project page is available at https://lalbj.github.io/projects/CoMemo/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。