提出新型视觉语言桥梁,实现无限上下文处理且显存占用极低。
Linear Multi-Timescale Retention as a Memory-Efficient Vision-Language Bridge

- 用线性时序保留机制替代注意力,实现O(N)复杂度
- 可处理26万张图像块,显存仅11.2GB,远超传统方法
- 显著提升物体持续识别与全局语义理解,适合长序列视觉任务
视觉语言模型在处理高分辨率图像时面临计算瓶颈,因Softmax多头注意力(MHA)具有O(N²)内存复杂度。尽管用独立MLP替代可降至O(N),但会丧失空间序列路由能力,严重影响全局场景理解与物体持续性。本文提出线性多时标保留(LIA-MTR)模块,通过ELU正向特征映射、自适应写入门控和对数线性递归衰减,将连续视觉序列数学压缩为有限记忆状态。理论分析证明其序列交互复杂度严格保持O(N)。实验表明,合成检索任务中LIA-MTR能完美传递16,000个标记的上下文,消除“中间丢失”问题;硬件测试显示其可原生处理262,144个视觉块,仅需11.2GB显存,而标准MHA在16,384块时即内存溢出。经66.5万条对话样本指令微调后,其在MME基准上表现优于行业标准MLP基线(71.00% vs. 68.11%),物体持续性提升10个百分点,全局语义提取更优。本工作建立了数学严谨、计算平坦的无限上下文视觉语言融合基础。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) face a critical computational bottleneck when processing high-resolution imagery due to the $O(N^2)$ memory complexity of Softmax Multi-Head Attention (MHA). While substituting MHA with independent Multi-Layer Perceptrons (MLPs) achieves $O(N)$ scaling, it strips the architecture of spatial sequence routing, severely degrading global scene understanding and object permanence. In this paper, we propose the Linear Multi-Timescale Retention (LIA-MTR) module, a memory-efficient cross-modal bridge. By integrating an ELU-based positive feature mapping with adaptive write-gating and log-linearly distributed recurrent decays, LIA-MTR mathematically compresses continuous visual sequences into bounded memory states. Theoretical analysis proves the architecture operates with strict $O(N)$ sequence-interaction complexity. Empirically, synthetic retrieval evaluations demonstrate that LIA-MTR flawlessly routes context across 16,000 tokens, eliminating the "Lost in the Middle" degradation typical of naive linear attention. Hardware benchmarking reveals infinite-context scaling capabilities, natively processing 262,144 visual patches within an 11.2 GB VRAM footprint, whereas standard MHA suffers out-of-memory failure at 16,384 patches. Furthermore, following instruction tuning on 665K conversational samples, LIA-MTR significantly outperforms an industry-standard MLP baseline on the MME benchmark (71.00% vs. 68.11%), driven by a 10% absolute improvement in object permanence and superior global semantic extraction. This work establishes a mathematically rigorous, computationally flat foundation for infinite-context Vision-Language integration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。