arXiv:2511.09868cs.CV2025-11中稿 · AAAI被引 3

解决视觉语言模型长距离依赖遗忘问题,无需训练即可提升全局理解能力。

Remember Me: Bridging the Long-Range Gap in LVLMs with Three-Step Inference-Only Decay Resilience Strategies

  • 三步无训练策略:语义增强、距离控制、远程重强化
  • 在VQA任务上显著提升性能,最长可达30%的准确率增益
  • 适合需要提升长文本/图像上下文理解的场景

大型视觉语言模型(LVLMs)在多模态任务中表现优异,但在使用旋转位置编码(ROPE)时面临长距离依赖建模难题。尽管ROPE能精准定位,但导致远距离标记对的注意力随距离递减,严重削弱模型对全局上下文的记忆能力。为此,我们提出仅推理阶段的三步抗衰减策略(T-DRS):(1) 语义驱动衰减恢复(SD-DRS),通过内容感知残差放大有意义的远距离信号;(2) 距离感知控制衰减恢复(DC-DRS),基于位置距离平滑调节注意力权重,抑制噪声并保留局部性;(3) 远程再强化衰减恢复(reRD-DRS),整合剩余远距离依赖以维持全局连贯性。T-DRS在不破坏局部归纳偏置的前提下恢复被抑制的长距离关系。在视觉问答(VQA)基准上的大量实验表明,该方法可在无训练情况下持续提升性能。代码已开源:https://github.com/labixiaoq-qq/Remember-me

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) have achieved impressive performance across a wide range of multimodal tasks. However, they still face critical challenges in modeling long-range dependencies under the usage of Rotary Positional Encoding (ROPE). Although it can facilitate precise modeling of token positions, it induces progressive attention decay as token distance increases, especially with progressive attention decay over distant token pairs, which severely impairs the model's ability to remember global context. To alleviate this issue, we propose inference-only Three-step Decay Resilience Strategies (T-DRS), comprising (1) Semantic-Driven DRS (SD-DRS), amplifying semantically meaningful but distant signals via content-aware residuals, (2) Distance-aware Control DRS (DC-DRS), which can purify attention by smoothly modulating weights based on positional distances, suppressing noise while preserving locality, and (3) re-Reinforce Distant DRS (reRD-DRS), consolidating the remaining informative remote dependencies to maintain global coherence. Together, the T-DRS recover suppressed long-range token pairs without harming local inductive biases. Extensive experiments on Vision Question Answering (VQA) benchmarks demonstrate that T-DRS can consistently improve performance in a training-free manner. The code can be accessed in https://github.com/labixiaoq-qq/Remember-me

视觉语言模型长距离依赖位置编码无训练优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。