arXiv:2503.11205cs.CV2025-03被引 6

提出新方法让冻结的图像模型读懂视频,既纠正注意力偏差又利用它提升效果。

LLaVA-MLB: Mitigating and Leveraging Attention Bias for Training-Free Video LLMs

  • 基于注意力得分分两阶段处理视频:压缩时保留时空结构,扩展时利用偏差
  • 在多个基准上超越现有方法,兼顾效率与准确率
  • 适合想用现成模型做视频理解的研究者和开发者

训练自由的视频大语言模型通过预训练图像大模型处理视频内容,无需额外训练。核心挑战在于受限于图像模型的令牌长度,难以保留关键视觉与时间信息。为此,我们提出一种基于大模型注意力分数的两阶段查询相关令牌选择方法:先压缩视频序列,再扩展序列。然而,在压缩阶段,图像模型常表现出位置注意力偏差,过度聚焦于后期帧,导致早期帧信息被忽略。为缓解此问题,我们提出网格化注意力池化以保持时空结构;同时引入视觉摘要尾部,有效利用该偏差,促进序列扩展中的整体视频理解。本方法(LLaVA-MLB)实现了注意力偏差的缓解与利用,使冻结的图像模型能进行细致的视频理解。在多个基准上的实验表明,该方法优于当前最优方案,在效率与准确率上均表现更优。代码将公开。

原文摘要 · Abstract (English)

Training-free video large language models (LLMs) leverage pretrained Image LLMs to process video content without the need for further training. A key challenge in such approaches is the difficulty of retaining essential visual and temporal information, constrained by the token limits in Image LLMs. To address this, we propose a two-stage method for selecting query-relevant tokens based on the LLM attention scores: compressing the video sequence and then expanding the sequence. However, during the compression stage, Image LLMs often exhibit a positional attention bias in video sequences, where attention is overly concentrated on later frames, causing early-frame information to be underutilized. To alleviate this attention bias during sequence compression, we propose Gridded Attention Pooling for preserving spatiotemporal structure. Additionally, we introduce Visual Summarization Tail to effectively utilize this bias, facilitating overall video understanding during sequence expansion. In this way, our method effectively Mitigates and Leverages attention Bias (LLaVA-MLB), enabling the frozen Image LLM for detailed video understanding. Experiments on several benchmarks demonstrate that our approach outperforms state-of-the-art methods, achieving superior performance in both efficiency and accuracy. Our code will be released.

视频理解注意力偏差零样本训练大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。