arXiv:2503.16036cs.CVcs.AI2025-03CVPR被引 24

用指令指导视频压缩,保留关键信息同时大幅减少计算量。

Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Models

  • 根据用户指令在局部和全局层面动态压缩视频帧
  • 在三个问答基准上提升2.43%性能,节省78.8%视觉令牌
  • 适合需要高效视频理解的多模态大模型应用

近期多模态大语言模型(MLLMs)面临海量视频帧带来的计算开销问题,通常通过压缩策略缓解。然而,现有方法(如平均池化)未考虑视觉内容对用户指令的相关性,导致可能有用信息丢失。为此,我们提出一种混合层级指令注入策略(HICom),利用指令作为条件,在局部和全局层面引导视频标记的条件压缩,以保留最多用户关注信息的同时减少计算负担。具体地,将指令条件注入局部分组视觉标记与全局可学习标记中,并通过注意力机制完成压缩。该混合压缩策略既突出指令相关视觉部分,又保留时空结构,便于大模型理解。为进一步释放HICom潜力,我们构建了新数据集HICom-248K,并引入新的条件预训练阶段。实验表明,相比当前最优方法,HICom在三个多项选择题基准上平均提升2.43%性能,同时减少78.8%视觉标记。代码已开源。

原文摘要 · Abstract (English)

Recent Multi-modal Large Language Models (MLLMs) have been challenged by the computational overhead resulting from massive video frames, often alleviated through compression strategies. However, the visual content is not equally contributed to user instructions, existing strategies (\eg, average pool) inevitably lead to the loss of potentially useful information. To tackle this, we propose the Hybrid-level Instruction Injection Strategy for Conditional Token Compression in MLLMs (HICom), utilizing the instruction as a condition to guide the compression from both local and global levels. This encourages the compression to retain the maximum amount of user-focused information while reducing visual tokens to minimize computational burden. Specifically, the instruction condition is injected into the grouped visual tokens at the local level and the learnable tokens at the global level, and we conduct the attention mechanism to complete the conditional compression. From the hybrid-level compression, the instruction-relevant visual parts are highlighted while the temporal-spatial structure is also preserved for easier understanding of LLMs. To further unleash the potential of HICom, we introduce a new conditional pre-training stage with our proposed dataset HICom-248K. Experiments show that our HICom can obtain distinguished video understanding ability with fewer tokens, increasing the performance by 2.43\% average on three multiple-choice QA benchmarks and saving 78.8\% tokens compared with the SOTA method. The code is available at https://github.com/lntzm/HICom.

视频压缩多模态模型指令注入令牌优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。