arXiv:2604.14149cs.CV2026-04NeurIPS被引 4

提出极端视频压缩方法,实现每帧仅1个标记,显著提升长视频理解性能。

One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding

论文配图:One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding
图 1 · 摘自论文原文
  • 设计可学习的分层压缩模块,实现逐标记压缩以减少信息损失。
  • 通过查询感知选帧机制,使模型在极低标记数下仍能保持高精度。
  • 仅用2.5%微调数据即提升46.2%准确率,适合资源受限场景使用。

长视频理解对视觉语言模型(VLM)构成挑战,因帧数庞大导致标记量激增,而大语言模型(LLM)上下文长度有限,迫使模型稀疏感知并丢失时间信息。为此,本文探索极端视频标记压缩,目标为在最终LLM层实现每帧仅1个标记。核心洞察是:传统基于启发式的压缩易造成信息丢失,需将LLM层转化为可学习、渐进式的标记级压缩模块(LP-Comp),从而支持2倍至4倍更多帧的处理且性能提升。为进一步提高标记效率,提出基于查询的帧级压缩(QC-Comp),利用LLM层内部注意力分数筛选与查询最相关的帧。同时,通过分段+局部注意力缓解长序列中的位置偏见问题。综合两种压缩策略,提出名为XComp的极端压缩模型,在仅需2.5%监督微调数据的情况下,将VideoChat-Flash微调后,在LVBench上准确率从42.9%提升至46.2%,并在多个长视频基准测试中表现优异。

原文摘要 · Abstract (English)

Long video understanding is inherently challenging for vision-language models (VLMs) because of the extensive number of frames. With each video frame typically expanding into tens or hundreds of tokens, the limited context length of large language models (LLMs) forces the VLMs to perceive the frames sparsely and lose temporal information. To address this, we explore extreme video token compression towards one token per frame at the final LLM layer. Our key insight is that heuristic-based compression, widely adopted by previous methods, is prone to information loss, and this necessitates supervising LLM layers into learnable and progressive modules for token-level compression (LP-Comp). Such compression enables our VLM to digest 2x-4x more frames with improved performance. To further increase the token efficiency, we investigate frame-level compression, which selects the frames most relevant to the queries via the internal attention scores of the LLM layers, named question-conditioned compression (QC-Comp). As a notable distinction from previous studies, we mitigate the position bias of LLM attention in long contexts, i.e., the over-concentration on the beginning and end of a sequence, by splitting long videos into short segments and employing local attention. Collectively, our combined token-level and frame-level leads to an extreme compression model for long video understanding, named XComp, achieving a significantly larger compression ratio and enabling denser frame sampling. Our XComp is finetuned from VideoChat-Flash with a data-efficient supervised compression tuning stage that only requires 2.5% of the supervised fine-tuning data, yet boosts the accuracy from 42.9% to 46.2% on LVBench and enhances multiple other long video benchmarks.

视频理解极端压缩多模态高效微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。