通过递归自适应融合视频帧令牌,实现高效低损压缩。
CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models

- 分离选择与融合:先选相似帧,再学权重融合
- 8倍压缩下保留97%准确率,效率显著提升
- 无需训练,适配预训练语言模型,适合视频理解任务
在视频理解中,视觉语言模型需处理大量视觉令牌,导致预填充阶段计算与内存开销急剧上升。这类视觉序列在时空维度上高度冗余,但高压缩比常伴随关键信息丢失。现有令牌压缩方法或采用无训练的启发式策略(内容适应性差),或引入需昂贵对齐训练的额外模块,难以平衡效率与自适应性。为此,我们提出CRAFT:基于递归自适应融合的视频令牌压缩。CRAFT通过解耦无参数的令牌选择与可学习的令牌融合:全局相似性决定合并哪些令牌,位置感知加权模块与内容自适应通道门则学习如何融合。整个压缩流程与查询无关。每个保留令牌均为原始令牌的线性组合,保持真实时空坐标,并与预训练语言模型输入分布对齐。在多个代表性视频基准上的实验表明,CRAFT持续优于现有最先进压缩方法。约8倍压缩时,保留约97%骨干网络平均准确率,显著提升效率。
原文摘要 · Abstract (English)
In video understanding, vision-language models (VLMs) must ingest massive numbers of visual tokens, causing the computational and memory cost of the prefill stage to rise sharply. Such visual sequences are highly redundant along the spatio-temporal dimension, yet a high compression ratio is often accompanied by the loss of critical details. Existing token-compression methods either employ heuristic, training-free compression with limited content adaptivity or introduce additional modules that require expensive alignment training, leaving the trade-off between efficiency and adaptivity unresolved. To alleviate this limitation, we propose CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens. CRAFT recursively merges tokens by decoupling parameter-free token selection from learnable token fusion: global similarity determines which tokens to merge, while a position-aware weighting module and a content-adaptive channel-wise gate learn how to fuse them. The whole compression pipeline is query-agnostic. Because every retained token is a linear combination of the original tokens, CRAFT preserves their true spatio-temporal coordinates and stays aligned with the pre-trained language model's input distribution. Experiments on multiple representative video benchmarks show that CRAFT consistently outperforms prior state-of-the-art token-compression methods. At about $8\times$ compression, it retains roughly $97\%$ of the backbone's average accuracy and shows significant efficiency improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。