用大模型自身注意力机制动态压缩视频帧，提升效率又不丢精度。
Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior
- 利用大模型注意力机制自动识别关键帧，动态调整保留比例。
- 在多个视频大模型上实现4.3倍推理加速，精度几乎不变。
- 无需训练，可无缝接入现有压缩方法，适合部署优化场景。
视频大语言模型（VLLM）虽具备强大视频理解能力，但长视频的视觉标记序列导致计算量呈二次增长，效率瓶颈突出。现有关键帧采样方法需额外计算开销，且二值化选帧策略效果不佳。为此，本文提出无需训练的动态标记压缩方法DyToK，利用VLLM自身的注意力机制挖掘查询相关的关键帧先验。分析发现，注意力层天然编码了与查询相关的关键帧优先级信息，使DyToK能动态调整每帧标记保留率，优先保留语义丰富帧，抑制冗余内容。大量实验表明，DyToK在多个VLLM（如LLaVA-OneVision、Qwen2.5-VL）上均达到领先的效率-精度平衡。其具备即插即用特性，可与VisionZip、FastV等压缩方法结合，实现4.3倍推理加速，同时保持原有精度。代码已开源。
原文摘要 · Abstract (English)
Recent advances in Video Large Language Models (VLLMs) have achieved remarkable video understanding capabilities, yet face critical efficiency bottlenecks due to quadratic computational growth with lengthy visual token sequences of long videos. While existing keyframe sampling methods can improve temporal modeling efficiency, additional computational cost is introduced before feature encoding, and the binary frame selection paradigm is found suboptimal. Therefore, in this work, we propose Dynamic Token compression via LLM-guided Keyframe prior (DyToK), a training-free paradigm that enables dynamic token compression by harnessing VLLMs' inherent attention mechanisms. Our analysis reveals that VLLM attention layers naturally encoding query-conditioned keyframe priors, by which DyToK dynamically adjusts per-frame token retention ratios, prioritizing semantically rich frames while suppressing redundancies. Extensive experiments demonstrate that DyToK achieves state-of-the-art efficiency-accuracy tradeoffs. DyToK shows plug-and-play compatibility with existing compression methods, such as VisionZip and FastV, attaining 4.3x faster inference while preserving accuracy across multiple VLLMs, such as LLaVA-OneVision and Qwen2.5-VL. Code is available at https://github.com/yu-lin-li/DyToK .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。