轻量视频编码器让视频大模型处理更长视频，提速35%且多处理8倍帧数。
LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs
- 用压缩令牌蒸馏训练轻量编码器，跳过冗余计算。
- 相比InternVL3-8B，端到端延迟降35%，可处理8倍帧数。
- 适合资源受限下需长视频理解的场景。
视频大模型在处理长视频时面临视觉标记数量爆炸的问题。现有方法多采用事后令牌压缩，虽减少视觉标记数，但使延迟瓶颈从语言模型转移至昂贵的每帧视觉编码。为此，我们提出LiteFrame，一种高效强健的视频编码主干网络。通过提出压缩令牌蒸馏（CTD）训练框架，让小型学生编码器直接学习大型教师模型生成的时空压缩高信息密度表示，有效规避冗余计算。结合语言模型适配（LMA），该方案实现新的延迟-精度权衡：相较InternVL3-8B，在固定算力下端到端延迟降低35%，帧数处理能力提升8倍，并在多个基准上提升平均视频理解准确率。结果表明，该路径为在有限算力下实现更长视频理解提供了新可能。
原文摘要 · Abstract (English)
The fundamental challenge in scaling Video Large Language Models (Video LLMs) to long-form video lies in managing the explosion of visual-token context length. Existing strategies predominantly focus on "post-hoc" token reduction -- reducing visual tokens after feature extraction to alleviate the LLM's computational overhead. While these methods effectively reduce the number of visual tokens, we observe that the primary latency bottleneck then shifts from the LLM to the expensive per-frame processing of the vision encoder. To address this, we introduce LiteFrame, a strong, yet highly efficient video encoder backbone for Video LLMs. To train LiteFrame, we propose Compressed Token Distillation (CTD), a novel training framework that teaches a compact student vision encoder to directly predict information-dense, spatio-temporally compressed representations produced by a large teacher vision model, effectively bypassing redundant computation. When coupled with further Language Model Adaptation (LMA), this approach results in a new latency-accuracy Pareto frontier -- compared with InternVL3-8B, LiteFrame provides a 35% reduction in end-to-end latency while processing 8$\times$ more frames and improves average video understanding accuracy across multiple benchmarks. Our results demonstrate a new potential path to unlocking longer-form video understanding under fixed compute budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。