提出慢快架构,让视频模型用更多帧同时保留细节。
Slow-Fast Architecture for Video Multi-Modal Large Language Models

- 用快慢双路视觉令牌,分别处理全局概览和指令相关细节。
- 输入帧数从16增至128,计算增3%却提升16%平均性能。
- 可插件式接入其他模型,适合追求效率的视频理解研究者。
在有限算力下平衡视频的时间分辨率与空间细节仍是视频多模态大模型的关键挑战。现有方法通常在输入大模型前按预设规则压缩视频表征,导致信息不可逆丢失且常忽略输入指令。为此,我们提出一种新颖的慢-快架构,自然规避这一权衡,实现更多输入帧的同时保持空间细节。受人类先快速浏览视频再聚焦关键部分的启发,该设计采用双令牌策略:1)“快”视觉令牌——一组紧凑的压缩视频特征——与文本嵌入一同输入大模型,提供快速概览;2)“慢”视觉令牌——未压缩的视频特征——通过专门设计的混合解码层与文本嵌入交叉注意力,实现指令感知的细节提取,复杂度为线性。我们系统性地优化了整体架构与关键组件。实验表明,该模型显著优于仅使用自注意力的基线,输入容量从16帧扩展至128帧,仅增加3%计算量,五个视频理解基准上平均性能提升16%。我们的7B模型在同类规模中达到最先进水平。此外,该慢-快架构为即插即用设计,可集成到其他视频多模态大模型以提升效率与可扩展性。
原文摘要 · Abstract (English)
Balancing temporal resolution and spatial detail under limited compute budget remains a key challenge for video-based multi-modal large language models (MLLMs). Existing methods typically compress video representations using predefined rules before feeding them into the LLM, resulting in irreversible information loss and often ignoring input instructions. To address this, we propose a novel slow-fast architecture that naturally circumvents this trade-off, enabling the use of more input frames while preserving spatial details. Inspired by how humans first skim a video before focusing on relevant parts, our slow-fast design employs a dual-token strategy: 1) "fast" visual tokens -- a compact set of compressed video features -- are fed into the LLM alongside text embeddings to provide a quick overview; 2) "slow" visual tokens -- uncompressed video features -- are cross-attended by text embeddings through specially designed hybrid decoder layers, enabling instruction-aware extraction of relevant visual details with linear complexity. We conduct systematic exploration to optimize both the overall architecture and key components. Experiments show that our model significantly outperforms self-attention-only baselines, extending the input capacity from 16 to 128 frames with just a 3% increase in computation, and achieving a 16% average performance improvement across five video understanding benchmarks. Our 7B model achieves state-of-the-art performance among models of similar size. Furthermore, our slow-fast architecture is a plug-and-play design that can be integrated into other video MLLMs to improve efficiency and scalability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。