让视频理解模型突破长时序瓶颈，动态聚焦关键帧。
VideoNSA: Native Sparse Attention Scales Video Understanding
- 用稀疏注意力机制只关注视频中重要帧，节省计算资源。
- 在128K令牌下仍保持稳定性能，超越传统压缩方法。
- 适合需要处理超长视频的多模态研究者与工程师。
多模态语言模型在视频理解方面受限于上下文长度：模型常遗漏关键转换帧，难以维持长时序连贯性。为此，我们引入原生稀疏注意力（NSA）并应用于视频-语言模型。通过在包含216,000条视频指令的数据集上对Qwen2.5-VL进行端到端训练，提出VideoNSA方法。采用硬件感知的混合注意力策略，保留文本部分的密集注意力，视频部分则使用NSA。相比基于令牌压缩和无需训练的稀疏基线，VideoNSA在长视频理解、时间推理和空间基准任务上表现更优。消融分析揭示四个关键发现：(1) 可可靠扩展至128K令牌；(2) 在固定预算下存在最优全局-局部注意力分配；(3) 不同任务有差异化的分支使用模式；(4) 可学习的联合稀疏注意力能诱导动态注意力汇聚点。
原文摘要 · Abstract (English)
Video understanding in multimodal language models remains limited by context length: models often miss key transition frames and struggle to maintain coherence across long time scales. To address this, we adapt Native Sparse Attention (NSA) to video-language models. Our method, VideoNSA, adapts Qwen2.5-VL through end-to-end training on a 216K video instruction dataset. We employ a hardware-aware hybrid approach to attention, preserving dense attention for text, while employing NSA for video. Compared to token-compression and training-free sparse baselines, VideoNSA achieves improved performance on long-video understanding, temporal reasoning, and spatial benchmarks. Further ablation analysis reveals four key findings: (1) reliable scaling to 128K tokens; (2) an optimal global-local attention allocation at a fixed budget; (3) task-dependent branch usage patterns; and (4) the learnable combined sparse attention help induce dynamic attention sinks. Project Page: https://enxinsong.com/VideoNSA-web/, Code: https://github.com/Espere-1119-Song/VideoNSA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。