AdaSpark通过动态选择视频块和关键帧，高效处理长视频同时保持细节。
AdaSpark: Adaptive Sparsity for Efficient Long-Video Understanding
- 按时空分块并动态选择需计算的视频块与关键帧。
- 计算量降低57% FLOPs，性能接近全量模型。
- 适合需要长时序理解的视频分析任务。
用视频大语言模型处理长视频计算成本高昂。现有高效方法常因不可逆信息丢弃损害细粒度感知，或因固定稀疏模式抑制长时依赖建模。本文提出AdaSpark，一种自适应稀疏框架：先将视频输入划分为3D时空立方体，再设计两个协同的上下文感知组件——（1）自适应立方体选择注意力（AdaS-Attn），根据查询动态选择相关立方体进行注意力计算；（2）自适应标记选择前馈网络（AdaS-FFN），仅处理每个立方体内最显著的标记。基于熵的Top-p机制根据输入复杂度动态分配算力。实验表明，AdaSpark在小时级视频基准上将计算量减少高达57% FLOPs，性能媲美密集模型，并有效保留细粒度与长程依赖关系。
原文摘要 · Abstract (English)
Processing long-form videos with Video Large Language Models (Video-LLMs) is computationally prohibitive. Current efficiency methods often compromise fine-grained perception through irreversible information disposal or inhibit long-range temporal modeling via rigid, predefined sparse patterns. This paper introduces AdaSpark, an adaptive sparsity framework designed to address these limitations. AdaSpark first partitions video inputs into 3D spatio-temporal cubes. It then employs two co-designed, context-aware components: (1) Adaptive Cube-Selective Attention (AdaS-Attn), which adaptively selects a subset of relevant video cubes to attend for each query token, and (2) Adaptive Token-Selective FFN (AdaS-FFN), which selectively processes only the most salient tokens within each cube. An entropy-based (Top-p) selection mechanism adaptively allocates computational resources based on input complexity. Experiments demonstrate that AdaSpark significantly reduces computational load by up to 57% FLOPs while maintaining comparable performance to dense models and preserving fine-grained, long-range dependencies, as validated on challenging hour-scale video benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。