arXiv:2512.12560cs.CVcs.AI2025-12被引 10

通过智能剪枝视频帧,实现快速在线视频理解。

StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understanding

  • 基于空间邻近性设计新冗余度量,精准识别可删帧。
  • 剪枝后准确率最高提升4%,延迟低于1毫秒。
  • 适合实时监控、AI眼镜等低延迟场景使用。

在线视频理解在公共监控和AI眼镜等应用中至关重要。然而,将多模态大语言模型(MLLMs)应用于该领域面临视频帧数量庞大导致的高显存占用与计算延迟问题。为此,本文提出一种令牌剪枝方法,在减少上下文长度的同时保留关键信息。具体地,引入一种新型冗余度量——最大空间邻近帧相似性(MSSAVT),同时考虑帧间相似性和空间位置关系。为解决剪枝与冗余评估间的双向依赖,设计掩码剪枝策略,确保仅互不相邻的帧被剪枝。此外,融合现有时间冗余剪枝方法以消除视频模态的时间冗余。在多个在线与离线视频理解基准测试中,本方法显著提升准确率(最高提升4%),且剪枝延迟极低(<1ms)。完整代码将公开。

原文摘要 · Abstract (English)

Online video understanding is essential for applications like public surveillance and AI glasses. However, applying Multimodal Large Language Models (MLLMs) to this domain is challenging due to the large number of video frames, resulting in high GPU memory usage and computational latency. To address these challenges, we propose token pruning as a means to reduce context length while retaining critical information. Specifically, we introduce a novel redundancy metric, Maximum Similarity to Spatially Adjacent Video Tokens (MSSAVT), which accounts for both token similarity and spatial position. To mitigate the bidirectional dependency between pruning and redundancy, we further design a masked pruning strategy that ensures only mutually unadjacent tokens are pruned. We also integrate an existing temporal redundancy-based pruning method to eliminate temporal redundancy of the video modality. Experimental results on multiple online and offline video understanding benchmarks demonstrate that our method significantly improves the accuracy (i.e., by 4\% at most) while incurring a negligible pruning latency (i.e., less than 1ms). Our full implementation will be made publicly available.

视频理解剪枝高效推理多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。