arXiv:2605.22158cs.AIcs.CV2026-05中稿 · ICLR被引 3

用相似性与差异性协同压缩视频,提升多模态大模型效率

ST-SimDiff: Balancing Spatiotemporal Similarity and Difference for Efficient Video Understanding with MLLMs

论文配图:ST-SimDiff: Balancing Spatiotemporal Similarity and Difference for Efficient Video Understanding with MLLMs
图 1 · 摘自论文原文
  • 构建时空图,用社区检测保留静态冗余,差分定位动态关键点
  • 仅用1/4令牌实现优于主流方法的视频理解性能,计算成本大幅降低
  • 无需训练,适合长视频处理场景,尤其适合资源受限的部署

多模态大语言模型在处理长视频时面临巨大计算开销,因其需大量视觉标记。现有方法主要通过重要性或相似性剪枝合并标记以减少冗余,但忽视了内容变化和转折点这一关键维度,且缺乏对时空关系的协同建模。为此,我们提出新视角:相似性用于识别冗余,差异性用于捕捉关键事件。据此设计无需训练的ST-SimDiff框架。首先从视觉标记构建时空图,统一建模复杂关联;随后采用并行双选策略:1)基于相似性的选择利用社区检测保留代表性标记,压缩静态信息;2)基于时间差分的选择精准定位内容变化点,保留反映关键动态转移的标记。该方法以极少量标记同时保留静态与动态内容。大量实验表明,本方法显著优于当前最优方案,同时大幅降低计算成本。代码已开源。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) face significant computational overhead when processing long videos due to the massive number of visual tokens required. To improve efficiency, existing methods primarily reduce redundancy by pruning or merging tokens based on importance or similarity. However, these approaches largely overlook a critical dimension of video content, i.e., changes and turning points, and they lack a collaborative model for spatio-temporal relationships. To address this, we propose a new perspective: similarity is for identifying redundancy, while difference is for capturing key events. Based on this, we designed a training-free framework named ST-SimDiff. We first construct a spatio-temporal graph from the visual tokens to uniformly model their complex associations. Subsequently, we employ a parallel dual-selection strategy: 1) similarity-based selection uses community detection to retain representative tokens, compressing static information; 2) temporal difference-based selection precisely locates content-changing points to preserve tokens that capture key dynamic shifts. This allows it to preserve both static and dynamic content with a minimal number of tokens. Extensive experiments show our method significantly outperforms state-of-the-art approaches while substantially reducing computational costs. Our code is available in https://github.com/bingjunluo/ST-SimDiff.

视频理解多模态模型高效推理时空建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。