arXiv:2608.27929cs.CV2026-08

无需训练,将视频切分为可复用的语义时间片段

Training-Free Temporal Abstraction for General Video Understanding

论文配图:Training-Free Temporal Abstraction for General Video Understanding
图 1 · 摘自论文原文
  • 用冻结的视频-文本模型提取短窗口嵌入,检测变化点划分视频段
  • 一次计算后跨任务复用,三类任务表现媲美专用方法
  • 适合资源受限场景,尤其擅长少帧/少令牌时的长视频理解

视频逐帧分析成本高,但多数视频理解任务需定位关键时刻。现有方法多针对特定任务设计,依赖专用数据或架构。本文探讨预训练视频-文本模型能否提供足够的时序结构以支持多种任务。提出STITCH,一种无需训练的方法,通过冻结的视频-文本主干对短视频窗口编码,并检测嵌入序列的变化点来划分语义时间块。这些时间块每视频仅计算一次,可跨任务复用。在通用事件边界检测、基于语言的片段检索、长视频视觉语言模型推理中的帧选择三个任务上,STITCH均保持竞争力,且在仅能处理少量帧或标记时表现尤为突出。结果表明,可复用的时间抽象是通用视频理解的有前景方向,使密集视频流一次性转化为可定位、检索、采样或推理的语义单元。

原文摘要 · Abstract (English)

Videos are expensive to analyze frame by frame, yet many video understanding tasks depend on knowing where relevant moments occur. A system may need to find when an action changes, locate the segment described by a sentence, or choose a few frames for a vision-language model. Existing methods often solve these problems separately, using task-specific training data or specialized architectures. We study whether a pretrained video-text model can provide enough temporal structure to support several of these tasks at once. We present STITCH, a training-free method that divides a video into semantically meaningful temporal chunks. STITCH embeds short video windows with a frozen video-text backbone and detects changes in the resulting embedding sequence. These chunks are computed once per video and reused across tasks. We evaluate STITCH on generic event boundary detection, language-based moment retrieval, and frame selection for long-video VLM reasoning. Across all three settings, STITCH remains competitive with more specialized methods while requiring no task-specific training, with especially clear gains when only a small number of frames or tokens can be processed. These results suggest that reusable temporal abstraction is a promising direction for general video understanding, allowing dense video streams to be converted once into semantic units that can be localized, retrieved, sampled, or reasoned over by downstream systems.

视频理解时间抽象零样本多任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。