提出TransVLM框架，用视觉语言模型精准识别视频转场时段。
TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions
- 引入光流作为运动先验，融合颜色与运动特征提升时间感知。
- 在公开数据集上超越传统方法和顶级视觉语言模型性能。
- 构建了可扩展的合成数据引擎，解决训练数据不平衡问题。
传统镜头边界检测（SBD）因仅关注孤立切点，常导致视频片段失真。本文提出镜头转场检测（STD）新任务，不再寻找模糊点，而是显式识别连续的时间转场段。为此，我们设计了TransVLM——一种将光流作为关键运动先验注入输入的视觉语言模型（VLM）。通过简单的特征融合策略，直接处理拼接的颜色与运动表示，在不增加语言主干视觉标记负担的前提下显著增强时间感知能力。为应对公共数据中严重的类别不平衡，我们开发了一个可扩展的数据生成引擎，用于合成多样化转场视频，并构建了完整的STD基准测试。大量实验表明，TransVLM在整体性能上优于传统启发式方法、专用时空网络及顶尖VLM。更多研究请访问HeyGen Research（https://www.heygen.com/research）及HeyGen Avatar-V（https://www.heygen.com/research/avatar-v-model）。项目页：https://chence17.github.io/TransVLM/
原文摘要 · Abstract (English)
Traditional Shot Boundary Detection (SBD) inherently struggles with complex transitions by formulating the task around isolated cut points, frequently yielding corrupted video shots. We address this fundamental limitation by formalizing the Shot Transition Detection (STD) task. Rather than searching for ambiguous points, STD explicitly detects the continuous temporal segments of transitions. To tackle this, we propose TransVLM, a Vision-Language Model (VLM) framework for STD. Unlike regular VLMs that predominantly rely on spatial semantics and struggle with fine-grained inter-shot dynamics, our method explicitly injects optical flow as a critical motion prior at the input stage. Through a simple yet effective feature-fusion strategy, TransVLM directly processes concatenated color and motion representations, significantly enhancing its temporal awareness without incurring any additional visual token overhead on the language backbone. To overcome the severe class imbalance in public data, we design a scalable data engine to synthesize diverse transition videos for robust training, alongside a comprehensive benchmark for STD. Extensive experiments demonstrate that TransVLM achieves superior overall performance, outperforming traditional heuristic methods, specialized spatiotemporal networks, and top-tier VLMs. For more related research, please visit HeyGen Research (https://www.heygen.com/research) and HeyGen Avatar-V (https://www.heygen.com/research/avatar-v-model). Project page: https://chence17.github.io/TransVLM/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。