用对齐语言的视觉特征，让视频摘要更精准。
Multimodal Abstractive Summarization of Instructional Videos with Vision-Language Models
- 用冻结的CLIP特征+时间建模，实现视觉与语言语义对齐。
- 在YouCook2上达33.0% ROUGE-1，仅需1/4特征维度。
- 预训练对齐比微调更有效，适合多模态摘要任务。
多模态视频摘要需要视觉特征与语言生成在语义上对齐。传统方法依赖用于物体分类训练的CNN特征，将视觉概念表示为离散类别，与自然语言不一致。我们提出ClipSum框架，利用冻结的CLIP视觉-语言特征，结合显式时间建模和维度自适应融合，用于教学类视频摘要。CLIP在4亿张图像-文本对上进行对比预训练，其视觉特征与文本解码器生成的语言概念在语义上对齐，从表示层面弥合了视觉-语言鸿沟。在YouCook2数据集上，ClipSum达到33.0% ROUGE-1，相比ResNet-152的30.5%提升显著，且仅需512维（低于ResNet-152的2048维），表明语义对齐比特征容量更重要。冻结的CLIP（33.0%）优于微调的CLIP（32.3%），说明保留预训练对齐比任务适配更有价值。
原文摘要 · Abstract (English)
Multimodal video summarization requires visual features that align semantically with language generation. Traditional approaches rely on CNN features trained for object classification, which represent visual concepts as discrete categories not aligned with natural language. We propose ClipSum, a framework that leverages frozen CLIP vision-language features with explicit temporal modeling and dimension-adaptive fusion for instructional video summarization. CLIP's contrastive pre-training on 400M image-text pairs yields visual features semantically aligned with the linguistic concepts that text decoders generate, bridging the vision-language gap at the representation level. On YouCook2, ClipSum achieves 33.0% ROUGE-1 versus 30.5% for ResNet-152 with 4x lower dimensionality (512 vs. 2048), demonstrating that semantic alignment matters more than feature capacity. Frozen CLIP (33.0%) surpasses fine-tuned CLIP (32.3%), showing that preserving pre-trained alignment is more valuable than task-specific adaptation. https://github.com/aqeeelmirza/clipsum
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。