用CLIP和多头注意力自动识别视频片头片尾,准确率超90%。
Automatic Detection of Intro and Credits in Video using CLIP and Multihead Attention
- 将视频每秒帧编码后,用CLIP结合多头注意力分类
- 测试集上F1达91.0%,召回率达97.0%
- 适合需要自动剪辑、摘要的视频平台使用
视频中片头片尾与正片之间的切换检测对内容分割、索引和推荐系统至关重要。人工标注耗时易错,传统启发式方法难以跨风格泛化。本文提出一种基于深度学习的方法,将问题建模为序列到序列分类任务:每秒视频帧被标记为“intro”或“film”。系统以1 FPS固定速率提取帧,用CLIP编码图像特征,并通过带可学习位置编码的多头注意力模型处理。在测试集上,该方法达到F1分数91.0%、精确率89.0%、召回率97.0%。模型支持实时推理,在CPU上达11.5 FPS,高端GPU上达107 FPS。该方法可应用于自动化内容索引、精彩片段检测与视频摘要。未来将探索融合音频和字幕的多模态学习以进一步提升性能。
原文摘要 · Abstract (English)
Detecting transitions between intro/credits and main content in videos is a crucial task for content segmentation, indexing, and recommendation systems. Manual annotation of such transitions is labor-intensive and error-prone, while heuristic-based methods often fail to generalize across diverse video styles. In this work, we introduce a deep learning-based approach that formulates the problem as a sequence-to-sequence classification task, where each second of a video is labeled as either "intro" or "film." Our method extracts frames at a fixed rate of 1 FPS, encodes them using CLIP (Contrastive Language-Image Pretraining), and processes the resulting feature representations with a multihead attention model incorporating learned positional encoding. The system achieves an F1-score of 91.0%, Precision of 89.0%, and Recall of 97.0% on the test set, and is optimized for real-time inference, achieving 11.5 FPS on CPU and 107 FPS on high-end GPUs. This approach has practical applications in automated content indexing, highlight detection, and video summarization. Future work will explore multimodal learning, incorporating audio features and subtitles to further enhance detection accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。