VideoLights通过多模块协同提升视频关键帧检测与片段检索的精度。
VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval
- 引入特征精炼与跨任务对齐机制，强化视频与文本特征一致性。
- 在三个基准上实现新最优性能，最高提升6.2%（QVHighlights）。
- 适合关注视频理解与多模态对齐的研究者和应用开发者。
现有联合预测模型在处理跨任务动态、实现强视频-文本对齐及有效注意力机制方面存在不足，且大语言/视觉-语言模型（LLMs/LVLMs）的潜力尚未充分挖掘。本文提出 VideoLights 框架，包含：(i) 卷积投影与特征精炼模块，结合对齐损失增强视频-文本特征一致性；(ii) 双向跨模态融合网络，生成强耦合的查询感知表示；(iii) 单向联合任务反馈机制，促进任务间协同优化；(iv) 硬正负样本损失，实现自适应学习；(v) 利用 LVLMs（如 BLIP-2）进行优异的多模态特征融合与合成数据上的智能预训练。在 QVHighlights、TVSum 与 Charades-STA 基准上的全面评估表明，VideoLights 显著超越现有基线，建立新最优性能。代码与模型检查点见 https://github.com/dpaul06/VideoLights。
原文摘要 · Abstract (English)
Prevailing joint prediction transformers for Video Highlight Detection and Moment Retrieval (HD/MR) exhibit deficiencies in handling cross-task dynamics, achieving robust video-text alignment, and utilizing effective attention mechanisms, with the potential of Large Language/Vision-Language Models (LLMs/LVLMs) being largely untapped. This paper introduces VideoLights, a novel HD/MR framework addressing these limitations by incorporating: (i) Convolutional Projection and Feature Refinement modules with an alignment loss for enhanced video-text feature congruity; (ii) a Bi-Directional Cross-Modal Fusion network for strongly coupled query-aware representations; (iii) a Uni-directional joint-task feedback mechanism for synergistic task improvement; (iv) hard positive/negative losses for adaptive learning; and (v) the leveraging of LVLMs (e.g., BLIP-2) for superior multimodal feature integration and intelligent pre-training with synthetic data. Comprehensive evaluations on QVHighlights, TVSum, and Charades-STA benchmarks demonstrate that VideoLights significantly surpasses existing baselines, establishing new state-of-the-art performances. Codes and model checkpoints are available at https://github.com/dpaul06/VideoLights .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。