提出LD-DETR模型,提升视频片段检索与亮点检测的准确率。
LD-DETR: Loop Decoder DEtection TRansformer for Video Moment Retrieval and Highlight Detection
- 用身份矩阵蒸馏相似性,减少语义重叠干扰
- 融合卷积层高效提取视频局部特征
- 循环解码器增强多模态信息解码能力
视频片段检索与亮点检测旨在根据文本查询定位视频中的对应内容。现有模型通常先通过对比学习对齐视频与文本特征,再融合并提取多模态信息,最后使用Transformer解码器解码。但存在三方面问题:(1) 数据集中不同样本间存在重叠语义信息,影响多模态对齐性能;(2) 现有模型难以高效提取视频局部特征;(3) 传统Transformer解码器无法充分解码多模态特征。为此,本文提出LD-DETR模型。首先将相似性矩阵蒸馏为身份矩阵,缓解语义重叠影响;其次设计方法使卷积层更高效提取多模态局部特征;最后将Transformer解码器输出反馈自身,实现充分解码。在四个公开基准数据集上评估,实验表明本模型在QVHighlight、Charades-STA和TACoS上优于当前最先进方法。代码已开源。
原文摘要 · Abstract (English)
Video Moment Retrieval and Highlight Detection aim to find corresponding content in the video based on a text query. Existing models usually first use contrastive learning methods to align video and text features, then fuse and extract multimodal information, and finally use a Transformer Decoder to decode multimodal information. However, existing methods face several issues: (1) Overlapping semantic information between different samples in the dataset hinders the model's multimodal aligning performance; (2) Existing models are not able to efficiently extract local features of the video; (3) The Transformer Decoder used by the existing model cannot adequately decode multimodal features. To address the above issues, we proposed the LD-DETR model for Video Moment Retrieval and Highlight Detection tasks. Specifically, we first distilled the similarity matrix into the identity matrix to mitigate the impact of overlapping semantic information. Then, we designed a method that enables convolutional layers to extract multimodal local features more efficiently. Finally, we fed the output of the Transformer Decoder back into itself to adequately decode multimodal information. We evaluated LD-DETR on four public benchmarks and conducted extensive experiments to demonstrate the superiority and effectiveness of our approach. Our model outperforms the State-Of-The-Art models on QVHighlight, Charades-STA and TACoS datasets. Our code is available at https://github.com/qingchen239/ld-detr.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。