用CLIP模型融合视觉与文本信息，实现高效语音活动检测
CLIP-VAD: Exploiting Vision-Language Models for Voice Activity Detection
- 用视频上半身+自动生成文本描述，通过CLIP模型提取特征
- 在三个基准上表现优于现有视觉VAD方法，接近音频视觉融合效果
- 无需大量音视频数据预训练，适合资源受限场景
语音活动检测（VAD）旨在自动判断人是否在说话并定位其语音时间。传统方法仅处理音频或视觉数据，或通过融合/联合学习结合双模态。本文受视觉-语言模型启发，提出一种新方法：利用对比语言-图像预训练（CLIP）模型，由视觉编码器分析个体上半身视频片段，文本编码器处理通过提示工程生成的文本描述，再通过深度神经网络融合编码嵌入以完成VAD。在三个VAD基准上的实验表明，该方法性能优于现有视觉VAD方法，且虽结构简单，但超越多种音视频联合方法，且无需在大规模音视频数据上进行预训练。
原文摘要 · Abstract (English)
Voice Activity Detection (VAD) is the process of automatically determining whether a person is speaking and identifying the timing of their speech in an audiovisual data. Traditionally, this task has been tackled by processing either audio signals or visual data, or by combining both modalities through fusion or joint learning. In our study, drawing inspiration from recent advancements in visual-language models, we introduce a novel approach leveraging Contrastive Language-Image Pretraining (CLIP) models. The CLIP visual encoder analyzes video segments composed of the upper body of an individual, while the text encoder handles textual descriptions automatically generated through prompt engineering. Subsequently, embeddings from these encoders are fused through a deep neural network to perform VAD. Our experimental analysis across three VAD benchmarks showcases the superior performance of our method compared to existing visual VAD approaches. Notably, our approach outperforms several audio-visual methods despite its simplicity, and without requiring pre-training on extensive audio-visual datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。