用大模型筛选高质量音视频数据，少样本也能训出强模型
Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio-Video Foundation Model
- 用LLM+双编码器对比学习筛选音视频对齐片段
- 仅用192小时数据，检索准确率超全量未筛选数据
- 适合资源有限但追求性能的多模态研究者
将音视频数据融合训练多模态基础模型仍具挑战。AVVA框架通过考虑超越时间同步的音视频场景对齐，并利用大语言模型（LLM）进行数据筛选。该方法设计评分机制，选取对齐的训练数据片段。采用Whisper进行音频分析，DINOv2进行视频分析，在双编码器结构中通过对比学习处理音视频对对。在AudioCaps、VALOR和VGGSound数据集上的评估表明，所提模型架构与数据筛选方法有效。相比DenseAV，AVVA在所有数据集上均显著提升视频到音频检索的top-k准确率，且仅使用192小时精选数据。消融实验显示，数据筛选策略通过提升质量换取数量，相较全量未筛选数据，使AudioCaps、VALOR和VGGSound的top-k检索准确率均有提升。
原文摘要 · Abstract (English)
Integrating audio and visual data for training multimodal foundational models remains a challenge. The Audio-Video Vector Alignment (AVVA) framework addresses this by considering AV scene alignment beyond mere temporal synchronization, and leveraging Large Language Models (LLMs) for data curation. AVVA implements a scoring mechanism for selecting aligned training data segments. It integrates Whisper, a speech-based foundation model, for audio and DINOv2 for video analysis in a dual-encoder structure with contrastive learning on AV pairs. Evaluations on AudioCaps, VALOR, and VGGSound demonstrate the effectiveness of the proposed model architecture and data curation approach. AVVA achieves a significant improvement in top-k accuracies for video-to-audio retrieval on all datasets compared to DenseAV, while using only 192 hrs of curated training data. Furthermore, an ablation study indicates that the data curation process effectively trades data quality for data quantity, yielding increases in top-k retrieval accuracies on AudioCaps, VALOR, and VGGSound, compared to training on the full spectrum of uncurated data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。