arXiv:2410.02768cs.CVcs.AI2024-10被引 5

用大模型自动生成问题并过滤低质量样本,提升视频与文本对齐效果。

Uncertainty-Guided Self-Questioning and Answering for Video-Language Alignment

  • 通过大模型自动生成视频问题,扩充训练数据
  • 引入不确定性评估机制,筛选高质量自生成问题
  • 在5个视频问答数据集上超越主流方法,适合多模态研究者

多模态模型发展迅速,但视频-文本对标注成本高且覆盖不全,常导致视频信息未被充分使用。为此,我们提出自训练框架BoViLA,利用大语言模型生成自问自答样本来增强训练数据,帮助模型更深入挖掘视频内容和大模型内部知识,从而提升模态对齐效果。然而,早期训练阶段生成的低质量问题可能干扰性能。为此,我们引入证据深度学习(EDL)评估自生成问题的不确定性,基于上下文中的模态对齐情况判断其质量。据我们所知,这是首个探索大模型自训练框架用于模态对齐的工作。我们在五个强基准视频问答任务上评估了BoViLA,结果优于多个先进方法,验证了其有效性与通用性。此外,还提供了对自训练框架及不确定性过滤机制的详细分析。代码将公开。

原文摘要 · Abstract (English)

The development of multi-modal models has been rapidly advancing, with some demonstrating remarkable capabilities. However, annotating video-text pairs remains expensive and insufficient. Take video question answering (VideoQA) tasks as an example, human annotated questions and answers often cover only part of the video, since the corresponding text is often short and monotonous, leading to underutilization of video. To address this, we propose a Bootstrapping Video-Language Alignment framework (BoViLA), a self-training method that augments question samples during training process through LLM-based self-questioning and answering, which help model exploit video information and the internal knowledge of LLMs more thoroughly to improve modality alignment. However, low-quality self-generated questions may instead contaminate the performance, especially in the early stages of training, as we have observed in our experiments. To filter bad self-generated questions, we introduce Evidential Deep Learning (EDL) to estimate uncertainty and assess the quality of self-generated questions by evaluating the modality alignment within the context. To the best of our knowledge, this work is the first to explore LLM-based self-training frameworks for modality alignment. We evaluate BoViLA on five strong VideoQA benchmarks, where it outperforms several state-of-the-art methods and demonstrate its effectiveness and generality. Additionally, we provide extensive analyses of the self-training framework and the EDL-based uncertainty filtering mechanism. The code will be made available.

视频问答自训练大模型模态对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。