arXiv:2411.07516cs.CVcs.CL2024-11被引 1

让AI看课件时不仅能答问题,还能给出详细解释。

SparrowVQE: Visual Question Explanation for Course Content Understanding

  • 用课件图文与讲稿对齐训练小模型,实现多模态理解。
  • 在自建数据集上超越现有方法,五项基准测试均领先。
  • 适合教育类AI、智能助教研发者使用。

视觉问答(VQA)旨在让AI理解图像并回答自然语言问题,但现有方法常给出过于简略的回答。本文提出视觉问答解释(VQE),提升VQA生成详尽解释的能力,满足对视觉内容更复杂的交互需求。我们从一个14周的流媒体机器学习课程中构建了MLVQE数据集,包含885张幻灯片图像、110,407字转录文本和9,416组精心设计的问题-答案对。为此,我们提出了SparrowVQE,一款仅30亿参数的小型多模态模型。采用三阶段训练:多模态预训练(对齐幻灯片图像与转录特征)、指令微调(用转录与问答对优化预训练模型)及领域微调(基于幻灯片图像与问答对调整)。最终模型通过SigLIP处理图像、Phi-2语言模型结合MLP适配器解析文本,实现图文信息融合。实验表明,SparrowVQE在自建的MLVQE数据集上表现优异,并在其他五个基准VQA数据集上优于现有最先进方法。源代码已开源。

原文摘要 · Abstract (English)

Visual Question Answering (VQA) research seeks to create AI systems to answer natural language questions in images, yet VQA methods often yield overly simplistic and short answers. This paper aims to advance the field by introducing Visual Question Explanation (VQE), which enhances the ability of VQA to provide detailed explanations rather than brief responses and address the need for more complex interaction with visual content. We first created an MLVQE dataset from a 14-week streamed video machine learning course, including 885 slide images, 110,407 words of transcripts, and 9,416 designed question-answer (QA) pairs. Next, we proposed a novel SparrowVQE, a small 3 billion parameters multimodal model. We trained our model with a three-stage training mechanism consisting of multimodal pre-training (slide images and transcripts feature alignment), instruction tuning (tuning the pre-trained model with transcripts and QA pairs), and domain fine-tuning (fine-tuning slide image and QA pairs). Eventually, our SparrowVQE can understand and connect visual information using the SigLIP model with transcripts using the Phi-2 language model with an MLP adapter. Experimental results demonstrate that our SparrowVQE achieves better performance in our developed MLVQE dataset and outperforms state-of-the-art methods in the other five benchmark VQA datasets. The source code is available at \url{https://github.com/YoushanZhang/SparrowVQE}.

视觉问答教育AI多模态解释生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。