用图神经网络蒸馏多模态常识知识，提升视觉问答准确率。
Multimodal Commonsense Knowledge Distillation for Visual Question Answering
- 构建视觉、问题与常识知识的统一关系图，通过图卷积网络融合信息。
- 在ScienceQA数据集上达到与微调相当的性能，无需额外训练。
- 适配任意师生模型，降低计算成本，适合资源有限场景使用。
现有多模态大语言模型和视觉语言预训练模型在通用视觉问答任务中表现优异，但在需要外部常识知识的问题上仍存在挑战，主要源于高质量提示生成困难及微调计算成本高。本文提出一种基于图的多模态常识知识蒸馏框架，利用图卷积网络（GCN）在教师-学生架构下构建包含常识知识、视觉对象与问题的统一关系图。该框架可灵活适配任意教师与学生模型，无需进一步微调，在ScienceQA数据集上取得具有竞争力的性能表现。
原文摘要 · Abstract (English)
Existing Multimodal Large Language Models (MLLMs) and Visual Language Pretrained Models (VLPMs) have shown remarkable performances in the general Visual Question Answering (VQA). However, these models struggle with VQA questions that require external commonsense knowledge due to the challenges in generating high-quality prompts and the high computational costs of fine-tuning. In this work, we propose a novel graph-based multimodal commonsense knowledge distillation framework that constructs a unified relational graph over commonsense knowledge, visual objects and questions through a Graph Convolutional Network (GCN) following a teacher-student environment. This proposed framework is flexible with any type of teacher and student models without further fine-tuning, and has achieved competitive performances on the ScienceQA dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。