用BERT改进图表问答，发现跨模态模块未必关键
BERT-VQA: Visual Question Answering on Plots
- 基于VisualBERT架构，融合ResNet101图像编码器与联合融合机制
- 对比实验表明跨模态模块对图表问答效果提升不显著
- 适合研究视觉语言模型在图表理解中作用的学者参考
视觉问答是自然语言理解领域的挑战性任务，要求模型在视觉与语言域间进行信息交互。本项目聚焦其中的图表问答子任务，提出BERT-VQA模型，基于VisualBERT架构，采用预训练ResNet101作为图像编码器，并引入潜在的联合融合机制。该模型与包含LSTM、CNN和浅层分类器的基线模型进行了对比训练与评估。最终结果推翻了核心假设——即VisualBERT中的跨模态模块对对齐图表元素与问题短语至关重要。研究揭示了图表问答任务的难度，以及不同模型架构在此问题上的适用性，提供了重要洞见。
原文摘要 · Abstract (English)
Visual question answering has been an exciting challenge in the field of natural language understanding, as it requires deep learning models to exchange information from both vision and language domains. In this project, we aim to tackle a subtask of this problem, namely visual question answering on plots. To achieve this, we developed BERT-VQA, a VisualBERT-based model architecture with a pretrained ResNet 101 image encoder, along with a potential addition of joint fusion. We trained and evaluated this model against a baseline that consisted of a LSTM, a CNN, and a shallow classifier. The final outcome disproved our core hypothesis that the cross-modality module in VisualBERT is essential in aligning plot components with question phrases. Therefore, our work provided valuable insights into the difficulty of the plot question answering challenge as well as the appropriateness of different model architectures in solving this problem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。