用生成式方法统一图文问答的检索与排序，提升答案准确性。
RAMQA: A Unified Framework for Retrieval-Augmented Multi-Modal Question Answering
- 结合点式排序与生成式重排，融合多模态编码器与大模型
- 在WebQA和MultiModalQA上优于强基线，显著提升准确率
- 适合需要高精度图文问答的应用场景
多模态检索增强问答（MRAQA）融合文本与图像，在信息检索与自然语言处理中备受关注。传统排序方法依赖小型编码器模型，难以适配现代基于解码器的生成式大语言模型（LLMs）。为此，我们提出RAMQA，一个将学习排序与生成式排列增强排序相结合的统一框架。首先以LLaVA为骨干训练点式多模态排序器；随后通过指令微调，用创新的自回归多任务学习训练LLaMA模型对前k个文档进行重排。该生成式排序模型能生成重排后的文档编号及候选文档中的具体答案，支持多种排列组合。在WebQA和MultiModalQA两个MRAQA基准上的实验表明，相比强基线有显著提升，验证了该方法的有效性。代码与数据见：https://github.com/TonyBY/RAMQA
原文摘要 · Abstract (English)
Multi-modal retrieval-augmented Question Answering (MRAQA), integrating text and images, has gained significant attention in information retrieval (IR) and natural language processing (NLP). Traditional ranking methods rely on small encoder-based language models, which are incompatible with modern decoder-based generative large language models (LLMs) that have advanced various NLP tasks. To bridge this gap, we propose RAMQA, a unified framework combining learning-to-rank methods with generative permutation-enhanced ranking techniques. We first train a pointwise multi-modal ranker using LLaVA as the backbone. Then, we apply instruction tuning to train a LLaMA model for re-ranking the top-k documents using an innovative autoregressive multi-task learning approach. Our generative ranking model generates re-ranked document IDs and specific answers from document candidates in various permutations. Experiments on two MRAQA benchmarks, WebQA and MultiModalQA, show significant improvements over strong baselines, highlighting the effectiveness of our approach. Code and data are available at: https://github.com/TonyBY/RAMQA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。