对比图文转换与视觉检索，发现大模型更依赖直接视觉信息
Exploration of Augmentation Strategies in Multi-modal Retrieval-Augmented Generation for the Biomedical Domain: A Case Study Evaluating Question Answering in Glycobiology
- 用图文转文本或直接返回图像两种方式增强医学问答
- 大模型（如GPT-4o）用视觉检索准确率达80.8%，优于文本增强
- 小模型更适合先转文本；轻量级检索器ColFlor效率高且效果好
多模态检索增强生成（MM-RAG）有望实现基于证据的生物医学问答，但尚未明确何时应将图表转为文本，或采用无需OCR的视觉检索（直接返回页面图像，由生成器解释）。本研究在以图像密集著称的糖生物学领域开展案例分析。构建了一个包含120道多选题的基准数据集，覆盖25篇论文，按检索难度分层（易：文本，中：图表，难：跨证据）。采用四种增强策略：无增强、文本RAG、多模态转换、后期交互式视觉检索（ColPali），结合Docling解析和Qdrant索引。评估了中等规模开源模型（Gemma-3-27B-IT）与前沿专有模型（如GPT-4o系列）。额外测试使用GPT-5系列及多个视觉检索器（ColPali/ColQwen/ColFlor）。每配置运行5次，以Agresti-Coull方法计算95%置信区间。结果显示，使用Gemma-3-27B-IT时，文本与多模态增强平均准确率分别为0.722–0.740，显著高于无OCR视觉检索的0.510。使用GPT-4o时，多模态达到0.808，文本为0.782，ColPali为0.745，差异较小。后续实验中，使用GPT-5系列，最优结果达0.828（+2%提升）。整体来看，不同视觉检索器间无显著差异。GPT-5-nano比大版本低约8–10%。模型能力决定最佳流程：小模型宜先转文本；大模型可直接用视觉检索。在检索器中，ColFlor在性能接近重型方案的同时体积更小，适合强生成器场景。
原文摘要 · Abstract (English)
Multi-modal retrieval-augmented generation (MM-RAG) promises grounded biomedical QA, but it is unclear when to (i) convert figures/tables into text versus (ii) use optical character recognition (OCR)-free visual retrieval that returns page images and leaves interpretation to the generator. We study this trade-off in glycobiology, a visually dense domain. We built a benchmark of 120 multiple-choice questions (MCQs) from 25 papers, stratified by retrieval difficulty (easy text, medium figures/tables, hard cross-evidence). We implemented four augmentations-None, Text RAG, Multi-modal conversion, and late-interaction visual retrieval (ColPali)-using Docling parsing and Qdrant indexing. We evaluated mid-size open-source and frontier proprietary models (e.g., Gemma-3-27B-IT, GPT-4o family). Additional testing used the GPT-5 family and multiple visual retrievers (ColPali/ColQwen/ColFlor). Accuracy with Agresti-Coull 95% confidence intervals (CIs) was computed over 5 runs per configuration. With Gemma-3-27B-IT, Text and Multi-modal augmentation outperformed OCR-free retrieval (0.722-0.740 vs. 0.510 average accuracy). With GPT-4o, Multi-modal achieved 0.808, with Text 0.782 and ColPali 0.745 close behind; within-model differences were small. In follow-on experiments with the GPT-5 family, the best results with ColPali and ColFlor improved by ~2% to 0.828 in both cases. In general, across the GPT-5 family, ColPali, ColQwen, and ColFlor were statistically indistinguishable. GPT-5-nano trailed larger GPT-5 variants by roughly 8-10%. Pipeline choice is capacity-dependent: converting visuals to text lowers the reader burden and is more reliable for mid-size models, whereas OCR-free visual retrieval becomes competitive under frontier models. Among retrievers, ColFlor offers parity with heavier options at a smaller footprint, making it an efficient default when strong generators are available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。