arXiv:2607.05438cs.IRcs.AI2026-07

提出后置选择性调用视觉模型,降低多模态问答成本。

Modality Relevance is not Modality Utility: Post-hoc Selective Modality Escalation for Cost-Aware Multimodal RAG

  • 先用文本和表格生成答案,再判断缺哪类模态
  • 在MultiModalQA上接近全视觉模型准确率,视觉调用减少超70%
  • 适合需控制推理成本的多模态应用

多模态检索增强生成(RAG)从文本、表格和图像等异构模态中获取证据。现有部署方式在回答前二选一:运行廉价的文本+表格管道,或为每张图像支付昂贵的视觉语言模型(VLM)开销。近期自适应系统通过问题条件预测需何种模态来预选,但本文指出这是错误决策点。在MultiModalQA上的溯源分析显示,模态相关性与实际必要性弱相关:大量含图像的题目仅靠文本和表格即可解答。预检索路由若按视觉相关性升级,会过度调用。我们提出后置选择性调用机制:先以文本和表格生成答案,用验证器定位缺失模态,仅对必要情况调用VLM。校准后的价值升级路由决定是否值得支付视觉成本。在MultiModalQA上,该方法达到全视觉模型精度,视觉调用次数大幅减少,接近理想升级率。该方案将路由信号层级从检索深度、推理步数扩展至模态维度,统一于成本感知的可选升级框架。

原文摘要 · Abstract (English)

Multimodal retrieval-augmented generation (RAG) grounds a generator in evidence drawn from heterogeneous modalities -- text, tables, and images. The dominant deployment choice is binary and made before the model has tried to answer: either run a cheap text(+table) pipeline, or pay for an expensive vision-language model (VLM) over every image. Recent adaptive systems improve on this by selecting the modality or fidelity pre-retrieval, from a question-conditioned predictor of which modality will be needed. We show that this is the wrong decision point. Through an oracle headroom analysis on MultiModalQA, we find that the relevance of a modality to a question is a weak predictor of whether that modality is actually needed to answer correctly: a large fraction of questions whose gold support includes an image are nonetheless answerable from text and tables alone, and a pre-retrieval router that escalates on apparent visual relevance over-escalates substantially relative to an oracle. We propose \textbf{post-hoc selective modality escalation}: answer cheaply from text and tables, run a verifier on the (query, draft answer, evidence) tuple that localizes which modality is missing, and pay for VLM evidence only there. A calibrated value-of-escalation router then decides whether the expected accuracy gain justifies the visual cost. On MultiModalQA, our router recovers the accuracy of an always-on VLM pipeline while issuing far fewer visual calls, and closes most of the gap to the oracle escalation rate. The result extends a routing-signal hierarchy established for retrieval depth and reasoning hops to a third axis -- modality -- under a single cost-aware selective-escalation view.

多模态RAG成本优化视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。