通过稀疏路由只计算关键视觉与知识信息，大幅降低多模态问答的计算开销。
Salient Knowledge Pathways: Sparse Cross-Modal Routing for Efficient Knowledge-Intensive Multimodal Question Answering
- 根据问题、图像和难度动态选择计算路径，只处理相关部分。
- 在5个基准上精度持平或超越密集模型，推理耗时减少2.7倍，算力降低6.8倍。
- 适合需要高效多模态问答的部署场景，如实时客服或移动设备应用。
知识密集型多模态问答（KI-MMQA）结合了三类高成本操作：长视觉序列、大规模外部语料的密集检索和全量跨模态融合。现有系统对每个查询均付出全部开销，但实际只有少量视觉内容和检索知识与问题相关。我们提出SKIP（显著知识注入路径），一种统一推理架构，通过联合考虑问题、图像和难度估计，将计算沿稀疏路径路由。SKIP融合问题引导的视觉标记剪枝、区域条件稀疏检索、双分图稀疏交叉注意力及推测性知识验证，并由自适应预算控制器按预测难度分配算力。我们推导出信息瓶颈边界，表明在合理问题-图像互信息假设下，最优视觉稀疏率应为 $O(1/\\/sqrt{N})$，并保证保留精度。在五个KI-MMQA基准（OK-VQA, A-OKVQA, InfoSeek, Encyclopedic-VQA, ViQuAE）上，SKIP在精度上匹配或超越强密集基线，同时使用3.4–6.8倍更少的浮点运算，端到端延迟降低2.7倍。代码已公开于：https://pmlrbd.github.io/skip/
原文摘要 · Abstract (English)
Knowledge-intensive multimodal question answering (KI-MMQA) sits at the intersection of three expensive primitives: long visual token sequences, dense retrieval over large external corpora, and full cross-modal fusion. Existing systems pay all three costs uniformly per query, even though only a small fraction of visual content and retrieved knowledge is actually relevant to any given question. We introduce SKIP (Salient Knowledge-Injected Pathways), a unified inference architecture that routes computation along sparse pathways jointly conditioned on the question, the image, and a difficulty estimate. SKIP combines question-guided visual token pruning, region-conditional sparse retrieval, bipartite sparse cross-attention, and speculative knowledge verification with an adaptive budget controller that allocates compute proportional to predicted question difficulty. We derive an information-bottleneck bound showing that the optimal visual sparsity rate scales as $O(1/\sqrt{N})$ under realistic question-image mutual-information assumptions, with retained accuracy guarantees. Across five KI-MMQA benchmarks (OK-VQA, A-OKVQA, InfoSeek, Encyclopedic-VQA, and ViQuAE), SKIP matches or exceeds the accuracy of strong dense baselines while using $3.4$--$6.8\times$ fewer FLOPs and $2.7\times$ less end-to-end latency. Code available at: https://pmlrbd.github.io/skip/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。