用智能代理动态拆分多模态检索生成流程，让问答更快更准。
Efficient Multimodal Planning Agent for Visual Question-Answering
- 训练多模态规划代理，按需决定每步检索生成操作
- 搜索时间减少60%以上，工具调用成本显著降低
- 适用于知识密集型视觉问答，适合追求效率的部署场景
视觉问答（VQA）是一项需要融合视觉与文本信息的复杂多模态任务。尽管多模态检索增强生成（mRAG）在提供图像和文本证据方面展现潜力，但现有方法通常依赖多阶段管道，存在内在依赖性，导致效率低下。为缓解这一问题并保持性能，本文提出一种训练多模态规划代理的方法，动态分解mRAG流程以解决VQA任务。该方法通过训练代理智能判断每一步mRAG的必要性，优化效率与效果之间的权衡。实验表明，该方法可减少冗余计算，相比现有方法将搜索时间缩短超60%，并降低昂贵的工具调用次数。同时，在六个不同数据集上，本方法平均性能优于所有基线，包括深度研究代理和精心设计的提示方法。代码将公开。
原文摘要 · Abstract (English)
Visual Question-Answering (VQA) is a challenging multimodal task that requires integrating visual and textual information to generate accurate responses. While multimodal Retrieval-Augmented Generation (mRAG) has shown promise in enhancing VQA systems by providing more evidence on both image and text sides, the default procedure that addresses VQA queries, especially the knowledge-intensive ones, often relies on multi-stage pipelines of mRAG with inherent dependencies. To mitigate the inefficiency limitations while maintaining VQA task performance, this paper proposes a method that trains a multimodal planning agent, dynamically decomposing the mRAG pipeline to solve the VQA task. Our method optimizes the trade-off between efficiency and effectiveness by training the agent to intelligently determine the necessity of each mRAG step. In our experiments, the agent can help reduce redundant computations, cutting search time by over 60\% compared to existing methods and decreasing costly tool calls. Meanwhile, experiments demonstrate that our method outperforms all baselines, including a Deep Research agent and a carefully designed prompt-based method, on average over six various datasets. Code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。