arXiv:2508.16051cs.AI2025-08EMNLP被引 3

无需训练,用动态图规划多模态多跳问答

MMAPG: A Training-Free Framework for Multimodal Multi-hop Question Answering via Adaptive Planning Graphs

  • 构建自适应规划图,动态决定推理路径和扩展方向
  • 在MultimodalQA和WebQA上达到或超越训练模型性能
  • 适合希望快速部署且不依赖训练的多模态系统开发者

多模态多跳问答需融合图像与文本等多元信息以得出答案。现有方法多采用串行检索与推理,每一步依赖前序输出,易受错误中间结果影响。同时,多模态模型训练成本高。为此,我们提出无需训练的MMAPG框架,基于自适应规划图,包含规划、检索与推理模块。规划模块分析当前图状态,决定下一步动作及扩展位置,实现灵活的推理路径探索。针对文本到非指定模态的检索,设计模态特异性策略,动态适配不同数据类型。该方法无需任务特定训练即可保留多模态信息特性,可无缝集成最新模型。在MultimodalQA和WebQA上的实验表明,本方法性能匹配或优于依赖训练的现有模型。

原文摘要 · Abstract (English)

Multimodal Multi-hop question answering requires integrating information from diverse sources, such as images and texts, to derive answers. Existing methods typically rely on sequential retrieval and reasoning, where each step builds on the previous output. However, this single-path paradigm makes them vulnerable to errors due to misleading intermediate steps. Moreover, developing multimodal models can be computationally expensive, often requiring extensive training. To address these limitations, we propose a training-free framework guided by an Adaptive Planning Graph, which consists of planning, retrieval and reasoning modules. The planning module analyzes the current state of the Adaptive Planning Graph, determines the next action and where to expand the graph, which enables dynamic and flexible exploration of reasoning paths. To handle retrieval of text to unspecified target modalities, we devise modality-specific strategies that dynamically adapt to distinct data types. Our approach preserves the characteristics of multimodal information without costly task-specific training, enabling seamless integration with up-to-date models. Finally, the experiments on MultimodalQA and WebQA show that our approach matches or outperforms existing models that rely on training.

多模态多跳问答无训练图推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。