用高阶超图建模多模态文档,精准捕捉图文表间复杂关系。
Hypergraph-based Multimodal Retrieval-Augmented Generation with Incremental Refinement

- 构建多模态超图,用超边统一表示图文表间的多向关联
- 仅对跨页锚点节点局部重构,计算开销低且减少噪声
- 适合长文档多模态生成任务,尤其关注跨页知识融合
现有多模态检索增强生成系统受限于传统简单图的二元连接范式,难以捕捉视觉图表、分散文本描述与底层数据之间的高阶复杂关系。现有优化策略常需全页重建,导致计算冗余并引入上下文噪声。本文提出Hyper-M2RAG框架,通过高阶超图表示学习重构多模态文档结构。将文档建模为多模态超图,以超边作为统一语义容器,封装文本、图像与表格间的多维关联,突破点对点建模局限。为缓解分页带来的语义碎片化问题,提出锚点驱动的增量精炼机制:不进行全局扫描,而是识别跨页锚点节点,利用其一跳邻域上下文重构局部超拓扑。该方法以极小计算开销有效弥合跨页知识断层。在多个多模态基准数据集上的实验表明,Hyper-M2RAG在检索精度与生成连贯性上显著优于现有方法。代码已开源:https://github.com/ShenAoChen2001/MMHRAG。
原文摘要 · Abstract (English)
Modern Multimodal Retrieval-Augmented Generation (M-RAG) systems are fundamentally limited by the binary connectivity paradigm of traditional simple graphs, which fails to capture the intricate, high-order correlations among heterogeneous entities, such as the N-ary relationships between a visual chart, its scattered textual descriptions, and underlying numerical data. Furthermore, existing refinement strategies often rely on exhaustive, full-page reconstruction to align cross-modal information, leading to prohibitive computational redundancy and the introduction of contextual noise in long-form document processing. In this paper, we propose Hyper-M2RAG, a novel framework that redefines multimodal document retrieval through High-order Hypergraph Representation Learning. We first formalize the document structure as a Multimodal Hypergraph, utilizing hyperedges as unified semantic containers to encapsulate multi-way associations across text, images, and tables, thereby transcending point-to-point modeling. To mitigate semantic fragmentation caused by physical pagination, we introduce an Anchor-driven Incremental Refinement mechanism. Rather than performing a global sweep, our approach identifies boundary-crossing anchor nodes and reconstructs their local hyper-topology using one-hop neighborhood contexts. This targeted refinement effectively bridges cross-page knowledge gaps with minimal computational footprints. Extensive evaluations on multimodal benchmarking datasets demonstrate that Hyper-M2RAG significantly outperforms state-of-the-art methods in both retrieval precision and generation coherence. Our code is available at https://github.com/ShenAoChen2001/MMHRAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。