让多模态问答更忠实于文档结构,提升图文对齐与证据定位。
HAM-RAG: Hierarchy-Aware Multimodal RAG for Structure-Faithful Interleaved Generation

- 利用文档层级信息统一检索与生成,保持原文组织逻辑。
- 在多个数据集上平均提升17.3%,在Wukong上图像相关性指标提升24.2%。
- 适合技术手册、操作指南等需严格遵循结构的多模态应用。
现有多模态RAG方法常将结构化文档拆分为孤立的文本与图像单元,削弱了源文档的组织结构和局部图文逻辑,影响证据选择与放置的准确性。本文提出HAM-RAG框架,通过文档层级作为跨检索与生成的引导信号,上下文化文本与视觉证据,保留原始位置及局部图文关系。我们构建了HAM-Bench,涵盖Wukong、Wiki、arXiv和Recipe四类数据集,覆盖游戏攻略、网页、科研论文与步骤式食谱。在多种骨干模型下,HAM-RAG相比最强非层级基线平均提升17.3%;在Wukong上,Img-CBS指标提升24.2%,显著改善局部图文对齐。主实验与消融研究共同表明,文档层级是实现忠实图像选择、精准放置与局部图文对齐的关键引导信号。该成果凸显层次感知建模对可靠多模态助手的重要性,尤其适用于技术手册、维护指南与工业SOP等需忠实还原结构与图文证据的场景。代码已开源:https://github.com/MCCodeAI/HAM-RAG.git。
原文摘要 · Abstract (English)
Existing multimodal RAG methods often flatten structured documents into isolated text and image units, weakening the source organization and local text-image logic needed for faithful evidence selection and placement. We propose HAM-RAG, a Hierarchy-Aware Multimodal RAG framework for structure-faithful interleaved generation. HAM-RAG uses document hierarchy as a grounding signal across retrieval and generation, contextualizing textual and visual evidence and preserving source position and local text-image relations in the prompt. We further introduce HAM-Bench, covering Wukong, Wiki, arXiv, and Recipe across game walkthroughs, web pages, scientific papers, and step-wise recipe documents. Across multiple backbones, HAM-RAG improves the main multimodal average by 17.3% over the strongest non-hierarchical baseline. On Wukong, HAM-RAG improves Img-CBS by 24.2% over the strongest non-hierarchical baseline, demonstrating substantially better local text-image alignment. The main experiments and ablation study together demonstrate that document hierarchy is a key grounding signal for faithful image selection, placement, and local text-image alignment. These findings highlight the value of hierarchy-aware grounding for reliable multimodal assistants that generate answers faithful to the source organization, procedural structure, and local text-image evidence of structured documents, such as technical manuals, maintenance guides, and industrial SOPs. The code is available at https://github.com/MCCodeAI/HAM-RAG.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。