arXiv:2607.20517cs.LG2026-07

用三重过滤提升复杂文档的多模态检索,让AI更准理解灾害报告。

Multimodal CoLRAG-TF: Triple-Filtered Retrieval for Complex PDFs

论文配图:Multimodal CoLRAG-TF: Triple-Filtered Retrieval for Complex PDFs
图 1 · 摘自论文原文
  • 融合文本、关键词、知识图谱三重过滤,精准定位信息块。
  • 检索召回率达99.09%,多跳问答准确率提升71.6%。
  • 适合处理含图表、术语密集的非结构化文档,如灾后报告。

针对异构PDF集合中多模态内容、领域术语和跨段落推理的挑战,本文提出Multimodal CoLRAG-TF,一种四轴融合架构,集成密集文本嵌入、BM25关键词匹配、知识图谱三元组过滤和图像相似性。系统从43份日文灾害案例PDF中提取2,403个内容块,通过混合OCR与LLM生成图文描述。为支持组合推理,提取11,414条OpenIE三元组并使用FAISS索引,实现亚秒级三元组查询与相关性层级传播。采用受HippoRAG2启发的粗到精检索策略(全文→章节→块),在最终融合前缩小搜索空间。贝叶斯优化显示三元组权重需主导(α_triple = 0.44)以对抗词汇偏差并维持多跳检索质量。在457对样本上评估,系统达到0.9909的检索召回率,多跳答案相似度相较单跳提升71.6%。基于视觉LLM的图像-案例映射管道进一步验证了方法在视觉输入上的适用性。结果表明,三重过滤的多模态融合对噪声且异构的PDF中的结构化推理至关重要,并可推广至灾后之外的领域。

原文摘要 · Abstract (English)

Retrieval-augmented generation (RAG) over heterogeneous PDF collections remains challenging due to multimodal content, domain-specific terminology, and the need for multi-hop reasoning across dispersed evidence. We present Multimodal CoLRAG-TF, a four-axis fusion architecture that integrates dense text embeddings, BM25 keyword matching, knowledge-graph triple filtering, and image-based similarity for robust retrieval over complex documents. Our system constructs a multimodal index of 2,403 blocks extracted from 43 Japanese disaster lesson PDFs, supported by a hybrid OCR pipeline and LLM-based caption generation. To enhance compositional reasoning, we extract 11,414 OpenIE triples and index them with FAISS, enabling sub-second triple lookup and hierarchical propagation of relevance signals. A HippoRAG2-inspired coarse-to-fine retriever (volume $\to$ chapter $\to$ block) narrows the search space before final fusion scoring. Bayesian optimization over fusion weights reveals that the triple axis must dominate ($α_\text{triple} = 0.44$) to counteract lexical bias and sustain multi-hop retrieval quality. Evaluated on a 457-pair benchmark, Multimodal CoLRAG-TF achieves a Retrieval Recall of 0.9909 and a 71.6$\%$ improvement in multi-hop answer similarity over single-hop queries. An image-to-lesson pipeline using a vision LLM further demonstrates the applicability of the approach to visual inputs. These results show that triple-filtered multimodal fusion is essential for structured reasoning over noisy, heterogeneous PDFs and provides a general framework applicable beyond the disaster domain.

多模态检索知识图谱PDF分析灾难报告

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。