提出LILaC框架，提升多模态多跳检索的精度与效率。
LILaC: Late Interacting in Layered Component Graph for Open-domain Multimodal Multihop Retrieval
- 构建分层组件图，分粗粒度与细粒度表示多模态内容
- 采用后期交互策略，先粗筛后精推，提升检索效率
- 在5个基准上达顶尖性能，无需额外微调
多模态文档检索旨在从包含文本、表格和视觉元素的文档中检索出与查询相关的组件。一个有效的多模态检索器需应对两大挑战：(1) 降低由固定单一粒度检索单元引发的无关内容干扰；(2) 通过有效捕捉组件间及跨文档的语义关系，支持多跳推理。为此，我们提出LILaC，一种具有两项核心创新的多模态检索框架。首先，引入分层组件图，显式表示多模态信息在两个层级——粗粒度与细粒度——上的结构，促进高效而精确的推理。其次，设计基于后期交互的子图检索方法，一种基于边的策略：先识别粗粒度节点以高效生成候选，再通过后期交互进行细粒度推理。大量实验表明，LILaC在所有五个基准测试上均达到当前最优性能，尤其值得注意的是无需额外微调。相关代码与数据已公开于github.com/joohyung00/lilac。
原文摘要 · Abstract (English)
Multimodal document retrieval aims to retrieve query-relevant components from documents composed of textual, tabular, and visual elements. An effective multimodal retriever needs to handle two main challenges: (1) mitigate the effect of irrelevant contents caused by fixed, single-granular retrieval units, and (2) support multihop reasoning by effectively capturing semantic relationships among components within and across documents. To address these challenges, we propose LILaC, a multimodal retrieval framework featuring two core innovations. First, we introduce a layered component graph, explicitly representing multimodal information at two layers - each representing coarse and fine granularity - facilitating efficient yet precise reasoning. Second, we develop a late-interaction-based subgraph retrieval method, an edge-based approach that initially identifies coarse-grained nodes for efficient candidate generation, then performs fine-grained reasoning via late interaction. Extensive experiments demonstrate that LILaC achieves state-of-the-art retrieval performance on all five benchmarks, notably without additional fine-tuning. We make the artifacts publicly available at github.com/joohyung00/lilac.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。