通过挖掘模型内部表示,用轻量模块提升单向量检索效果。
MINER: Mining Multimodal Internal Representation for Efficient Retrieval

- 从Transformer各层提取关键信息,融合成紧凑向量。
- 在多个数据集上提升4.5% nDCG@5,接近双向量方法性能。
- 无需修改主干网络,保持高效存储与推理速度。
视觉文档检索在富视觉文档中至关重要。现有方法分为两类:晚交互检索器通过细粒度标记级匹配实现高质量,但每页存储数百个向量,索引开销大、服务成本高;而密集单向量检索器虽存储和延迟优势明显,但因将全部信息压缩为单一最终嵌入,性能持续落后。本文首次对单向量检索器进行逐层诊断,发现检索相关信号存在于内部表示中。基于此,提出MINER(Mining Multimodal Internal Representation for Efficient Retrieval),一个轻量级插件模块,无需修改主干网络,即可探测并融合跨层内部信号至单一紧凑嵌入。第一阶段“检索对齐层探测”在每层附加轻量探针,识别携带检索相关信息的维度;第二阶段“自适应稀疏多层融合”对选定层实施性能自适应神经元级掩码,并融合保留信号至最终稠密向量。在ViDoRe V1/V2/V3上,MINER在多数基准上超越现有单向量检索器,最高提升4.5% nDCG@5;相比强晚交互基线,在某些场景下将nDCG@5差距缩小至0.2,同时保持密集检索的存储与服务优势。
原文摘要 · Abstract (English)
Visual document retrieval has become essential for accessing information in visually rich documents. Existing approaches fall into two camps. Late-interaction retrievers achieve strong quality through fine-grained token-level matching but store hundreds of vectors per page, incurring large index footprints and high serving costs. By contrast, dense single-vector retrievers retain storage and latency advantages but consistently lag in quality because they compress all information into a single final-layer embedding. In this work, we first conduct a layerwise diagnostic on single-vector retrievers, revealing that retrieval-relevant signal resides in internal representations. Motivated by these findings, we propose MINER (Mining Multimodal Internal RepreseNtation for Efficient Retrieval), a lightweight plug-in module that probes and fuses internal signals across transformer layers into a single compact embedding without modifying the backbone or sacrificing single-vector efficiency. The first Retrieval-Aligned Layer Probing stage attaches a lightweight probe at each layer, surfacing which dimensions carry retrieval-relevant information. The subsequent Adaptive Sparse Multi-Layer Fusion stage applies performance-adaptive neuron-level masking to the selected layers and fuses the surviving signals into the final dense vector. Across ViDoRe V1/V2/V3, MINER outperforms existing dense single-vector retrievers on the majority of benchmarks, with up to 4.5% nDCG@5 improvement over its corresponding backbone. Compared to strong late-interaction baselines, in some settings MINER substantially narrows the nDCG@$5$ gap to $0.2$ while preserving the storage and serving advantages of dense retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。