arXiv:2608.12987cs.IRcs.AI2026-08

提出双角色标识符框架,实现跨模态高效精准检索。

Generative Universal Multimodal Retrieval with Dual-role Identifiers

论文配图:Generative Universal Multimodal Retrieval with Dual-role Identifiers
图 1 · 摘自论文原文
  • 用双重角色标识符,既可自回归生成又可无序集合比对。
  • 在M-BEIR和图文检索上超越现有生成式方法,接近密集向量模型性能。
  • 适合需要多模态、高效率检索的系统设计者参考。

生成式信息检索(GIR)通过训练生成器直接输出相关项标识符,替代传统的索引-检索-排序流程。然而仍面临三大挑战:左到右解码易受前缀错误和局部最优影响;多数研究局限于单模态,跨文本、图像及图文混合项的指令感知检索未被充分探索;尽管离散标识符提升效率,其检索精度仍低于先进密集向量方法。为此,我们提出DrIG框架,采用双角色标识符支持多模态、跨域通用检索。每个候选项分配一个残差量化标识符,兼具序列与集合双重角色:序列角色下,首词建模模态,后续词逐步捕获细粒度语义;集合角色下,相同词元被重新解释为无序集合,提供不受前缀影响的相关性先验,指导受限束搜索,缓解局部最优问题。在M-BEIR基准与图文评估数据集上的实验表明:(1) DrIG在多种任务中持续优于顶尖生成式多模态基线,混合重排序在效率与效果间取得良好平衡;(2) 消融与缩放分析揭示了基础大模型、束宽、重排序深度及融合策略对性能的影响,为系统设计提供实用指导。

原文摘要 · Abstract (English)

Generative information retrieval (GIR) has emerged as a compelling alternative to the conventional index-retrieve-then-rank retrieval pipeline by training a generator to produce the identifiers of relevant items directly. Despite its promise, a number of open challenges still remain. First, constrained left-to-right decoding is vulnerable to prefix-level errors and local optima. Second, most prior GIR research remains largely unimodal, leaving instruction-aware retrieval across text, image, and mixed image-text items underexplored. Third, although discrete identifier-based GIR offers higher efficiency, its retrieval accuracy still lags behind that of the cutting-edge dense-vector-based retrieval methods. Motivated by these challenges, we propose DrIG, a novel Generative framework for universal multimodal retrieval featuring Dual-role Identifiers, which supports diverse retrieval tasks across multiple modalities and domains. Each candidate is assigned a single residual-quantized identifier that serves two complementary roles. In its sequential role, the identifier is decoded autoregressively, where the first token explicitly models modality and the remaining tokens capture progressively finer semantics. In its set-based role, the same tokens are reinterpreted as an unordered set to provide a prefix-independent relevance prior, which guides constrained beam search and alleviates local-optimum errors. Extensive experiments on the M-BEIR benchmark and the text-to-image evaluation datasets show that:(1)DrIG consistently outperforms state-of-the-art generative multimodal baselines across diverse tasks, while hybrid reranking achieves a favorable efficiency-effectiveness trade-off against strong dense retrievers. (2)Ablation and scaling analyses reveal how the base LMM, beam size, reranking depth, and fusion strategy affect retrieval performance, providing practical guidance for system design.

生成式检索多模态双角色标识符高效检索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。