用扩散模型生成文档标识符,提升检索准确率与效率可控性。
DiffuGR: Generative Document Retrieval with Diffusion Language Models
- 将文档检索转为离散扩散过程,分步去噪生成标识符。
- 在多个基准上超越传统自回归方法,且支持灵活控制速度与精度。
- 适合需要高精度与可调节推理延迟的实用检索场景。
生成式检索(GR)将文档检索重构为端到端生成文档标识符(DocID)的任务。现有方法主要依赖自回归解码,存在两大缺陷:(i) DocID生成与自然语言生成不匹配,早期错误导致完全失效;(ii) 无法动态平衡检索效率与准确率,影响实际应用。为此,我们提出基于扩散语言模型的生成式文档检索方法——DiffuGR。DiffuGR将DocID生成建模为离散扩散过程:训练时通过随机掩码污染DocID,训练扩散语言模型在检索感知目标下恢复;推理时并行生成大量置信度高的DocID,并通过可控去噪步数逐步优化。相比自回归方式,DiffuGR引入先生成后精炼的新机制,并实现对质量与延迟权衡的显式控制。大量实验表明,DiffuGR在多个常用检索基准上优于强基线自回归生成式检索器。此外,通过调整去噪预算,验证了其在质量-延迟间灵活调控的能力。
原文摘要 · Abstract (English)
Generative retrieval (GR) reframes document retrieval as an end-to-end task of generating sequential document identifiers (DocIDs). Existing GR methods predominantly rely on left-to-right auto-regressive decoding, which suffers from two fundamental limitations: (i) a \emph{mismatch between DocID generation and natural language generation}, whereby an incorrect DocID token generated at an early step can lead to entirely erroneous retrieval; and (ii) an \emph{inability to dynamically balance the trade-off between retrieval efficiency and accuracy}, which is crucial for practical applications. To tackle these challenges, we propose generative document retrieval with diffusion language models, termed \emph{DiffuGR}. DiffuGR formulates DocID generation as a discrete diffusion process. During training, DocIDs are corrupted through a stochastic masking process, and a diffusion language model is trained to recover them under a retrieval-aware objective. For inference, DiffuGR generates DocID tokens in parallel and refines them through a controllable number of denoising steps. Unlike auto-regressive decoding, DiffuGR introduce \emph{a novel mechanism to first generate plenty of confident DocID tokens and then refine the generation through diffusion-based denoising}. Moreover, DiffuGR also offers \emph{explicit runtime control over the quality-latency tradeoff}. Extensive experiments on widely-applied retrieval benchmarks show that DiffuGR outperforms strong auto-regressive generative retrievers. Additionally, we verify that DiffuGR achieves flexible control over the quality-latency trade-off via variable denoising budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。