利用扩散模型生成过程中的低置信度词预测，提前检索增强答案质量。
Self-Augmenting Retrieval for Diffusion Language Models
- 用生成过程中被丢弃的低置信度词作为前瞻信号引导检索。
- 在五个多跳问答数据集上，速度比现有方法快8倍，性能更优。
- 无需训练、适配任意扩散语言模型，适合高效推理场景。
离散扩散语言模型通过并行迭代去噪整个响应生成文本。每一步中，模型为每个掩码位置预测暂定词元，将高置信度预测保留到输出，丢弃低置信度结果。我们发现，这些被丢弃的词元实际上可作为检索增强生成的有用前瞻信号：即使置信度低，它们也常在去噪早期就浮现关键实体，从而可在输出最终确定前检索更强证据。为此，我们提出针对扩散语言模型的自增强检索框架（SARDI），利用这些前瞻词元在去噪过程中动态引导检索。SARDI无需训练、对检索器无依赖，适用于任何具备推理能力的离散扩散语言模型。在五个多跳问答基准上，SARDI 的吞吐量相比当前训练自由扩散和自回归检索基线最高提升8倍。
原文摘要 · Abstract (English)
Discrete diffusion language models generate text by iteratively denoising an entire response in parallel. At each step, they predict tentative tokens for every masked position, committing the confident predictions to the output and discarding the unconfident ones. We show that the discarded tokens are in fact a useful lookahead signal for retrieval-augmented generation: even low-confidence tokens often surface salient entities early in the denoising trajectory, enabling retrieval of stronger evidence before the output is finalized. We exploit this through Self-Augmenting Retrieval for Diffusion Language Models (SARDI), a dynamic RAG framework that uses these lookahead tokens to guide retrieval during denoising. SARDI is training-free, retriever-agnostic, and applicable to any reasoning-capable discrete diffusion language model. Across five multi-hop QA benchmarks, SARDI outperforms current training-free diffusion and autoregressive retrieval baselines at up to $8\times$ higher throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。