用扩散模型做文本嵌入，显著提升长文档检索效果。
Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective
- 采用具有双向注意力的扩散语言模型替代自回归模型
- 长文档检索性能提升20%，推理密集型任务提升8%
- 适合需要全局上下文理解的复杂文本任务
基于大语言模型（LLM）的嵌入模型得益于大规模预训练与后训练，在文档检索等通用文本嵌入任务上已超越BERT和T5。然而，LLM在自回归预训练中使用单向注意力，与文本嵌入任务所需的双向特性不匹配。为此，我们提出采用扩散语言模型进行文本嵌入，其固有的双向架构及在推理任务中的优异表现提供了新思路。这是首个对扩散语言嵌入模型的系统性研究，结果表明该模型在长文档检索上优于基于LLM的模型20%，在推理密集型检索上提升8%，在指令遵循型检索上提升2%，同时在传统文本嵌入基准上表现相当。分析验证了双向注意力对长且复杂文本中全局上下文编码的关键作用。
原文摘要 · Abstract (English)
Large language model (LLM)-based embedding models, benefiting from large scale pre-training and post-training, have begun to surpass BERT and T5-based models on general-purpose text embedding tasks such as document retrieval. However, a fundamental limitation of LLM embeddings lies in the unidirectional attention used during autoregressive pre-training, which misaligns with the bidirectional nature of text embedding tasks. To this end, We propose adopting diffusion language models for text embeddings, motivated by their inherent bidirectional architecture and recent success in matching or surpassing LLMs especially on reasoning tasks. We present the first systematic study of the diffusion language embedding model, which outperforms the LLM-based embedding model by 20% on long-document retrieval, 8% on reasoning-intensive retrieval, 2% on instruction-following retrieval, and achieve competitive performance on traditional text embedding benchmarks. Our analysis verifies that bidirectional attention is crucial for encoding global context in long and complex text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。