通过双向重构训练，让大模型的结尾标记更懂上下文语义。
Training LLMs to be Better Text Embedders through Bidirectional Reconstruction
- 用双向生成重建任务优化结尾标记的语义表征
- 在MTEB基准上超越现有方法，多模型均达新最佳
- 适合需要精准文本检索与重排序的应用场景
大语言模型（LLM）正被广泛探索为强大的文本嵌入工具。现有基于LLM的文本嵌入方法通常依赖最终标记（如[EOS]）的嵌入表示，但这些标记未专门训练以捕捉完整上下文语义，限制了其在检索和重排序任务中的表现。本文在对比学习前引入新的训练阶段，通过双向生成重构任务——EBQ2D（基于嵌入的查询到文档）和EBD2Q（基于嵌入的文档到查询）——交替进行，以锚定[EOS]嵌入并重建查询-文档对的任一侧。实验结果表明，该额外训练阶段显著提升了不同规模的LLM在大规模文本嵌入基准（MTEB）上的性能，实现了多种基座模型下的新最优结果。
原文摘要 · Abstract (English)
Large language models (LLMs) have increasingly been explored as powerful text embedders. Existing LLM-based text embedding approaches often leverage the embedding of the final token, typically a reserved special token such as [EOS]. However, these tokens have not been intentionally trained to capture the semantics of the whole context, limiting their capacity as text embeddings, especially for retrieval and re-ranking tasks. We propose to add a new training stage before contrastive learning to enrich the semantics of the final token embedding. This stage employs bidirectional generative reconstruction tasks, namely EBQ2D (Embedding-Based Query-to-Document) and EBD2Q (Embedding-Based Document-to-Query), which interleave to anchor the [EOS] embedding and reconstruct either side of Query-Document pairs. Experimental results demonstrate that our additional training stage significantly improves LLM performance on the Massive Text Embedding Benchmark (MTEB), achieving new state-of-the-art results across different LLM base models and scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。