让大模型统一生成查询并检索,通过稳定文档嵌入提升效果。
DEPT: Document Embedding Preservation Tuning for Unified Query Expansion and Retrieval

- 用单一模型端到端生成扩展查询和文档向量
- 在5个BEIR数据集上超越现有方法,平均性能提升3.2%以上
- 适合需要高效检索与查询扩增的场景,支持在线难例挖掘
大型语言模型(LLMs)既能扩展模糊查询,又能将文本编码为稠密向量,暗示了查询扩展与检索的统一建模可能。现有系统通常依赖提示生成、独立训练模块或分阶段优化,导致生成的扩展与最终检索目标间接对齐。本文提出一个单解码器架构的端到端模型,同一模型既生成扩展查询,又编码扩展后查询与候选文档。该统一设置带来动态目标问题:检索监督应优化查询侧扩展,但同一更新也会改变作为检索目标的文档嵌入。为此提出文档嵌入保持微调(DEPT),在微调过程中保持文档嵌入接近缓存的初始嵌入,同时允许检索梯度通过直通解码传回生成器。DEPT将查询与文档联合移动转化为在近似稳定的白化文档嵌入上进行查询侧适应,支持索引复用和在线难例挖掘。在Qwen3-4B-Instruct-2507与LLaMA-3.2-3B-Instruct上,于五个BEIR数据集上的实验表明,DEPT在平均检索质量上优于无需训练、独立训练及分阶段统一基线,且消融实验验证了保持、白化、端到端扩展训练与在线负样本的作用。代码已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) can both expand underspecified queries and encode text as dense representations, suggesting a unified model for query expansion and retrieval. Existing systems usually rely on prompted expansions, independently trained modules, or staged optimization, leaving generated expansions only indirectly aligned with the retrieval loss that judges them. We train a single decoder-only LLM end to end, where the same model generates the expansion and encodes both the expanded query and candidate documents. This unified setting creates a moving-target problem: retrieval supervision should improve query-side expansion, but the same update also shifts the document embeddings that serve as retrieval targets. We introduce Document Embedding Preservation Tuning (DEPT), which keeps tuned document embeddings close to cached initial embeddings while allowing retrieval gradients to pass through straight-through decoding into the generator. DEPT converts joint query--document movement into query-side adaptation against approximately stable, whitened document embeddings that support index reuse and online hard-negative mining. Experiments with Qwen3-4B-Instruct-2507 and LLaMA-3.2-3B-Instruct on five datasets in BEIR benchmark show that DEPT improves average retrieval quality over training-free, independently trained, and staged unified baselines, while ablations isolate the effects of preservation, whitening, end-to-end expansion training, and online negatives. Code is available at https://github.com/ILSparkle/DEPT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。