用嵌入向量替代文本示例,让大模型更高效生成高质量文本编码。
Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders

- 用连续嵌入代替原始文本示例,减少序列长度负担。
- 在MTEB基准上超越仅用公开数据训练的先进模型。
- 适合需要高效文本编码的检索与相似度任务场景。
大型语言模型(LLMs)已被广泛用于生成嵌入表示。尽管近期研究显示,通过在输入前添加少量任务相关的示例(in-context learning, ICL),可有效提升模型的表征能力,但这一方法会因序列长度增加而带来显著的令牌开销。本文提出EPIC——一种基于嵌入的上下文提示训练策略,利用ICL生成高质量嵌入,同时降低训练与推理时的计算负担。该方法将离散的文本示例替换为其对应的连续嵌入,不仅促使模型在对比学习中对齐语义相关文本对,还要求模型将示例嵌入作为上下文提示的一部分进行理解。结果表明,经EPIC训练的模型在推理时无论是否使用上下文提示,均能实现优异的嵌入性能。全面实验表明,本方法在MTEB基准上达到新的最先进水平,优于仅使用公开检索数据训练的前沿模型。大量消融实验进一步验证了该机制的有效性与必要性。
原文摘要 · Abstract (English)
Large language models (LLMs) have been widely explored for embedding generation. While recent studies show that in-context learning (ICL) effectively enhances the representational capability of LLMs by prepending a few task-related demonstrations, it causes substantial token overhead due to the increased sequence length. In this work, we propose EPIC, a novel embedding-based in-context prompt training strategy that leverages ICL to generate high-quality embeddings while reducing computational burden during both training and inference. This approach replaces discrete text demonstrations with their corresponding continuous embeddings, which not only encourages the LLM to align semantically-related text pairs during contrastive learning, but also requires the model to interpret demonstration embeddings as part of the in-context prompt. Consequently, EPIC-trained models achieve excellent embedding performance both with or without in-context prompts at inference time. Comprehensive experiments demonstrate that our method establishes new state-of-the-art results on the MTEB benchmark, surpassing frontier models trained solely on publicly available retrieval data. Extensive ablation studies further validate the effectiveness and necessity of our mechanism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。