用少量例子让文本嵌入模型学会新任务，效果更好。
Making Text Embedders Few-Shot Learners
- 在查询中加入少量示例，引导嵌入模型生成更优表示。
- 在MTEB和AIR-Bench上达到新最佳性能，提升显著。
- 保持原框架简单有效，适合追求高精度的嵌入应用。
具有仅解码器架构的大语言模型展现出卓越的上下文学习能力，能够通过输入中的示例有效处理熟悉或新颖的任务。我们意识到这一能力的潜力，提出利用大语言模型的上下文学习特性来增强文本嵌入生成过程。为此，我们引入了新型模型 bge-en-icl，通过少量示例生成高质量文本嵌入。该方法将与任务相关的示例直接融入查询端，显著提升了多个任务的表现。此外，我们还研究了如何有效利用大语言模型作为嵌入模型，包括不同的注意力机制、池化方法等。结果表明，保留原始框架通常能获得最佳效果，凸显了简洁性的重要性。在MTEB和AIR-Bench基准上的实验结果显示，我们的方法达到了新的最先进（SOTA）水平。模型、代码和数据集已公开于 https://github.com/FlagOpen/FlagEmbedding。
原文摘要 · Abstract (English)
Large language models (LLMs) with decoder-only architectures demonstrate remarkable in-context learning (ICL) capabilities. This feature enables them to effectively handle both familiar and novel tasks by utilizing examples provided within their input context. Recognizing the potential of this capability, we propose leveraging the ICL feature in LLMs to enhance the process of text embedding generation. To this end, we introduce a novel model bge-en-icl, which employs few-shot examples to produce high-quality text embeddings. Our approach integrates task-related examples directly into the query side, resulting in significant improvements across various tasks. Additionally, we have investigated how to effectively utilize LLMs as embedding models, including various attention mechanisms, pooling methods, etc. Our findings suggest that retaining the original framework often yields the best results, underscoring that simplicity is best. Experimental results on the MTEB and AIR-Bench benchmarks demonstrate that our approach sets new state-of-the-art (SOTA) performance. Our model, code and dataset are freely available at https://github.com/FlagOpen/FlagEmbedding .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。