让大模型自己生成语义保留的嵌入向量，无需标注数据。
LLM2Vec-Gen: Generative Embeddings from Large Language Models
- 用可训练特殊标记压缩大模型输出，生成固定长度嵌入。
- 在MTEB上比无监督教师提升8.8%，推理与安全性能显著增强。
- 嵌入可解码回文本，适合需要可解释性的场景。
基于对比学习微调大语言模型文本嵌入器会将输入输出映射到新表示空间，丢失大模型输出语义。我们提出LLM2Vec-Gen，一种自监督方法，直接在大模型输出空间生成嵌入，通过学习模拟模型潜在响应来实现。具体地，在输入末尾添加可训练特殊标记，优化其以压缩大模型自身输出为固定长度嵌入，由无监督嵌入教师和重建目标引导。关键在于大模型主干保持冻结，训练仅需无标签查询。LLM2Vec-Gen在大规模文本嵌入基准（MTEB）上达到当前最优自监督性能，较无监督教师提升8.8%。由于嵌入保留大模型输出空间语义，继承了安全性对齐能力（有害内容检索减少最高22.6%）和推理能力（推理密集型检索提升最高35.6%）。此外，所学嵌入具备可解释性：可解码回文本以揭示其语义内容。代码、数据及预训练模型已公开于https://github.com/McGill-NLP/llm2vec-gen。
原文摘要 · Abstract (English)
Fine-tuning LLM-based text embedders via contrastive learning maps inputs and outputs into a new representational space, discarding the LLM's output semantics. We propose LLM2Vec-Gen, a self-supervised alternative that instead produces embeddings directly in the LLM's output space by learning to represent the model's potential response. Specifically, trainable special tokens are appended to the input and optimized to compress the LLM's own response into a fixed-length embedding, guided by an unsupervised embedding teacher and a reconstruction objective. Crucially, the LLM backbone remains frozen and training requires only unlabeled queries. LLM2Vec-Gen achieves state-of-the-art self-supervised performance on the Massive Text Embedding Benchmark (MTEB), improving by 8.8% over the unsupervised embedding teacher. Since the embeddings preserve the LLM's response-space semantics, they inherit capabilities such as safety alignment (up to 22.6% reduction in harmful content retrieval) and reasoning (up to 35.6% improvement on reasoning-intensive retrieval). Finally, the learned embeddings are also interpretable: they can be decoded back into text to reveal their semantic content. Our code, data, and pretrained models are publicly available at https://github.com/McGill-NLP/llm2vec-gen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。