不训练模型，通过重路由键值状态实现高效文本嵌入。
KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs
- 利用模型最后一层的键值状态作为前置上下文，让所有词元共享全局信息。
- 在MTEB评测中超越现有无训练方法最高10%，支持最长4096词元序列。
- 无需微调，适配Qwen、Mistral等主流解码器模型，适合快速部署。
尽管大语言模型是强大的嵌入骨干网络，但在无训练场景下仍面临两个结构限制：因果注意力机制使早期词元无法获取后续上下文，而下一个词预测目标则使表示偏向生成而非语义压缩。为此，我们提出KV-Embedding框架，激活冻结大模型的潜在表征能力。方法基于观察：每一层最后词元的键值（KV）状态编码了序列的压缩视图。通过将这些状态作为前置前缀重路由，可在单次前向传播中使所有词元访问全局上下文。为确保模型无关性，引入基于内在维度的自动层选择策略。在Qwen、Mistral和Llama骨干模型上对MTEB的评估显示，该方法相比现有无训练基线最高提升10%，且在长达4096词元的序列上保持鲁棒性能。结果表明，内部状态操作可作为输入修改的有效替代方案，推动对大模型内部机制用于表示学习的进一步探索。
原文摘要 · Abstract (English)
While LLMs are powerful embedding backbones, their application in training-free settings faces two structural challenges: causal attention restricts early tokens from accessing subsequent context, and the next-token prediction objective biases representations toward generation rather than semantic compression. To address these limitations, we propose KV-Embedding, a framework that activates the latent representation power of frozen LLMs. Our method leverages the observation that the key-value (KV) states of the final token at each layer encode a compressed view of the sequence. By re-routing these states as a prepended prefix, we enable all tokens to access sequence-level context within a single forward pass. To ensure model-agnostic applicability, we introduce an automated layer selection strategy based on intrinsic dimensionality. Evaluations on MTEB across Qwen, Mistral, and Llama backbones show that KV-Embedding outperforms existing training-free baselines by up to 10%, while maintaining robust performance on sequences up to 4,096 tokens. These results demonstrate that internal state manipulation offers an efficient alternative to input modification, and we hope this work encourages further exploration of LLM internals for representation learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。