用大模型提升词典型文本嵌入，更紧凑且高效。
Enhancing Lexicon-Based Text Embeddings with Large Language Models
- 通过聚类词表减少冗余，将每个维度对应一个语义簇。
- 在MTEB上超越密集嵌入，维度与之相当但更紧凑。
- 无需特殊目标即可高效剪枝，适合需要轻量化的场景。
近期的大语言模型（LLMs）在通用文本嵌入任务中表现优异。尽管密集嵌入占据主导地位，我们首次提出基于词典的嵌入方法（LENS），利用LLMs实现竞争性性能。LENS通过词元嵌入聚类整合词汇空间，缓解了LLM词表中的冗余问题。为进一步提升性能，研究了双向注意力和多种池化策略。LENS通过将每个维度分配给特定词元簇，简化了词汇匹配过程，语义相似词被归为一类。大量实验表明，LENS在大规模文本嵌入基准（MTEB）上优于密集嵌入，同时保持与之相当的维度。此外，LENS天然支持高效嵌入维度剪枝，无需类似马特约什卡表示学习的专用目标。值得注意的是，将LENS与密集嵌入结合，在MTEB的检索子集（即BEIR）上达到当前最优性能。
原文摘要 · Abstract (English)
Recent large language models (LLMs) have demonstrated exceptional performance on general-purpose text embedding tasks. While dense embeddings have dominated related research, we introduce the first lexicon-based embeddings (LENS) leveraging LLMs that achieve competitive performance on these tasks. LENS consolidates the vocabulary space through token embedding clustering to handle the issue of token redundancy in LLM vocabularies. To further improve performance, we investigate bidirectional attention and various pooling strategies. Specifically, LENS simplifies lexical matching with redundant vocabularies by assigning each dimension to a specific token cluster, where semantically similar tokens are grouped together. Extensive experiments demonstrate that LENS outperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB), delivering compact representations with dimensionality comparable to dense counterparts. Furthermore, LENS inherently supports efficient embedding dimension pruning without any specialized objectives like Matryoshka Representation Learning. Notably, combining LENS with dense embeddings achieves state-of-the-art performance on the retrieval subset of MTEB (i.e., BEIR).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。