用稀疏自编码器构建可解释的检索模型，提升多语言和跨领域检索效果。
Learning Retrieval Models with Sparse Autoencoders
- 用稀疏自编码器将大模型输出分解为语义清晰的稀疏特征
- 在多语言与跨领域任务上超越传统词汇空间检索方法
- 支持轻量级版本，适合资源受限场景部署
稀疏自编码器（SAEs）能够将大型语言模型产生的密集表示分解为可解释的潜在特征。我们提出，SAEs是学习稀疏检索（LSR）的天然基础，其目标是将查询和文档编码为针对高效检索优化的高维稀疏表示。与现有基于词汇空间投影的LSR方法不同，基于SAE的表示能生成更具语义结构、表达力更强且语言无关的特征。基于此，我们提出了SPLARE方法来训练基于SAE的LSR模型。实验使用最近发布的开源SAEs，结果表明该方法在多语言和跨领域设置中持续优于词汇基LSR。SPLARE-7B是一个多语言检索模型，能为多种语言和领域生成可泛化的稀疏潜在嵌入，在MMTEB的多语言和英文检索任务上取得最佳表现。我们还开发了一个2B参数的轻量级版本，显著降低计算开销。
原文摘要 · Abstract (English)
Sparse autoencoders (SAEs) provide a powerful mechanism for decomposing the dense representations produced by Large Language Models (LLMs) into interpretable latent features. We posit that SAEs constitute a natural foundation for Learned Sparse Retrieval (LSR), whose objective is to encode queries and documents into high-dimensional sparse representations optimized for efficient retrieval. In contrast to existing LSR approaches that project input sequences into the vocabulary space, SAE-based representations offer the potential to produce more semantically structured, expressive, and language-agnostic features. Building on this insight, we introduce SPLARE, a method to train SAE-based LSR models. Our experiments, relying on recently released open-source SAEs, demonstrate that this technique consistently outperforms vocabulary-based LSR in multilingual and out-of-domain settings. SPLARE-7B, a multilingual retrieval model capable of producing generalizable sparse latent embeddings for a wide range of languages and domains, achieves top results on MMTEB's multilingual and English retrieval tasks. We also developed a 2B-parameter variant with a significantly lighter footprint.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。