用语义概念替换词表，让SPLADE更高效准确
From Tokens to Concepts: Leveraging SAE for SPLADE
- 用稀疏自编码器学习语义概念替代原始词表
- 在域内域外任务上性能接近SPLADE，效率更高
- 适合需要多语言多模态的检索系统
学习型稀疏信息检索模型（如SPLADE）在效率与效果间取得了良好平衡。然而，它们依赖底层骨干词表，可能因一词多义和同义现象限制性能，并给多语言和多模态应用带来挑战。为解决这一问题，我们提出用通过稀疏自编码器（SAE）学习的潜在语义概念空间替代骨干词表。本文研究了这两类概念的兼容性，探索了训练方法，并分析了SAE-SPLADE模型与传统SPLADE模型的差异。实验表明，SAE-SPLADE在域内和域外任务上的检索性能与SPLADE相当，同时具备更高效率。
原文摘要 · Abstract (English)
Learned Sparse IR models, such as SPLADE, offer an excellent efficiency-effectiveness tradeoff. However, they rely on the underlying backbone vocabulary, which might hinder performance (polysemicity and synonymy) and pose a challenge for multi-lingual and multi-modal usages. To solve this limitation, we propose to replace the backbone vocabulary with a latent space of semantic concepts learned using Sparse Auto-Encoders (SAE). Throughout this paper, we study the compatibility of these 2 concepts, explore training approaches, and analyze the differences between our SAE-SPLADE model and traditional SPLADE models. Our experiments demonstrate that SAE-SPLADE achieves retrieval performance comparable to SPLADE on both in-domain and out-of-domain tasks while offering improved efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。