arXiv:2601.01684cs.IRcs.CL2026-01被引 6

用两阶段训练让小模型实现高效精准检索,跑在普通电脑上

LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum

  • 分两阶段训练:先弱监督微调,再用精选难例精调
  • 80亿模型在MTEB上达60.2 nDCG@10,内存比密集模型少74%
  • 适合资源有限但要高精度检索的落地场景

尽管密集检索模型是当前信息检索的主流,但其部署受限于高内存需求和对GPU加速器的依赖。学习型稀疏检索通过倒排索引实现高效搜索,却长期未受重视。本文提出LACONIC,基于Llama3架构(1B、3B、8B)的系列学习型稀疏检索模型。采用简化两阶段训练流程:(1) 弱监督预微调以适配因果大模型进行双向上下文建模;(2) 使用精心筛选的困难负样本进行高信号微调。实验表明,LACONIC有效缩小与密集模型的性能差距:8B版本在MTEB Retrieval基准上达到60.2 nDCG@10,截至2026年2月5日位列排行榜第15名,且索引内存比等效密集模型减少74%。该模型在普通CPU硬件上实现高检索效果,仅需极小计算预算,为真实世界搜索应用提供了可扩展、高效的解决方案。代码与训练好的检查点已完全开源,支持复现。

原文摘要 · Abstract (English)

While dense retrieval models have been the standard for state-of-the-art information retrieval, their deployment is often constrained by high memory requirements and reliance on GPU accelerators for vector similarity search at scale. Learned sparse retrieval offers a compelling alternative by enabling efficient search via inverted indices, yet it has historically received less attention than dense approaches. In this paper, we introduce LACONIC, a family of learned sparse retrievers based on the Llama3 architecture (1B, 3B, and 8B). We propose a streamlined two-phase training curriculum consisting of (1) weakly supervised pre-finetuning to adapt causal LLMs for bidirectional contextualization and (2) high-signal finetuning using curated hard negatives. Our results demonstrate that LACONIC effectively bridges the performance gap with dense models: the 8B variant achieves a state-of-the-art 60.2 nDCG@10 on the MTEB Retrieval benchmark, ranking 15th on the leaderboard as of February 5th, 2026, while utilizing 74% less index memory than an equivalent dense model. By delivering high retrieval effectiveness on commodity CPU hardware with a fraction of the compute budget required by competing models, LACONIC provides a scalable and efficient solution for real-world search applications. We fully open source our code implementation and trained checkpoints to facilitate reproducibility.

稀疏检索轻量化大模型检索系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。