用分数向量蒸馏大模型，让小模型更快更准地检索。
Score-Only Distillation for Compact Dense Retrieval
- 仅用教师模型的得分向量训练学生模型，不依赖隐藏状态。
- 在8个任务上恢复了基线到教师差距的50%性能。
- 适合追求高效推理的部署场景，尤其在线服务应用。
大型嵌入模型虽能提升检索质量，但在线服务时开销巨大。本文研究小型检索器能否仅通过教师模型的得分向量学习其排序行为，而无需访问教师隐藏状态。学生模型在由真实正样本与负样本构成的数据行上训练，使用我们设计的数据生成管道；同时独立评估了学生-教师硬负样本挖掘的扩展效果。采用以行为中心的得分向量目标函数，实现了一种内存高效的统一全对配对均方误差损失（PairMSE）。在固定八任务评估面板上，蒸馏协议恢复了高达50%的基线至教师性能差距。经蒸馏的0.6B参数学生模型，查询编码速度比顺序在线教师融合快4.7倍，文档编码快9.7倍。蒸馏后外部迁移性能表现参差，因此证据支持在匹配检索协议下压缩教师排序行为。
原文摘要 · Abstract (English)
Large embedding models improve retrieval quality, but serving large encoders online is expensive. We study whether a compact retriever can learn teacher ranking behavior from score vectors without access to teacher hidden states. The student trains on rows built from ground-truth positives and negative candidates produced by our data generation pipeline; we evaluate student-teacher hard-negative mining separately as an extension. We use a row-centered score-vector objective, a memory-efficient implementation of uniform all-pairs PairMSE loss. On a fixed eight-task evaluation panel, our distillation protocol recovers up to 50% of the base-to-teacher gap. The distilled 0.6B student is 4.7 times faster for query encoding and 9.7 times faster for document encoding than sequential online teacher fusion. External-transfer performance after distillation remains mixed, so our evidence supports compression of teacher rankings under matched retrieval protocols.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。