通过均匀采样分数分布，提升知识蒸馏的检索性能
Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation for Dense Retrieval
- 采用分层采样策略覆盖全部得分范围，避免仅依赖难例
- 在跨域和域内测试中显著优于Top-K与随机采样
- 适合需要强泛化能力的检索模型训练场景
通过知识蒸馏（KD）从交叉编码器教师模型迁移知识，已成为训练检索模型的标准方法。现有研究多聚焦于挖掘难负样本以增强区分性，但对训练数据的系统构成及教师得分分布的关注较少。本文指出，仅关注难负样本会阻碍学生模型学习教师的完整偏好结构，影响泛化能力。为此，提出分层采样策略，均匀覆盖整个得分谱。在域内与跨域基准上的实验表明，该方法能有效保持教师得分的方差与熵，作为稳健基线，在多种设置下显著优于Top-K与随机采样。结果表明，蒸馏的核心在于保留教师所感知的多样化相对得分范围。
原文摘要 · Abstract (English)
Transferring knowledge from a cross-encoder teacher via Knowledge Distillation (KD) has become a standard paradigm for training retrieval models. While existing studies have largely focused on mining hard negatives to improve discrimination, the systematic composition of training data and the resulting teacher score distribution have received relatively less attention. In this work, we highlight that focusing solely on hard negatives prevents the student from learning the comprehensive preference structure of the teacher, potentially hampering generalization. To effectively emulate the teacher score distribution, we propose a Stratified Sampling strategy that uniformly covers the entire score spectrum. Experiments on in-domain and out-of-domain benchmarks confirm that Stratified Sampling, which preserves the variance and entropy of teacher scores, serves as a robust baseline, significantly outperforming top-K and random sampling in diverse settings. These findings suggest that the essence of distillation lies in preserving the diverse range of relative scores perceived by the teacher.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。