拆解排序蒸馏中局部性与熵的影响,发现传统方法可能无效甚至有害。
Disentangling Locality and Entropy in Ranking Distillation
- 区分样本选择与教师模型熵的独立影响,理论证明其正交性
- 实验证明对比学习有效样本在蒸馏中可能无益或有害
- 建议减少对对比对依赖,更关注真实场景下的训练动态
排序模型训练涉及采样策略与标注策略两大关键决策。现代语义搜索系统通常采用“难负样本”采样识别挑战项,并通过蒸馏将强模型的排序知识迁移。但这些方法日益复杂昂贵,例如SentenceTransformers使用12个模型集成,且数据来源不透明影响复现性。尽管如此,当前采样与标注策略尚未充分消融,其有效性来源仍不清晰。为更好理解模型提升原因并降低训练成本,本文对神经排序中的采样与蒸馏过程进行全面消融。我们从理论上推导了样本选择影响的模型几何结构与教师排序熵对优化的影响具有正交性,建立数据增强改善排序模型偏差的有效条件。在主流基准和常见架构上的实验表明,过去在对比目标下有效的采样策略在蒸馏中可能无效甚至有害。我们进一步研究输入与目标层面的数据增强如何影响效果及模型内在行为。本工作旨在推动更高效的方法,减少对对比对的关注,转而直接理解排序场景下的训练动态,以更贴近真实应用。
原文摘要 · Abstract (English)
The training process of ranking models involves two key data selection decisions: a sampling strategy, and a labeling strategy. Modern ranking systems, especially those for performing semantic search, typically use a ``hard negative'' sampling strategy to identify challenging items using heuristics and a distillation labeling strategy to transfer ranking "knowledge" from a more capable model. In practice, these approaches have grown increasingly expensive and complex, for instance, popular pretrained rankers from SentenceTransformers involve 12 models in an ensemble with data provenance hampering reproducibility. Despite their complexity, modern sampling and labeling strategies have not been fully ablated, leaving the underlying source of effectiveness gains unclear. Thus, to better understand why models improve and potentially reduce the expense of training effective models, we conduct a broad ablation of sampling and distillation processes in neural ranking. We frame and theoretically derive the orthogonal nature of model geometry affected by example selection and the effect of teacher ranking entropy on ranking model optimization, establishing conditions in which data augmentation can effectively improve bias in a ranking model. Empirically, our investigation on established benchmarks and common architectures shows that sampling processes that were once highly effective in contrastive objectives may be spurious or harmful under distillation. We further investigate how data augmentation, in terms of inputs and targets, can affect effectiveness and the intrinsic behavior of models in ranking. Through this work, we aim to encourage more computationally efficient approaches that reduce focus on contrastive pairs and instead directly understand training dynamics under rankings, which better represent real-world settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。