微调反而让检索模型变差，原因在于破坏了预训练学好的嵌入结构。
When Fine-Tuning Fails: Lessons from MS MARCO Passage Ranking
- 对比五种微调方法，全参数与高效微调均表现不如基础模型
- 基础模型在MS MARCO上MRR@10达0.3026，所有微调方案均低于此
- 适合关注预训练迁移局限性的检索系统研究者
本文探究了预训练Transformer模型在MS MARCO段落排序任务中微调后性能下降的反直觉现象。通过涵盖五种模型变体的全面实验，包括全参数微调和参数高效的LoRA适配，我们发现所有微调方法均劣于基础模型sentence-transformers/all-MiniLM-L6-v2（MRR@10: 0.3026）。分析表明，微调破坏了基础模型在10亿句对数据上预训练所建立的最优嵌入空间结构，其中包含910万条MS MARCO样本。UMAP可视化显示嵌入空间逐渐扁平化，训练动态与计算效率指标进一步验证该结论。这些结果挑战了在饱和基准上迁移学习有效性的常规认知，暗示架构创新可能是实现显著提升的必要条件。
原文摘要 · Abstract (English)
This paper investigates the counterintuitive phenomenon where fine-tuning pre-trained transformer models degrades performance on the MS MARCO passage ranking task. Through comprehensive experiments involving five model variants-including full parameter fine-tuning and parameter efficient LoRA adaptations-we demonstrate that all fine-tuning approaches underperform the base sentence-transformers/all- MiniLM-L6-v2 model (MRR@10: 0.3026). Our analysis reveals that fine-tuning disrupts the optimal embedding space structure learned during the base model's extensive pre-training on 1 billion sentence pairs, including 9.1 million MS MARCO samples. UMAP visualizations show progressive embedding space flattening, while training dynamics analysis and computational efficiency metrics further support our findings. These results challenge conventional wisdom about transfer learning effectiveness on saturated benchmarks and suggest architectural innovations may be necessary for meaningful improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。