对比单阶段与多阶段微调，发现两者在段落重排序上效果相当。
Exploring the Effectiveness of Multi-stage Fine-tuning for Cross-encoder Re-rankers
- 采用对比学习的单阶段微调，性能与多阶段方法相当。
- 无需人工标注数据，仅靠大模型蒸馏也能达到优秀效果。
- 适合追求高效、低资源的检索系统开发者参考。
当前最先进的交叉编码器在段落重排序任务中表现优异。传统微调方法需要大量人工标注数据、对比学习目标及启发式采样的负样本。另一种近期方法则是通过知识蒸馏，让模型模仿高性能大语言模型的排序结果。本文系统评估了点对点交叉编码器在单阶段和双阶段微调下的表现。实验表明，使用对比学习的单阶段微调性能与多阶段方法相当。代码已公开于 https://github.com/fpezzuti/multistage-finetuning。
原文摘要 · Abstract (English)
State-of-the-art cross-encoders can be fine-tuned to be highly effective in passage re-ranking. The typical fine-tuning process of cross-encoders as re-rankers requires large amounts of manually labelled data, a contrastive learning objective, and a set of heuristically sampled negatives. An alternative recent approach for fine-tuning instead involves teaching the model to mimic the rankings of a highly effective large language model using a distillation objective. These fine-tuning strategies can be applied either individually, or in sequence. In this work, we systematically investigate the effectiveness of point-wise cross-encoders when fine-tuned independently in a single stage, or sequentially in two stages. Our experiments show that the effectiveness of point-wise cross-encoders fine-tuned using contrastive learning is indeed on par with that of models fine-tuned with multi-stage approaches. Code is available for reproduction at https://github.com/fpezzuti/multistage-finetuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。