arXiv:2505.07166cs.IRcs.CL2025-05中稿 · SIGIR-2025被引 4

验证预训练与微调在密集检索中的知识获取作用,发现效果因模型结构而异。

Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition

  • 对比不同编码方式和模型架构,检验微调是否能补充预训练缺失的知识
  • 预训练主导性能,微调主要调整激活而非重构知识(除特定模型外)
  • 结果对BERT+DPR有效,但不适用于均值池化或解码器模型

密集检索器利用预训练语言模型(如BERT、LLaMA)作为主干网络,通过对比学习微调以将文本编码为语义表示,再通过浅层相似性操作(如内积)进行比较。近期研究质疑微调在密集检索中的作用,认为检索知识主要在预训练阶段获得,微调无法补足预训练未掌握的知识。本文重新审视该观点,此前研究仅基于BERT编码器和DPR模型。本研究扩展至其他表示方法(使用CLS token vs. 均值池化)、主干架构(仅编码器BERT vs. 仅解码器LLaMA)及数据集(MSMARCO + Natural Questions)。结果表明,在DPR设置中,预训练知识决定检索性能,微调主要调整神经元激活而非重组知识;但此规律不适用于均值池化的Contriever或基于解码器的LLaMA模型。所有实验代码已公开于https://github.com/ielab/DenseRetriever-Knowledge-Acquisition,确保可复现。

原文摘要 · Abstract (English)

Dense retrievers utilize pre-trained backbone language models (e.g., BERT, LLaMA) that are fine-tuned via contrastive learning to perform the task of encoding text into sense representations that can be then compared via a shallow similarity operation, e.g. inner product. Recent research has questioned the role of fine-tuning vs. that of pre-training within dense retrievers, specifically arguing that retrieval knowledge is primarily gained during pre-training, meaning knowledge not acquired during pre-training cannot be sub-sequentially acquired via fine-tuning. We revisit this idea here as the claim was only studied in the context of a BERT-based encoder using DPR as representative dense retriever. We extend the previous analysis by testing other representation approaches (comparing the use of CLS tokens with that of mean pooling), backbone architectures (encoder-only BERT vs. decoder-only LLaMA), and additional datasets (MSMARCO in addition to Natural Questions). Our study confirms that in DPR tuning, pre-trained knowledge underpins retrieval performance, with fine-tuning primarily adjusting neuron activation rather than reorganizing knowledge. However, this pattern does not hold universally, such as in mean-pooled (Contriever) and decoder-based (LLaMA) models. We ensure full reproducibility and make our implementation publicly available at https://github.com/ielab/DenseRetriever-Knowledge-Acquisition.

检索模型知识获取预训练可复现

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。