LoRA微调中,中间层注意力变化决定重排序性能提升。
The Emergence of Relevance Through Axiomatic Attention Patterns During LoRA Fine-Tuning

- 仅微调网络中段注意力层,即可恢复超一半性能增益。
- 注意力更新区域与关键词匹配、罕见词敏感等特征相关。
- 适合研究大模型可解释性与高效微调的读者。
LoRA微调是适应大语言模型用于重排序的标准方法,但尚不清楚任务相关的相关性行为在何处学习以及伴随的注意力变化为何。通过消融实验和注意力分析,我们识别出对RankLLaMA性能提升有贡献的LoRA注意力更新位置,并检验这些提升是否与可解释的相关性注意力模式(如词汇匹配、罕见性敏感度、查询-文档交互)一致。结果表明:在全网络中使用LoRA微调的MLP时,仅将注意力更新限制在紧凑的中段网络区域,即可恢复超过一半的性能增益;而在此区域省略注意力微调对性能的影响大于其他区域。此外,影响性能最大的区域与微调后注意力增强的公理化信息检索特征高度重叠。罕见性敏感度、文档-查询交互及若干组合特征与排名性能提升高度相关。结果支持一种可解释的、基于相关性的重排序行为涌现机制,并为优化重排序器适配策略提供方向。
原文摘要 · Abstract (English)
LoRA fine-tuning is standard for adapting LLMs to reranking, but it remains unclear where in the network task-specific relevance behavior is learned and what attention-level changes accompany that learning. Through ablation and attention experiments, we identify where LoRA attention updates to RankLLaMA improve performance and whether those gains coincide with interpretable relevance-oriented attention patterns such as lexical matching, rarity sensitivity, and query-document interaction. We find that given LoRA fine-tuned MLPs throughout the network, restricting LoRA attention updates to a compact mid-network region is sufficient for recovering over half of the performance gained by applying LoRA to all attention layers, and that omitting attention fine-tuning in this region hurts performance more than elsewhere in the network. Additionally, we show that regions where applying LoRA affects performance the most overlap with regions where fine-tuning increased attention to axiomatic IR features. Rarity sensitivity, document-query interaction, and several compositional features are highly correlated with gains in ranking performance. Our results support an interpretable, correlational account of how relevance-oriented behavior emerges during LoRA fine-tuning and point toward improved strategies for adapting rerankers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。