长上下文微调反而提升短任务表现，揭示了模型知识偏好机制。
When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models
- 分离分析MHA与FFN组件，发现两者均受益于长上下文微调。
- 长微调促进上下文知识，短微调偏向参数知识，存在知识偏好偏差。
- 混合训练可缓解偏差，为模型微调提供可解释指导。
大型语言模型（LLMs）在自然语言处理任务中取得了显著性能。随着真实应用对更长上下文窗口的需求增加，基于长上下文数据的持续预训练和监督微调（SFT）已成为常见方法。尽管长上下文预训练的影响已得到广泛研究，其在监督微调中的影响仍不明确。本文系统研究了微调数据长度对短上下文任务行为的影响。反直觉的是，我们发现长上下文SFT提升了短上下文性能，这与长上下文预训练常导致性能下降的现象相反。为揭示其内在机制，我们首先解耦并分析多头注意力（MHA）和前馈网络（FFN）两个关键组件，发现二者独立受益于长上下文SFT。进一步研究它们的交互，揭示出一种知识偏好偏差：长上下文SFT促进上下文知识，而短上下文SFT更偏好参数知识，因此完全依赖长上下文SFT并非最优。最后，我们证明混合训练可缓解此偏差，为大模型微调提供可解释的指导。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved impressive performance across natural language processing (NLP) tasks. As real-world applications increasingly demand longer context windows, continued pretraining and supervised fine-tuning (SFT) on long-context data has become a common approach. While the effects of data length in continued pretraining have been extensively studied, their implications for SFT remain unclear. In this work, we systematically investigate how SFT data length influences LLM behavior on short-context tasks. Counterintuitively, we find that long-context SFT improves short-context performance, contrary to the commonly observed degradation from long-context pretraining. To uncover the underlying mechanisms of this phenomenon, we first decouple and analyze two key components, Multi-Head Attention (MHA) and Feed-Forward Network (FFN), and show that both independently benefit from long-context SFT. We further study their interaction and reveal a knowledge preference bias: long-context SFT promotes contextual knowledge, while short-context SFT favors parametric knowledge, making exclusive reliance on long-context SFT suboptimal. Finally, we demonstrate that hybrid training mitigates this bias, offering explainable guidance for fine-tuning LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。